Web-based collaborative ink writing via a computer network

Abstract

Techniques for network-based collaborative ink writing sessions via a computer network are described by this disclosure. In one embodiment, a first computer is capable of serializing detected pointer inputs at the first computer into data representing one or more ink objects, where the ink objects individually represent an image or animation corresponding to the detected pointer inputs. The first computer can then transmit data representing the serialized ink objects via the computer network to a second computer. After receiving the transmitted data, the second computer can deserialize the received serialized ink objects, reconstruct an image or animation corresponding to the detected pointer inputs at the first computer, and present an image or animation of the detected pointer inputs to a second user on a second web browser that displays the same web page on the second computer, thereby enabling a network-based collaborative session between the first and second users.

Description

BACKGROUND OF THE INVENTION

[0001] Modern computing devices such as desktop computers, smartphones, tablets, and laptop computers typically may include a touchscreen as an input / output component. A user may interact with such a computing device via the touchscreen using a stylus, pen, mouse, touchpad, or even the user's finger. For example, a user may use a stylus or finger to navigate via the touchscreen through menus, draw pictures, write notes, sign documents, or perform other suitable operations. SUMMARY OF THE INVENTION

[0002] This Summary of the Invention is provided to introduce in a simplified form a few concepts that are further described below in the Detailed Description. This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] In computing, pen computing generally refers to a user interface using a pen, stylus, the user's finger, mouse, or touchpad as a pointer input device for providing digital ink (referred to herein as "ink" for brevity) or other suitable types of pointer input to a computer. In some computing devices, a pen computing application employing such a user interface may allow a user to use a stylus, pen, finger, mouse, or touchpad to handwrite comments, notes, edit marks, or other annotations for a book or document displayed on a touchscreen or other suitable type of output device. For example, a user may use a pen computing application to handwrite annotations such as "This is interesting", "Read again", "Funny", or other comments or notes, thereby referring to a book or document displayed on the touchscreen. Handwritten annotations may also include graphical representations such as checkmarks, smiley faces, angry faces, etc. The pen computing application may also allow the user to share his / her handwritten annotations as a still image, animation, video, or other suitable type of media with other users.

[0004] Sharing annotations as digital ink can be an excellent tool for collaboration in a team environment. However, such sharing typically requires the use of a specific pen computing application (such as Windows provided by Microsoft Corporation of Redmond, Washington ), and it is not available in a web browser. For example, when a user reads a web page in a web browser, the user may provide a handwritten annotation as ink on a tablet. To share the handwritten annotation, the user may save the ink as an image and then share the saved image as an attachment in an email, for example, with other users. Such sharing techniques can be cumbersome and do not facilitate real-time collaboration with other users in a team. Thus, when working in a team environment on the web, the user's collaboration experience may be limited.

[0005] Some embodiments of the disclosed technology can address at least some of the above difficulties by implementing an application configured to facilitate web-based collaborative inking through a user (e.g., as an extension of a web browser and / or an integrated part of a web browser). Such web-based collaboration can be across different types of web browsers and / or computing platforms. Such web-based collaboration can also be real-time or at different points in time. For example, users can collaborate simultaneously during an interactive session to ink on a single web page. In another example, a user can share ink from a previous collaborative session with other users and allow the other users to continue inking and / or editing the shared ink at a later date / time. Such web-based collaboration can also allow the shared ink to remain "wet", i.e., for a preset or user-selectable period of time, the ink can be edited by the sharing user(s) and / or the shared user(s).

[0006] In some embodiments, the application can be configured to inject one or more rendering contexts (e.g., as individual transparent overlays or ink rendering overlays) into a web page displayed in a web browser on a touch screen of a first computer after receiving user input from a first user to activate web-based collaborative inking. For example, in one implementation, the application can be configured to inject a scalable vector graphics (SVG) rendering context into the web page by inserting appropriate code of an extensible markup language (XML) or other suitable type of markup language into the document object model (DOM) of the web page. In other implementations, the application can also inject a canvas rendering context configured according to the HyperText Markup Language (HTML) 5 standard, InkML overlay, Ink Serialization Format (ISF) overlay, or according to other suitable formats. In further implementations, multiple rendering contexts can be used. For example, the ink can be anchored to a content item on the web page, and an ink rendering overlay can be injected at the location and / or (multiple) z-depth of the anchored content item. Thus, other web page content (e.g., tooltips, pop-up menus, modal dialogs, etc.) on the same web page can be correctly rendered above / below the injected ink rendering overlay.

[0007] Then, the application can be configured to detect pointer input from a user via a touch screen or other suitable input device of a computer as ink writing. In response to detecting the pointer input, the application can be configured to serialize an ink object corresponding to the detected pointer input. In the context of data manipulation, serialization is the process of converting a data structure or object state into a format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., via a computer network) and then reconstructed later (e.g., in a different computer environment). Thus, by serializing the detected pointer input from the user, the application can generate ink data that can be stored, transmitted, and reconstructed to represent the detected pointer input on the touch screen. In one example implementation, the detected pointer input can be converted into one or more <path>Object. In other embodiments, the detected pointer input can also be converted into one or more canvas objects, InkML objects, ISF objects, or other suitable types of objects.

[0008] In some embodiments, after receiving another user input to initiate a network-based collaboration session, the application can be configured to push or otherwise send serialized ink objects to a remote server, along with a session ID, session policy, access credentials, and / or other suitable information. According to embodiments of the disclosed technology, the remote server can be configured to store the serialized ink objects received from the application in a session buffer corresponding to the session ID, and enforce any session policies for accessing and / or updating the received serialized ink objects, as described in more detail below.

[0009] On a second computer, after receiving a user input from a second user to join a collaboration session initiated by a first user, another copy of the application on the second computer can be configured to inject a rendering context onto the same web page displayed in a web browser on the second computer, as described above with reference to injecting a rendering context on the first computer. The application can then be configured to send a request to the remote server, along with user identification, credentials, or other suitable information, to join the collaboration session. After the second user is authenticated, the remote server can be configured to provide the serialized ink objects to the second computer via a computer network. After receiving the serialized ink objects from the remote server, the application at the second computer can be configured to deserialize the ink objects and render the deserialized ink objects in the injected rendering context on the web page at the second computer. For example, in one exemplary embodiment, <path>The objects may be sequentially deserialized to generate an animation of pointer input received from a first user at a first computer. In some embodiments, the generated animation may be a generally accurate representation of the pointer input received at the first computer by including data such as stroke type, stroke width, stroke speed, and / or other suitable parameters. In other embodiments, the generated animation may be a simplified representation of the pointer input received at the first computer by representing all strokes with the same color, stroke width, or other suitable parameters.

[0010] In some embodiments, the second user may be allowed to participate in a network-based collaborative session by providing additional ink and / or modifying existing ink. For example, the second user may provide additional pointer input on the second computer to edit, erase, add or otherwise modify the ink object from the first user on the second computer. In response, the application at the second computer may be configured to serialize the received pointer input into additional and / or updated ink objects, and transmit such ink objects to the remote server. The remote server may then add and / or update the ink objects stored on the remote server accordingly. In some embodiments, the ink objects of different users may be tracked and redrawn / edited by other users individually. In other embodiments, the ink objects of different users may be merged into a group of ink objects. In another embodiment, the remote server may also utilize suitable applications and / or computing services to convert, reduce and / or reconfigure the ink objects into different object formats. In yet another embodiment, the remote server may also utilize suitable applications and / or computing services to perform optical character recognition (OCR), language translation and / or other suitable operations.

[0011] In other embodiments, based on a session policy received from the first user at the remote server (or other suitable entity, such as a session administrator), the second user may be allowed to perform some of the aforementioned actions. For example, the session policy may limit the second user's access to ink objects to viewing only. In this way, the remote server may provide the second user with a serialized ink object to be rendered by an application at the second computer. However, any additional serialized ink objects received from the second user via the second computer may be ignored. In another example, the session policy may limit the second user's access to ink objects to only adding, but not modifying, any ink objects created by the first user. In this way, the remote server may add additional ink objects from the second user to the session buffer, but ignore any ink objects that represent modifications to the original ink objects from the first user.

[0012] Certain embodiments of the disclosed technology thus enable web-based collaborative ink writing for multiple users across various types of web browsers and / or computing platforms. Such web-based collaboration can also be real-time or at different points in time. For example, users can collaborate simultaneously to ink write on the same web page, or users can continue to ink write and / or edit the shared ink from a previous session at a later date / time. Such web-based collaboration can also allow the shared ink to remain "wet," i.e., editable by the sharing user(s) and / or the shared user(s) for a period of time. By allowing web-based collaborative ink writing via a web browser, the computational load at various computers of the users can be reduced by not running a specific application while improving the users' collaboration experience.

[0013] Although the disclosed technology is described in the context of being facilitated by a remote server, in additional embodiments, the remote server can be omitted when a first computer and a second computer can collaborate to facilitate web-based collaborative ink writing in a peer-to-peer manner. For example, both the first computer and the second computer can each be individually configured as a server having a corresponding session buffer for storing serialized ink objects and provide serialized ink objects from the session buffer to the other computer during a collaborative session. In another example, one of the first computer or the second computer can be configured as a server while the other computer is configured as a client. In yet another embodiment, the foregoing collaborative ink writing technology can be facilitated by one or more database servers, authentication servers, network gateways, and / or other suitable computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figures 1A - 1C is a schematic diagram of a computing system implementing web-based collaborative ink writing during certain stages of operation in accordance with an embodiment of the present technology.

[0015] Figure 2A and Figure 2B is a schematic diagram of another computing system implementing web-based collaborative ink writing during certain stages of operation in accordance with an embodiment of the present technology.

[0016] Figure 3 is illustrative of Figures 1A - 2B example software components of a computing device in

[0017] Figures 4A - 4C shows a flowchart of a method of web-based collaborative ink writing in accordance with an embodiment of the disclosed technology.

[0018] Figure 5 is a schematic diagram showing Figures 1A - 2B example hardware components of a computing system in Detailed Implementation Modes

[0019] Certain embodiments of computing systems, devices, components, modules, routines, and processes for implementing network-based collaborative ink writing are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art will also understand that the disclosed technology may have additional embodiments or may be practiced without several details of the embodiments described below. Figures 1A - 5 As used herein, the term "ink" or "digital ink" refers to a digital image, animation, or other suitable type of data that represents one or more strokes received at an input component (e.g., a touch screen) of a computing device by a user using a stylus, pen, the user's finger, a mouse, a touchpad, or other suitable pointer input device related to pen computing. A stroke or combination of strokes may form one or more "ink objects" having corresponding shapes, lengths, repeating patterns, vectors, or other suitable characteristics. Ink objects may be encoded according to various formats. In one example, an ink object may include an SVG-based

[0020] <path>Object. In other examples, the ink object can include a canvas, InkML, ISF, or other suitable types of objects.

[0021] As used herein, "serialization" generally refers to the process of converting a data structure or object state into a bit format that can be stored (e.g., in a file or memory buffer) or transmitted (e.g., via a computer network), and then reconstructed in real-time, near real-time, or at a later time (e.g., in the same or a different computer environment). Depending on the serialization format, when the resulting series of bits is reread or "deserialized", the series of bits can be used to recreate a clone that is semantically generally the same as the original object, such as an image or animation of the original stroke. Thus, by serializing the detected pointer input, an application can generate data representing the ink object, which can be stored, transmitted, and reconstructed on a destination computer to represent the detected pointer input received at the source computer, thereby enabling network-based "collaborative ink writing".

[0022] Sharing digital ink can be an excellent tool for collaboration in a team environment. However, such sharing typically requires the use of a specific pen computing application (such as those provided by Microsoft Corporation of Redmond, Washington, like Windows ) and is not available in a web browser. For example, when a user reads a web page in a web browser, the user can provide handwritten annotations as ink on a tablet. To share the handwritten annotations, the user can save the ink as an image and then share the saved image as an attachment in, for example, an email to other users. Such sharing techniques can be cumbersome and do not facilitate real-time collaboration with other users in the team. Thus, when working in a team environment on the web, the user's collaboration experience may be limited.

[0023] Several embodiments of the disclosed technology can address at least some of the above difficulties by implementing an application configured to facilitate network-based collaborative ink writing by a user (e.g., as an extension of a web browser and / or an integrated part of a web browser). Such network-based collaboration can span different types of web browsers and / or computing platforms. Such network-based collaboration can also be real-time or at different points in time. For example, users can collaborate simultaneously during a collaborative session to ink write on a single web page. In another example, a user can share ink from a previous collaborative session with other users and allow the other users to continue ink writing and / or editing the shared ink at a later date / time. Such network-based collaboration can also allow the shared ink to remain "wet", i.e., for a period of time, the ink can be edited by the shared and / or sharing user(s), as described in more detail below with reference to Figures 1A - 5 more detail.

[0024] Figures 1A - 1C is a schematic diagram of a computing system that implements web-based collaborative ink writing during certain stages of operation in accordance with an embodiment of the present technology. As Figure 1A shown, the computing system 100 may include a computer network 104 that interconnects client devices 102 (shown as a first client device 102a and a second client device 102b in Figure 1A ), and one or more remote servers (referred to herein as remote server 106 for simplicity) interconnected to a data storage device 108. The computer network 104 may include an enterprise intranet, a social network, the Internet, or other suitable types of networks. Although embodiments of the disclosed technology are described in the context of an enterprise intranet below, in other embodiments, the disclosed technology may also be implemented in other suitable types of computing systems with additional and / or different components.

[0025] The client devices 102 may each individually include a computing device that facilitates collaborative ink writing by users 101 (identified as a first user 101a and a second user 101b) via the computer network 104. For example, in an illustrative embodiment, the client devices 102 each individually include a desktop computer. In other embodiments, the client devices 102 may also each individually include a laptop computer, a tablet computer, a smartphone, or other suitable types of computing devices. Although two users 101 are shown in Figure 1A for illustrative purposes, in other embodiments, the computing system 100 may facilitate any suitable number of members 101 during a collaborative ink writing session via the computer network 104.

[0026] As Figure 1A shown, the client devices 102 may each individually execute suitable instructions to provide a web browser 120 (or other suitable application configured to render web pages). In the illustrated embodiment, the web browser 120 may include a user interface having an address bar 122 that contains a web address (e.g., "http: / / www.jokes.com"), and a body portion 121 that contains the content of a web page 123 corresponding to the web address in the address bar 122. For example, the web page 123 may include a statement "My dog used to chase people on bikes. It got so bad, I finally had to take his bike away" as the text in the web page 123. In other examples, the body portion 121 may also contain images, videos, and / or other suitable types of content.

[0027] The web browser 120 may be configured to facilitate collaborative ink writing by, for example, including, installing, or otherwise accessing an ink application 130 ( Figure 3 as shown in FIG. to facilitate collaborative ink writing. The ink application 130 can be an extension of the web browser 120, integrated into the web browser 120, a stand-alone application, or other suitable type of application program. The ink application 130 can be configured to provide user interface elements for activating collaborative ink writing. For example, as Figure 1A shown, the web browser 120 can include an "Ink Writing" button 124. After being launched by the first user 101a, the web browser 120 can insert a rendering context 128 (shown in dashed lines for clarity) on top of the web page 123. The rendering context 128 can include an overlay configured to render and present pointer inputs received from the user 101 on the web page 123. In one embodiment, inserting the rendering context 128 can insert an SVG portion representing a transparent overlay into the DOM of the web page 123. Example code for such an SVG portion is as follows:

[0028]

[0029] In other embodiments, code for a canvas, InkML, ISF, or other suitable type of image format can be inserted to supplement or replace the SVG portion. In other examples, collaborative ink writing can be activated via (a) menu item(s) or other suitable user interface elements, or can be automatically activated when the web page 123 is loaded into the web browser 120.

[0030] The first user 101a can then provide pointer inputs to the web page 123 via, for example, the touch screen of the first client device 102a. In response, the client device 102 can be configured to detect such pointer inputs, convert the detected pointer inputs into one or more ink objects 110 (as Figure 1B shown), and render the ink objects 110 on the inserted rendering context 128. For example, as Figure 1B shown, the first user 101a can write a handwritten comment 127, namely "This is interesting", on the web page 123 by referring to the text in the web page 123. In response, the ink application 130 can convert the handwritten manuscript of the first user 101a into one or more <path>object, and will <path>The object is inserted into the SVG part of the DOM of the web page 123 on the first client device 102a. Example <path>The objects are as follows:

[0031] <path d="M 100 350 q 150 -300 300 0" Stroke="blue" stroke-width="5" fill="none" / >

[0032] The inserted <path>The object can then be rendered on the rendering context 128 to overlay on the web page 123 as if the comment 127 were on the web page 123. In other examples, pointer inputs from the first user 101a can be captured along with animations, strokes, or other suitable types of objects.

[0033] The first user 101a can also decide to share his / her comment 127 with the second user 101b or additional users 101 (not shown). In the illustrated embodiment, the ink application 130 can be configured to provide another user interface element (e.g., a "Broadcast" button 125) on the web page 123. After being initiated by the first user 101a, the ink application 130 can serialize one or more ink objects 110 (shown as ink objects 110 1 , 110 2 , ... , 110 n ), and transmit the serialized ink objects 110 to the remote server 106.

[0034] In some embodiments, the remote server 106 can be configured to apply a default session policy to a collaboration session between the first user 101a and the second user 101b. In other embodiments, the first user 101a can also configure and transmit an optional set of session policies 111 to the remote server 106. The session policies 111 can include data specifying access levels to the ink objects 110. For example, the session policies 111 can specify that the second user 101b can only read, can only read and add, or can only read, add, and modify the shared ink objects 110.

[0035] In response to receiving the ink objects 110 and the optional session policies 111 from the first user 101a, the remote server 106 can be configured to store the ink objects 110 and the corresponding session identifiers in a data storage device. The remote server 106 can also be configured to facilitate authentication for the second user 101b to access the collaborative ink writing session, and provide the received ink objects 110 to the second client device 102a after successful authentication according to the default session policy or the optional session policies 111 from the first user 101a.

[0036] As Figure 1B As shown, to participate in a collaborative ink writing session, the second user 101b can also activate the "Ink Writing" button 124. In response, a rendering context 128 can be inserted onto the web page 123 in the web browser 120 on the second client device 102b. After receiving the ink object 110 from the remote server 106, the ink extension at the second client device 102b can deserialize the ink object 110 and render the deserialized ink object 110 as one or more images or animations in the inserted rendering context 128 at the second client device 102b. The aforementioned detection, serialization, deserialization, and rendering performed by the first client device 102a and the second client device 102b can be periodic (e.g., once every 20 milliseconds), event-based (e.g., after detecting a new pointer input), or based on other suitable bases. Thus, in some embodiments, when the first user 101a is writing his / her comment on the first client device 102a, the comment 127' from the first user 101a can be presented to the second user 101b in real time or near real time. In other embodiments, the ink object 110 can also remain "wet" to allow the first user 101a and / or the second user 101b to view, add, or modify at a later date / time.

[0037] Figure 1C An embodiment is shown where the session policy 111 allows the second user 101b to add and modify the comment 127' from the first user 101a. As Figure 1C shown, the second user 101b can handwrite an additional comment 129 on the web page 123 at the second client device 102b. In response, the ink application 130 at the second client device 102b can be configured to detect the pointer input and convert / serialize the pointer input into one or more additional ink objects 110' (shown as ink objects 110’ 1 , 110’ 2 ,..., 110’ n ), and transmit the serialized ink objects 110' to the remote server 106. The remote server 106 can then provide the additional ink objects 110' to the first client device 102a via the computer network 104. After receiving the additional ink objects 110', the ink application 130 at the first client device 101a can be configured to deserialize the additional ink objects 110' and render the deserialized additional ink objects 110' as one or more images or animations on the rendering context 128. Thus, when the second user 101b is writing the comment 129, the additional comment 129 from the second user 101b can be presented to the first user 101a.

[0038] Also as Figure 1C As shown, the second user 101 can also modify the comment 127 by the first user 101a on the second client device 101b. For example, the second user 101b can add an exclamation mark "!" to the end of the comment 127 of the first user 101a. In response, the ink application 130 at the second client device 102b can modify, for example, one or more ink objects 110 corresponding to the comment 127, and transmit the modified ink objects 110 to the remote server 106. The remote server 106 can then determine whether such a modification is allowed according to the session policy 111( Figure 1B ). After determining that such a modification for the second user 101b is allowed, the remote server 106 can modify (e.g., rewrite) a copy of the stored ink object 110 originally received from the first user 101a based on the modified ink object 110 received from the second user 101b. The modified ink object 110 can then be provided to the first device 102a and presented, as described above. Thus, the modification to the comment 127 can be presented to the first user 101a in a collaborative manner. After determining that such a modification is not allowed, the remote server 106 can be configured to reject and / or ignore the ink object 110' received from the second user 101b to implement access control.

[0039] Although Figures 1A - 1C the computing system 100 shown is configured to facilitate collaborative ink writing using the remote server 106, in some embodiments, the computing system 100 can also facilitate such collaborative ink writing without the remote server 106. For example, as Figure 2A shown, the first client device 102a and the second client device 102b can facilitate collaborative ink writing in a peer-to-peer manner. For example, the first client device 102a and the second client device 102b can be individually configured as servers for managing and providing ink objects 110 to other shared client devices 100. In Figure 2A the illustrated embodiment, the first client device 102a can be operatively coupled to or include a data storage device 108 for storing the ink objects 110 from the first user 101a. The first client device 101a can then be configured to provide the stored ink objects 110 to the second client device 101b based on, for example, the session policy 111( Figure 1B ). As Figure 2B shown, the second client device 102a can also be operatively coupled to or include a data storage device 108' for storing the ink objects 110 from the second user 101b. Then, the second client device 101b can be configured to provide the stored ink objects 110' to the first client device 101b based on, for example, the session policy 111.

[0040] Figure 3 is a schematic diagram of an example software component of a computing device described in Figures 1A - 2B . In Figure 3 , the components and functions therein are shown in the context of the first client device 102a and the second client device 102b sharing ink strokes. As such, the ink application 130 at the first client device 102a may include components different from those at the second client device 102b. However, in some embodiments, such sharing may be two-way. As such, the ink application 130 at both the first client device 102a and the second client device 102b may include all of the components described below.

[0041] In Figure 3 and other figures herein, individual software components, objects, classes, modules, and routines may be computer programs, steps, or processes written as source code in C, C++, C#, Java, and / or other suitable programming languages. Components may include, but are not limited to, one or more modules, objects, classes, routines, properties, processes, threads, executables, libraries, or other components. Components may be in source code or binary form. Components may include aspects of source code prior to compilation (e.g., classes, properties, steps, routines), compiled binary units (e.g., libraries, executables), or artifacts instantiated and used at runtime (e.g., objects, processes, threads).

[0042] Components within a system may take different forms within the system. As an example, a system including a first component, a second component, and a third component may, but is not limited to, a system containing a first component having characteristics as source code, a second component as a binary compiled library, and a third component as a thread created at runtime. Computer programs, steps, or processes may be compiled into object, intermediate, or machine code and presented for execution by one or more processors of a personal computer, web server, laptop computer, smartphone, and / or other suitable computing device. Similarly, components may include hardware circuitry. Those of ordinary skill in the art recognize that hardware may be considered rigid software, and software may be considered liquefied hardware. Just to give one example, software instructions in a component may be burned into a programmable logic array circuit, or may be designed as a hardware circuit with appropriate integrated circuits. Similarly, hardware may be simulated by software. Various implementations of source, intermediate, and / or object code and associated data may be stored in a computer memory, which includes read-only memory, random access memory, disk storage media, optical storage media, flash devices, and / or other suitable computer-readable storage media excluding propagated signals.

[0043] As Figure 3 As shown, the ink application 130 may include an ink detector 132, a serializer 134, and an ink output 136, which are operatively coupled to each other at the first client device 102a. The ink detector 132 may be configured to detect one or more pointer inputs 131 from the first user 101a and convert the detected pointer inputs 131 into electrical or other suitable types of signals. The ink detector 132 may include a touch screen driver and / or other suitable hardware / software components.

[0044] The ink detector 132 may then provide the detected pointer inputs 131 to the serializer 134 for further processing. The serializer 134 may be configured to convert the electrical signals corresponding to the detected pointer inputs 131 into one or more ink objects 110 that can be reconstructed, for example, by the second client device 102b. Example techniques for such conversion were described above with reference to Figures 1A - 1C . The ink output 136 may then be configured to transmit the ink objects 110 to the remote server 106 via the computer network 104 ( Figure 1A ). The foregoing detection, serialization, and transmission performed by the various components of the ink application 130 may be periodic (e.g., once every 20 milliseconds), event-based (e.g., based on detected new pointer inputs 131), or based on other suitable bases.

[0045] As described above, the remote server 106 may be configured to receive the ink objects 110 from the first client device 102a, store the received ink objects 110, and provide the ink objects 110 to the second client device 102b upon request. As Figure 3 shown, the ink application 130' at the second client device 102b may include a deserializer 141 and a rendering component 143 that are operatively coupled to each other. The deserializer 141 may be configured to deserialize or reconstruct one or more images or animations based on the received ink objects 110. For example, the deserializer 141 may be configured to combine multiple ink objects 110 (e.g., multiple <path>objects) to form a single image or animation. The rendering component 143 can be configured to render and present the image and / or animation reconstructed from the received ink objects.

[0046] Figures 4A - 4B FIG. shows a flowchart of a method of web-based collaborative ink writing according to an embodiment of the disclosed technology. Although embodiments of these methods are described in the context of the Figures 1A - 2B computing system 100 below, in other embodiments, these methods can also be implemented in computing systems with additional and / or different components.

[0047] As Figure 4A shown, the process 200 of collaborative ink writing can include generating serialized ink objects at a first computer at stage 202. The ink objects can be generated according to SVG, canvas, InkML, ISF, or other suitable image and / or animation formats. Example operations for generating serialized ink objects are described below with reference to Figure 4B The process 200 can then include transmitting the ink objects to a second computer at stage 204. In certain implementations, as Figures 1A - 1C shown, the transmission of the ink objects can be facilitated by the remote server 106. In other implementations, as Figure 2A and 2B shown, the ink objects can be transmitted in a peer-to-peer manner. Then, the process 200 can include deserializing the ink objects at stage 206 and rendering the deserialized ink objects as one or more images and / or animations. Example operations for deserializing ink objects are described below with reference to Figure 4C The example operations for generating serialized ink objects can include detecting pointer input via, for example, a touch screen or other suitable input device at stage 212, as

[0048] As Figure 4B shown. This operation can then include a decision stage for determining whether pointer input has been received. In response to determining that pointer input has been received, the operation proceeds to stage 216 to serialize the detected pointer input into one or more ink objects. Referring to Figures 1A - 1C , example implementations for serializing pointer input are described in more detail above. In response to determining that no pointer input has been received, the operation returns to detecting pointer input at stage 212.

[0049] As Figure 4C As shown, example operations for deserializing an ink object can include receiving a serialized ink object at stage 222. The operation can then proceed to deserialize the ink object at stage 224. In some embodiments, the deserialization can be performed when the ink object is received. In other embodiments, the deserialization can be performed during a waiting period. For example, the deserialization can be performed on ink objects received within the previous 40 milliseconds or other suitable time period. The operation can then include rendering one or more images and / or animations at stage 226 based on the deserialized ink object.

[0050] Figure 5 is a schematic diagram of an example hardware component of a computing device showing components of computing system 100 suitable for Figures 1A - 2B In a very basic configuration 202, computing device 102 can include one or more processors 204 and system memory 206. A memory bus 208 can be used for communication between processor 204 and system memory 206.

[0051] Depending on the desired configuration, processor 204 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. Processor 204 can include more levels of cache, such as a level 1 cache 210 and a level 2 cache 212, processor cores 214, and registers 216. Example processor cores 214 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can also be used with processor 204, or in some implementations, memory controller 218 can be an internal part of processor 204.

[0052] Depending on the desired configuration, system memory 206 can be of any type, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 can include an operating system 220, one or more applications 222, and program data 224. This described basic configuration 202 is shown in FIG. 2 by those components within the inner dashed line.

[0053] The computing device 102 may have additional features or functionality and additional interfaces to facilitate communication between the basic configuration 202 and any other devices and interfaces. For example, the bus / interface controller 230 may be used to facilitate communication between the basic configuration 202 and one or more data storage devices 232 via a storage device interface bus 234. The data storage devices 232 may be removable storage devices 236, non-removable storage devices 238, or a combination thereof. Examples of removable and non-removable storage devices include disk devices (such as floppy disk drives and hard disk drives (HDDs)), optical disc drives (such as compact disc (CD) drives or digital versatile disc (DVD) drives), solid state drives (SSDs), and tape drives, among others. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, or other data.

[0054] System memory 206, removable storage device 236, and non-removable storage device 238 are examples of computer readable storage media. Computer readable storage media includes storage hardware or devices, examples of which include but are not limited to RAM, ROM, EEPROM, flash memory, or other storage technologies, CD-ROM, digital versatile disc (DVD), or other optical storage devices, magnetic cassettes, tapes, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 102. Any such computer readable storage media may be part of the computing device 102. The term "computer readable storage media" does not include propagated signals and communication media.

[0055] The computing device 102 may also include an interface bus 240 to facilitate communication from various interface devices (e.g., output device 242, peripheral interface 244, and communication device 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 220, which may be configured to communicate with various external devices, such as a display or speakers, via one or more A / V ports 222. Example peripheral interfaces 244 include a serial interface controller 224 or a parallel interface controller 226, which may be configured to communicate with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I / O ports 228. Example communication devices 246 include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via one or more communication ports 264 over a network communication link.

[0056] A network communication link can be an example of a communication medium. A communication medium can generally be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal whose one or more characteristics are set or changed in a manner that encodes information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR), and other wireless media. The term computer-readable medium as used herein can include storage media and communication media.

[0057] Computing device 102 can be implemented as part of a small portable (or mobile) electronic device such as a cellular phone, a personal digital assistant (PDA), a personal media player device, a wireless network viewing device, a personal headphone device, a dedicated application device, or a hybrid device that includes any of the above functions. Computing device 102 can also be implemented as a personal computer including laptop and non-laptop computer configurations.

[0058] From the foregoing, it will be appreciated that specific embodiments of the present disclosure have been described herein for purposes of illustration, but that various modifications may be made without departing from the present disclosure. Additionally, many elements of one embodiment can be combined with other embodiments to supplement or replace elements of other embodiments. Accordingly, the technology is not limited by the appended claims.< / path> < / path> < / path> < / path> < / path> < / path> ​< / path> < / path>

Claims

1. A method for network-based collaborative ink writing between a first computer and a second computer interconnected via a remote server computer over a computer network, the method comprises: after detecting a pointer input to a web page in a first web browser at the first computer from a first user using a stylus, pen, mouse, touchpad or finger, serializing the detected pointer input sequence at the first computer into data representing one or more ink objects, the one or more ink objects individually representing at least a portion of an image corresponding to the detected pointer input at the first computer; transmitting the data representing the serialized ink objects together with a session ID and a session policy to the remote server computer, the session policy being related to accessing and / or updating the transmitted data representing the serialized ink objects, the remote server computer being configured to forward the data representing the serialized ink objects to the second computer via the computer network according to the session policy; and after receiving the transmitted data at the second computer, deserializing the received serialized ink objects; inserting programming code of the deserialized ink objects into a Document Object Model (DOM) of the web page in a second web browser on the second computer; and using the deserialized ink objects inserted into the DOM of the web page on the second computer to reconstruct the image corresponding to the detected pointer input at the first computer; and presenting the detected pointer input from the first user as one or more ink objects of the web page on a second web browser, thereby implementing a network-based collaborative session between the first user and the second user, the second web browser presenting the same web page on the second computer.

2. The method according to claim 1, wherein: the second computer includes a rendering context on the same web page in the second web browser; and presenting the image includes presenting the image corresponding to the detected pointer input on the first computer via the rendering context.

3. The method according to claim 1, wherein: serializing the detected pointer input includes generating the one or more ink objects according to a vector image format; and reconstructing the image includes reconstructing the image as an animation.

4. A method for network-based collaborative ink writing between a first computer and a second computer interconnected via a remote server computer over a computer network, the method comprises: after receiving a user input at the first computer to activate collaborative ink writing in a web browser presenting a web page, detecting a pointer input from the user to the web page in the web browser at the first computer; and in response to detecting the pointer input to the first computer, Serialize the pointer input detected at the web page into data representing one or more ink objects, where each of the one or more ink objects individually represents at least a portion of an image corresponding to the detected pointer input; Insert programming code for the one or more ink objects into the Document Object Model (DOM) of the web page; And Transmit the data representing the serialized one or more ink objects, along with a session ID and a session policy, related to accessing and / or updating the transmitted data representing the serialized ink objects, to the remote server computer, where the remote server computer is configured to forward the data representing the serialized ink objects to the second computer according to the session policy, and where the second computer is configured to deserialize the one or more ink objects to reconstruct and render on the second computer the detected pointer input on the first computer by inserting the programming code of the deserialized one or more ink objects into the DOM of the same web page on the second computer.

5. The method according to claim 4, wherein the second computer is configured to render on the first computer the image corresponding to the detected pointer input as an animation of a plurality of images.

6. The method according to claim 4, further comprising: After receiving, at the first computer, the user input to activate collaborative ink writing in a web browser, Insert a rendering context at the first computer onto the rendered web page in the web browser; And In response to detecting the pointer input to the first computer, via the inserted rendering context on the web page, render the one or more ink objects that individually represent at least a portion of the image corresponding to the detected pointer input.

7. The method according to claim 4, wherein: The second computer includes another rendering context, which is inserted onto a copy of the same web page in another web browser; and The second computer is configured to render on the first computer the image corresponding to the detected pointer input via the another rendering context.

8. The method according to claim 4, wherein: The remote server computer is further configured to: Store the received serialized ink objects in a storage device based on the session ID; And Upon request, transmit the stored serialized ink objects to the second computer via the computer network.

9. The method according to claim 4, wherein: The remote server computer is further configured to: Store the received serialized ink objects in a storage device based on the session ID; And Upon request, transmit the stored serialized ink objects to the second computer via the computer network according to the received session policy.

10. The method according to claim 4, wherein: The session policy indicates that the second computer cannot add or modify the serialized ink object from the first computer; and wherein the remote server computer is further configured to: store the received serialized ink object in a storage device based on the session ID; and after a request, transmit the stored serialized ink object to the second computer via the computer network, while preventing the second computer from adding or modifying the stored ink object in the storage device at the remote server computer.

11. The method according to claim 4, wherein: the session policy indicates that the second computer can add or modify the serialized ink object from the first computer; and wherein the remote server computer is further configured to: store the serialized ink object received from the first computer in a storage device based on the session ID; receive additional ink objects from the second computer; and store the additional ink objects received from the second computer in the storage device based on the session ID.

12. The method according to claim 4, further comprising: receiving additional ink objects from the second computer at the first computer; and in response to receiving the additional ink objects at the first computer deserializing the additional ink objects from the second computer; and rendering and presenting the deserialized additional ink objects as one or more images on the web page at the first computer.

13. The method according to claim 4, wherein transmitting the data representing the serialized ink object to the second computer comprises: the data representing the serialized ink object from the first computer via the computer network through a peer-to-peer connection to the second computer.

14. A computing device interconnected with another computing device via a remote server computer through a computer network, the computing device comprises: a processor; a display; and a memory operably coupled to the processor and the display, the memory containing instructions executable by the processor to cause the computing device to: receive pointer input from a user using a stylus, pen or finger on a web page presented in a web browser on the display of the computing device; and in response to receiving the pointer input, generate data representing a serialized ink object that individually corresponds to at least a portion of an image or animation of the pointer input received on the display; insert programming code of the serialized ink object into the Document Object Model (DOM) of the web page; and Transmit the data representing the serialized ink object, along with the session ID and session policy, to the remote server computer via the computer network, where the session policy is related to accessing and / or updating the data representing the serialized ink object, and the remote server computer is configured to forward the data representing the serialized ink object to the other computing device according to the session policy, wherein the serialized ink object allows the other computing device to deserialize the ink object to reconstruct and render, on the other computing device, the image or animation corresponding to the input pointer detected on the computing device by inserting the programming code of the deserialized ink object into the DOM of the same web page on the other computing device.

15. The computing device according to claim 14, wherein the memory includes additional instructions executable by the processor to cause the computing device to: Insert a rendering context onto the web page presented in the web browser; and In response to detecting the pointer input, render, via the rendering context inserted on the web page, the ink object that individually represents at least a portion of the image or animation corresponding to the detected pointer input.

16. The computing device according to claim 14, wherein: The other computing device includes another rendering context inserted onto a copy of the same web page inserted into another web browser; and The other computing device is configured to render, via the other rendering context, the image or animation corresponding to the detected pointer input on the computing device.

17. The computing device according to claim 14, wherein: The remote server computer is further configured to: Store the received serialized ink object in a storage device based on the session ID; and After a request, transmit the stored serialized ink object to the other computing device via the computer network.

18. The computing device according to claim 14, wherein: The remote server computer is further configured to: Store the received serialized ink object in a storage device based on the session ID; and After a request, transmit the stored serialized ink object to the other computing device and additional computing devices via the computer network.

19. The computing device according to claim 14, wherein: The session policy allows the remote server device to transmit the stored serialized ink object to the other computing device via the computer network according to the received session policy.

20. The computing device according to claim 14, wherein the memory includes additional instructions executable by the processor to cause the computing device to: Receive additional ink objects from the other computing device ； and In response to receiving the additional ink objects, Deserialize the additional ink objects received from the other computing device; and Render and present the deserialized additional ink objects as one or more images or animations on the web page in the web browser.

Images