Method and system for reducing latency on a collaborative platform - Patents.com

Abstract

A system and method are provided for reducing latency on a collaboration platform. The collaboration platform includes a display, a moderator device, one or more member devices, and a receiver in communication with the display, the moderator device, and the one or more member devices. To reduce latency on the collaboration platform, the receiver generates an overlay image based on user input received from the display and a user type of the user input, generates an overlaid image based on the overlay image, and transmits the overlaid image for display, while the collaboration application generates a new real image based on the user input for display.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 62 / 941,677, filed November 27, 2019, the entire contents of which are incorporated herein by reference.

[0002] SUMMARY The present disclosure generally relates to methods and systems for generating overlay images, for at least temporary display, based on user input to reduce latency on a collaborative platform. [Background technology]

[0003] Methods and products for projecting content both by wired connection and wirelessly via a network are well known in the art. One example is the Miracast® wireless display standard, certified by the Wi-Fi Alliance, which defines a protocol for displaying multimedia between devices using Wi-Fi CERTIFIED Wi-Fi Direct®. Implementing Wi-Fi Direct, Miracast® provides operations for negotiating video capabilities, setting content protection, streaming content, and maintaining video sessions. Unlike Bluetooth® technology, Wi-Fi CERTIFIED Miracast® allows for transmitting up to 1,080p HD or even higher resolution video, and is therefore suitable for video streaming and screen-to-screen content projection. For example, Miracast® allows for wireless streaming of video content from a laptop computer to a television display.

[0004] Undesirable latency in content projection systems occurs during collaboration, for example, when making edits to content being projected onto a display, e.g., a computing device, on which the original data file is not stored. For example, in a classroom setting, a teacher's desktop may have an original data file stored thereon, which may be projected onto a display at the front of the classroom, visible to students in the classroom, using a content projection system known in the art. A receiver is typically used to transmit data between the teacher's desktop or student's tablet and the display. For example, the receiver may be coupled to the display via a USB cable to transfer user input data, and further coupled to the display via an HDMI cable to transfer images. Additionally, the receiver may communicate with the teacher's desktop and student's tablet wirelessly over a network (e.g., a local network, a corporate network, or the Internet).

[0005] When an original file, e.g., a math problem stored on a teacher's desktop, is projected onto a display, e.g., a touch screen, a student may attempt to solve the math problem by drawing directly on the display. Input data representing the user input is transferred from the display to the receiver via a USB cable so that as the student begins to draw, e.g., the number "3" on the display, the formation of the number "3" begins to appear on the display. The receiver then transmits the user input data via WiFi to the teacher's desktop, where the original file is stored. A processor on the teacher's desktop then modifies the original file based on the user input data, e.g., adds the number "3" to the math problem as the student draws it, thereby generating a new real image, which is transmitted via WiFi to the receiver. The receiver then transmits the new real image to the display via an HDMI cable so that as the student draws it, the formation of the value "3" appears on the display. The data flow from the display, to the receiver, to the teacher's desktop, back to the receiver, and then finally back to the display occurs continuously as students draw on the display, introducing latency into the collaborative content projection system.

[0006] It is therefore desirable to provide a system and method for reducing latency in a collaborative content projection system. Summary of the Invention [Means for solving the problem]

[0007] The present invention is directed to a system and method for generating an overlay image based on a user input for at least temporary display, and reducing latency on a collaborative platform. For example, according to one aspect of the present invention, a method for reducing latency on a collaborative platform is provided. The method includes receiving a first real image by a first device, e.g., a receiver, from a third device, e.g., a moderator device, receiving user input data indicating the user input on a second device, e.g., a display, by the first device via a USB cable, transmitting the user input data to the third device by the first device, determining an overlay image based on the user input data by the first device, determining an overlaid image based on the overlay image and the first real image by the first device, transmitting the overlaid image to a second device, e.g., via an HDMI cable, and displaying the overlaid image on the second device, e.g., via a touch screen display, by the first device.

[0008] Further, a portion of the overlay image of the overlaid image may be displayed on the second device for a predetermined period of time. For example, the predetermined period of time may be at least as long as the latency on the collaborative platform. For example, the overlay image may include a leading edge and a trailing edge such that as the leading edge extends at a rate on the second device, the trailing edge is removed from the second device at that rate. Alternatively, as the number of spatial coordinates of the leading edge increases on the second device, a portion of the spatial coordinates of the trailing edge may be removed from the second device depending on the latency and / or speed of the user input data.

[0009] According to some aspects of the invention, the overlay image determined by the first device may include a first portion of the overlay image indicative of a user input at the second device based on the user input data, and an extended and predicted portion of the overlay image based on the user input data. For example, the first device may predict the extended portion of the overlay image based on at least one of spatial or temporal coordinates of the user input data, e.g., via at least one of extrapolation, machine learning, artificial intelligence, or neural networks. For example, the first device may predict the extended portion of the overlay image based on a velocity of the user input data. The extended portion of the overlay image may include a curved portion formed from a plurality of finite line segments, such that predicting the extended portion of the overlay image by the first device includes predicting the curved portion based on an angle of each finite line segment of the plurality of finite line segments.

[0010] In addition, the user input data may include input type data indicating at least one of thickness, color, or marker or eraser type. According to one aspect of the present invention, the method further includes determining, by the first device, an input type based on the user input data and machine learning. For example, the input type may be determined by analyzing a pattern of spatial input of the user input data from the second device. Thus, the determined overlay image may be determined based on the determined input type.

[0011] According to another aspect of the present invention, the method may further include receiving, by the first device, data indicative of the input type from the third device. For example, the first device may receive the data indicative of the input type from an application running on the third device via a defined TCP port. Alternatively, the first device may receive the data indicative of the input type from an operating system running on the third device via a User Input Back Channel (UIBC) extension. The third device and the first device may communicate via a wireless connection. The present specification also provides, for example, the following items: (Item 1) 1. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, the first image from the third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; and determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; A method comprising: (Item 2) 2. The method of claim 1, wherein determining, by the first device, the overlay image includes determining, by the first device, a first portion of the overlay image indicative of the user input on the second device based on the user input data. (Item 3) 3. The method of claim 2, wherein determining, by the first device, the overlay image includes predicting, by the first device, an extended portion of the overlay image based on the user input data. (Item 4) 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on at least one of spatial coordinates or temporal coordinates of the user input data. (Item 5) 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on a velocity of the user input data. (Item 6) 4. The method of claim 3, wherein the extended portion of the overlay image comprises a curved portion comprising a plurality of finite line segments, and predicting, by the first device, the extended portion of the overlay image includes predicting the curved portion based on an angle of each finite line segment of the plurality of finite line segments. (Item 7) 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on at least one of extrapolation, machine learning, artificial intelligence, or neural networks. (Item 8) 4. The method of claim 3, wherein determining the overlay image by the first device includes determining, by the first device, the overlay image comprising the first portion and an extended portion of the overlay image. (Item 9) Item 14. The method of item 1, wherein a portion of the overlay image is displayed on the second device for a predetermined period of time. (Item 10) 10. The method of claim 9, wherein the predetermined period is at least as long as a latency period on the collaborative platform. (Item 11) 2. The method of claim 1, wherein the overlay image comprises a leading end and a trailing end, such that as a number of leading end spatial coordinates increases on the second device, a portion of the trailing end spatial coordinates are removed from the second device depending on the length of latency on the collaborative platform. (Item 12) 2. The method of claim 1, wherein the overlay image comprises a maximum amount of spatial coordinates such that when additional spatial coordinates beyond the maximum amount of spatial coordinates are displayed, initial display spatial coordinates are removed from the overlay image. (Item 13) Item 10. The method of item 1, wherein the overlay image comprises a leading edge, a trailing edge, and a maximum spatial length such that as the leading edge extends, the trailing edge is removed to maintain the maximum spatial length of the overlay image in the overlaid image displayed on the second device. (Item 14) Item 10. The method of item 1, wherein the overlay image comprises a leading end and a trailing end, such that as the leading end extends onto the second device at a rate, the trailing end is removed from the second device at the rate. (Item 15) 2. The method of claim 1, wherein the overlay image comprises a leading end and a trailing end such that as a number of spatial coordinates of the leading end increase on the second device, a portion of the spatial coordinates of the trailing end are removed from the second device depending on the rate at which the spatial coordinates increase. (Item 16) 2. The method of claim 1, further comprising determining, by the first device, an input type corresponding to the user input on the second device, the input type comprising at least one of a thickness, a color, or a marker or eraser type. (Item 17) Item 17. The method of item 16, wherein determining, by the first device, the input type includes determining, by the first device, the input type based on the user input data and machine learning. (Item 18) 2. The method of claim 1, further comprising receiving, by the first device, data from the third device indicating an input type corresponding to the user input. (Item 19) Item 19. The method of item 18, wherein receiving data indicative of the input type by the first device includes receiving, by the first device, data indicative of the input type from an application running on the third device via a defined TCP port. (Item 20) Item 19. The method of item 18, wherein receiving, by the first device, data indicative of the input type includes receiving, by the first device, data indicative of the input type from an operating system running on the third device via a User Input Back Channel (UIBC) extension. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 1A is a block diagram of a collaboration platform in accordance with an illustrative embodiment of the invention.

[0013] [Figure 1B] FIG. 1B is a block diagram of the collaborative platform of FIG. 1A illustrating various communication mechanisms in accordance with the principles of the present invention.

[0014] [Diagram 2] FIG. 2 is a schematic diagram of a collaboration platform in an exemplary configuration, according to one aspect of the present invention.

[0015] [Figure 3A]3A-3D are schematic diagrams of example hardware and software components of an example display, receiver, moderator device, and member device, respectively. [Figure 3B] 3A-3D are schematic diagrams of example hardware and software components of an example display, receiver, moderator device, and member device, respectively. [Figure 3C] 3A-3D are schematic diagrams of example hardware and software components of an example display, receiver, moderator device, and member device, respectively. [Figure 3D] 3A-3D are schematic diagrams of example hardware and software components of an example display, receiver, moderator device, and member device, respectively.

[0016] [Figure 4A] FIG. 4A is a block diagram of a collaboration platform according to one aspect of the present invention.

[0017] [Figure 4B] FIG. 4B is a sequence diagram for using the collaborative platform according to the illustrative embodiment depicted in FIG. 4A.

[0018] [Figure 5A] FIG. 5A is a flow chart illustrating exemplary steps for reducing latency on a collaborative platform in accordance with the principles of the present invention.

[0019] [Figure 5B] FIG. 5B is a flow chart illustrating steps of generating the overlaid image of FIG. 5A.

[0020] [Figure 5C] FIG. 5C illustrates overlaid image generation in accordance with the principles of the present invention.

[0021] [Figure 6] 6A-6E illustrate steps for reducing latency on a collaborative platform in accordance with the principles of the present invention.

[0022] [Figure 7-1] 7A-7D illustrate overlay image prediction generation in accordance with the principles of the present invention. [Figure 7-2] 7A-7D illustrate overlay image prediction generation in accordance with the principles of the present invention.

[0023] [Figure 8] 8A and 8B illustrate user type data collection according to one aspect of the present invention.

[0024] [Figure 9A] FIG. 9A is a block diagram of an alternative embodiment of a collaborative platform in accordance with another aspect of the present invention.

[0025] [Figure 9B] FIG. 9B is a sequence diagram for using the collaborative platform according to the illustrative embodiment depicted in FIG. 9A.

[0026] [Figure 10A] FIG. 10A is a block diagram of another alternative embodiment of a collaborative platform in accordance with yet another aspect of the present invention.

[0027] [Figure 10B] FIG. 10B is a sequence diagram for using the collaborative platform according to the illustrative embodiment depicted in FIG. 10A.

[0028] [Figure 11] FIG. 11 is a flow chart illustrating alternative exemplary steps for reducing latency on a collaborative platform in accordance with the principles of the present invention.

[0029] [Figure 12A] FIG. 12A is a block diagram of yet another alternative embodiment of a collaborative platform according to yet another aspect of the present invention.

[0030] [Figure 12B] FIG. 12B is a sequence diagram for using the collaborative platform according to the illustrative embodiment depicted in FIG. 12A.

[0031] The foregoing and other features of the present invention will become apparent from the following description and the appended claims, taken in conjunction with the accompanying drawings. It is to be understood that these drawings depict only several embodiments in accordance with the present disclosure and are, therefore, not to be considered limiting of its scope, and that the present disclosure will be described with additional specificity and detail through the use of the accompanying drawings.

DETAILED DESCRIPTION OF THE INVENTION

[0032] Detailed Description of the Invention There are many instances where a computer user may desire to share the display of their computer screen with others. For example, in relation to a class being given in a classroom setting, a teacher may desire to display a problem to the students in the classroom and have the students solve the problem on the display so that the students' work is visible to the entire classroom. For example, the problem may be stored in a computer file on the teacher's computer and displayed on a main display visible to the students in the classroom. A selected student may then perform the work directly on the main display so that the work is visible to the students in the classroom. In such an instance, it may be advantageous to display an overlay image that quickly and easily inserts the student's work over the original problem on the main display. As will be understood by those skilled in the art, the principles of the present invention described herein may be used remotely across a campus or other geographical location via WiFi or the Internet for other collaborative endeavors, such as meetings or presentations, outside of a classroom setting.

[0033] The present invention is directed to a collaboration platform that is used to facilitate reducing latency in the collaboration platform while presenting materials in real-time, for example, in a classroom setting or product presentation meeting. For example, the present invention allows a user to provide user input, such as markings, onto a displayed original image such that the user input is insertively overlaid onto the original image on the main display almost immediately after the user input is provided, and before an actual image can be generated by the collaboration platform. The collaboration platform involves a main display, a moderator device, one or more member devices, and a receiver that communicates with the display, the moderator device, and the one or more member devices. The moderator device may be used by a teacher / administrator and may store an original data file, and an original image may be displayed on the main display based on the original data file such that a student may edit the original data file by providing user input via the main display. The receiver is configured to launch an overlay image generation application, which generates an overlay image based on user input provided by the student via the display, and displays the overlay image over the original image, while the collaborative platform updates the original data file based on the user input data for display on the main display. By displaying the overlay image before displaying the updated image generated using the actual data, the receiver reduces latency within the collaborative platform.

[0034] FIG. 1A is a block diagram of an illustrative collaboration platform constructed in accordance with the principles of the present invention. Collaboration platform 100 includes a display 105, a receiver 120 and a network 101 in which receiver 120 serves as a hub, a moderator client, e.g., a moderator device 130 for use by a teacher, and, optionally, one or more member clients, e.g., one or more member devices 140 for use by students. Receiver 120 may be a Screen Beam® wireless display kit available from Actiontec Electronics, Inc. (Sunnyvale, Calif). In one preferred embodiment, receiver 120 is Miracast®-enabled and compatible. In FIG. 1A, three member devices 140 are depicted, but as one skilled in the art would understand, fewer or more than three member devices may be used within collaboration platform 100.

[0035] As shown in FIG. 1A, the moderator device 130 and the member device 140 interact with the receiver 120 wirelessly through the network 101. As shown in FIG. 1B, the network 101 may be based on wireless communication such that the moderator device 130 and the member device 140 interact with the receiver 120 via WiFi or the Internet. The network 101 may be a local peer-to-peer network, e.g., a Wi-Fi peer-to-peer interface. The display 105 may be any suitable computing device, e.g., a touch screen device, providing an interface for presenting information received from the receiver 120 to an external system, a user, or a memory, as well as for collecting user input directly through the interface of the display 105, e.g., through a touch sensor built into the interface. In alternative embodiments, the display 105 may comprise multiple individual displays and may further comprise a display associated with each of the member device 140 and / or the moderator device 130. Similarly, when the user interacts directly on the screen of the member device 140 to make edits, the member device 130 may be any suitable computing device, e.g., a touch screen device, as described above.

[0036] The receiver 120 may be coupled to the display 105 by one or more wired connections. For example, as shown in FIG. 1B, the receiver 120 and the display 105 may connect using a Universal Serial Bus (USB) cable to communicate user input data, and the receiver 120 and the display 105 may connect using a High-Definition Multimedia Interface (HDMI) cable to communicate images. Alternatively, the receiver 120 and the display 105 may connect using a wireless connection such as Bluetooth. Thus, the receiver 120 receives an original image, e.g., a still image, from the moderator device 130 via WiFi, and conveys the original image provided by the moderator device 130 to the display 105 via an HDMI cable, which is shown in parentheses on the display 105. Thus, the local display of the moderator device 130 and the display 105 may display the same information (e.g., the same graphics, videos, images, charts, presentations, documents, programs, applications, windows, views, etc.). In addition, the receiver 120 receives user input data indicative of user input from the display 105 via a USB cable and / or a wireless connection such as Bluetooth, and conveys the user input data provided by the display 105 via the WiFi display 105 to the moderator device 130 for processing.

[0037] The moderator device 130 processes the user input data provided by the display 105, modifies the original image stored in its memory based on the received user input data, and generates an image for redistribution to the receiver 120 via WiFi and ultimately to the display 105 via the receiver 120. As would be understood by one skilled in the art, the data flow paths, i.e., user input data from the display 105 to the receiver 120 via USB and / or Bluetooth, user input data from the receiver 120 to the moderator device 130 via WiFi, generation of a real image based on the user input data by the moderator device 130, real image from the moderator device 130 to the receiver 120 via WiFi, and real image from the receiver 120 to the display 105 via HDMI, will suffer from time delays due to the latency of the content projection system.

[0038] According to one aspect of the present invention, the moderator device 130 may designate a member device 140 as a moderator, as described in U.S. Patent Application Serial No. 14 / 986,468, the entire contents of which are incorporated herein by reference. Thus, the moderator device 130 may choose to share the screen of the member device 140 on the display 105, such that user input provided by the user on the display 105 will be transmitted to the member device 140 and modify the original file stored in the memory of the member device 140.

[0039] In accordance with another aspect of the invention, receiver 120 may be incorporated into moderator device 130. For example, receiver 120 may be incorporated into a laptop that serves as moderator device 130. In accordance with another aspect of the invention, any suitable arrangement of receiver 120 and display 105 may be employed. For example, receiver 120 and display 105 may be separate components or may be combined into a single device.

[0040] Figure 2 depicts an embodiment of a collaborative platform 100 configured according to the principles of the present invention for use in a classroom setting. As shown in Figure 2, the main display 105 is visible to the students in the classroom and includes an input / output device 110, such as a touch screen, so that a student can directly provide user input to the display 105 that communicates with the receiver 120. According to another aspect of the present invention, a student can provide user input to a member device 140 via an input / output device 145 that communicates directly with the receiver 120, which will then be displayed on the display 105. As shown in Figure 2, the teacher's desktop computer is designated as a moderator device 130 having an input / output device 135, such as a touch screen, while the wireless tablet located at each student's desk serves as a member device 140 having an input / output device 145, such as a touch screen. As described above, the moderator device 130 and the member device 140 communicate wirelessly with the receiver 120.

[0041] According to another aspect of the present invention, the collaborative platform 100 may be used across multiple classrooms and / or other collaborative work environment settings. For example, the moderator device 130 may be in a first classroom having a first display and a first plurality of member devices, and the moderator device 130 may communicate, for example, via WiFi, with a second display and a second plurality of member devices in a second classroom. Thus, students in the second classroom may modify an image displayed on the second display such that the modifications to the image are visible on the first and second displays in the first and second classrooms, thereby modifying the original file stored on the moderator device 130 in the first classroom.

[0042] 3A-3D, exemplary functional blocks are provided that respectively represent hardware and software components of the display 105, the receiver 120, the moderator device 130, and the member device 140. Referring now to FIG. 3A, the hardware and software components of the display 105 may include a processing unit 106, a memory 107, a storage device 111, a communication unit 108, a power supply 109, and an input / output (I / O) device 110.

[0043] The processing unit 106 may be one or more processors configured to run the operating system 112 and perform the tasks and operations of the display 105 described herein. The memory 107 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read only memory (ROM)), flash memory, or any combination thereof. The communication unit 108 may be any known communication infrastructure facilitating communication via any known wired or wireless connection. For example, the communication unit 108 may transmit information, e.g., user input data, to the receiver 120 of the collaborative platform 100 via a wireless connection, such as a USB cable and / or Bluetooth®, and may receive information, e.g., images, from the receiver 120 via an HDMI® cable. The power source 109 may be a battery or may connect the display 105 to a wall outlet or any other external power source. The storage device 111 may include, but is not limited to, removable and / or non-removable storage devices, such as, for example, magnetic disks, optical disks, or tapes.

[0044] The input device of the I / O device 110 may be one or more devices coupled to or incorporated within the display 105 for inputting data into the display 105. For example, the input device of the I / O device 110 may be a touch input device (e.g., a touchpad or touchscreen) or an array of positioning sensors configured to receive user input from a user and generate user input data indicative of the user input. In addition, the input device of the I / O device 110 may be in conjunction with a smart stylet that interacts with the array of positioning sensors. The output device of the I / O device 110 may be any device coupled to or incorporated within the display 105 for outputting or otherwise displaying an image. Thus, the I / O device 110 may be a touchscreen for receiving and displaying an image.

[0045] An operating system 112 may be stored in the storage device 111 and executed on the processing unit 106. The operating system 112 may be suitable for controlling the general operation of the display 105 to achieve the functionality of the display 105 described herein. The display 105 may also optionally launch a graphics library, other operating systems, and / or any other application programs. Of course, it should be understood that the display 105 may include additional or fewer components than those illustrated in FIG. 3A and may include more than one of each type of component.

[0046] Now referring to FIG. 3B, the hardware and software components of the receiver 120 may include a processing unit 121, a memory 122, a storage device 126, a communication unit 123, a power supply 124, and an input / output (I / O) device 125.

[0047] The processing unit 121 may be one or more processors configured to run the operating system 127, collaboration application 128, and overlay image generator application 129 and perform the tasks and operations of the receiver 120 described herein. The memory 122 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read only memory (ROM)), flash memory, or any combination thereof. The communication unit 123 may be any known communication infrastructure that facilitates communication via any known wired or wireless connection. For example, the communication unit 123 may receive information, e.g., user input data, from the display 105 via a wireless connection such as a USB cable and / or Bluetooth®, and real images from the moderator device 130 via WiFi, and may transmit information, e.g., images, to the display 105 via an HDMI® cable. Additionally, the communication unit 123 may communicate both the user input data and the images to the moderator device 130 and / or the member device 140 via the network 101, e.g., WiFi. According to one aspect of the invention, the communication unit 123 may receive information, e.g., data indicative of one or more user types of user input, from the moderator device 130, e.g., via a defined TCP port or UIBC extension.

[0048] The power source 124 may be a battery or may connect the receiver 120 to a wall outlet or any other external power source. The storage device 126 may include removable and / or non-removable storage devices, such as, but not limited to, magnetic disks, optical disks, or tape. The input devices of the I / O device 125 may be one or more devices coupled to or incorporated within the receiver 120 for inputting data to the receiver 120. The output devices of the I / O device 110 may be any device coupled to or incorporated within the receiver 120 for outputting or otherwise displaying images.

[0049] The collaboration application 128 may be stored in the storage device 126 and executed on the processing unit 121. The collaboration application 128 may be a software application and / or a software module having one or more sets of instructions suitable for performing the operations of the receiver 120 described herein, including facilitating the exchange of information with the moderator device 130. For example, the collaboration application 128 may cause the receiver 120 to receive user input data from the display 105 via the communication unit 123, e.g., via a USB cable and / or a wireless connection such as Bluetooth, and to communicate the user input data to the moderator device 130 via the communication unit 123, e.g., via WiFi. In addition, the collaboration application 128 may further cause the receiver device 130 to receive a real image from the moderator device 130 via the communication unit 123, e.g., via WiFi, and to communicate an image overlaid based on the real image to the display 105, e.g., via an HDMI cable. According to another aspect of the present invention, the collaborative application 128 may cause the receiver 120 to receive data indicative of one or more user types from the moderator device 130 via a communication unit 123, e.g., a defined TCP port or a modified User Input Back Channel (UIBC), as described in further detail below.

[0050] The overlay image generator application 129 may be stored in the storage device 126 and executed on the processing unit 121. The overlay image generator application 129 may be a software application and / or software module having one or more sets of instructions suitable for performing the operations of the receiver 120 described herein, including facilitating the exchange of information with the display 105, the moderator device 130, and the member device 140. For example, the overlay image generator application 129 may cause the processing unit 121 of the receiver 120 to process and analyze user input data received from the display 105 via the collaboration application 128, generate an overlay image based on the user input data, generate an overlaid image based on the overlay image, and transmit the overlaid image to the display 105 via the communication unit 123, e.g., via an HDMI® cable, for display. In addition, the overlay image generator application 129 may cause the receiver 120 to derive one or more user types based on user input data received from the display 105 via the collaborative application 128, such that an overlay image is also generated based on the user type, as described in further detail below.

[0051] Alternatively, the overlay image generator application 129 may cause the receiver 120 to generate the overlay image based on data indicative of one or more user types received from the moderator device 130 via the communication unit 123, e.g., a defined TCP port, instead of deriving one or more user types based on user input data received from the display 105, as described in further detail below. According to another embodiment of the invention, the overlay image generator application 129 may cause the receiver 120 to generate the overlay image based on data indicative of one or more user types received from the moderator device 130 via the communication unit 123, e.g., a modified user input back channel (UIBC), instead of deriving one or more user types based on user input data received from the display 105, as described in further detail below.

[0052] An operating system 127 may be stored in the memory device 126 and executed on the processing unit 121. The operating system 127 may be suitable for controlling the general operation of the receiver 120 and may interface with an overlay image generator application 129 to achieve the functionality of the receiver 120 described herein. The receiver 120 may also optionally launch a graphics library, other operating systems, and / or any other application programs. Of course, it should be understood that the receiver 120 may include additional or fewer components than those illustrated in FIG. 3B and may include more than one of each type of component.

[0053] Now referring to FIG. 3C, the hardware and software components of the moderator device 130 may include a processing unit 131, a memory 132, a storage device 136, a communication unit 133, a power supply 134, and an input / output (I / O) device 135.

[0054] The processing unit 131 may be one or more processors configured to run the operating system 137, collaboration application 138, and optional overlay image application 139 and perform the tasks and operations of the moderator device 130 described herein. The memory 132 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read only memory (ROM)), flash memory, or any combination thereof. The communication unit 133 may be any known communication infrastructure that facilitates communication via any known wired or wireless connection. For example, the communication unit 133 may receive information, e.g., user input data, from the receiver 120 via WiFi and may transmit information, e.g., images, to the receiver 120 via WiFi. The power source 134 may be a battery or may connect the moderator device 130 to a wall outlet or any other external power source. The storage device 136 may include, but is not limited to, removable and / or non-removable storage devices, such as, for example, magnetic disks, optical disks, or tapes.

[0055] The input device of the I / O device 135 may be one or more devices coupled to or incorporated within the moderator device 130 for inputting data into the moderator device 130. For example, the input device of the I / O device 135 may be a touch input device (e.g., a touchpad or touchscreen) or an array of location sensors configured to receive user input from a user and generate user input data indicative of the user input. In addition, the input device of the I / O device 135 may be in conjunction with a smart stylet that interacts with the array of location sensors. The output device of the I / O device 135 may be any device coupled to or incorporated within the moderator device 130 for outputting or otherwise displaying images. Thus, the I / O device 135 may be a touchscreen for receiving and displaying images.

[0056] The collaboration application 138 may be stored in the storage device 136 and executed on the processing unit 131. The collaboration application 138 may be a software application and / or software module having one or more sets of instructions suitable for performing the operations of the moderator device 130 described herein, including facilitating the exchange of information with the receiver 120. For example, the collaboration application 138 may cause the moderator device 130 to transmit, via the communication unit 133, e.g., via WiFi, a first actual image from an original image file stored on the storage device 136 to the receiver 120 for display via the display 105. Furthermore, the collaboration application 138 may cause the moderator device 130 to receive user input data from the receiver 120 via the communication unit 133, e.g., via WiFi. The collaborative application 138 may further cause the processing unit 131 to process and analyze the user input data received from the receiver 120, modify the original image file stored on the storage device 136 by generating a real image based on the user input data, and store the real image on the storage device 136. In addition, the collaborative application 138 may cause the moderator device 130 to transmit the real image, e.g., the real image stored on the storage device 136, via the communication unit 133, e.g., via WiFi, to the receiver 120 for display via the display 105.

[0057] An optional overlay image application 139 may be stored in the storage device 136 and executed on the processing unit 131. The overlay image application 139 may be a software application and / or software module having one or more sets of instructions suitable for performing the operations of the moderator device 130 described herein, including facilitating the exchange of information with the receiver 120. For example, the overlay image application 139 may cause the processing unit 131 of the moderator device 130 to derive user type data indicative of one or more user types from user input data received by the moderator device 130 through the collaboration application 138, and transmit the user type data to the receiver 120 via the communication unit 133, for example, via a defined TCP port.

[0058] An operating system 137 may be stored in the storage device 136 and executed on the processing unit 131. The operating system 137 may be suitable for controlling the general operation of the moderator device 130 and may work with collaboration applications 138 and optional overlay image applications 139 to achieve the functionality of the moderator device 130 described herein. The moderator device 130 may also optionally launch a graphics library, other operating systems, and / or any other application programs. Of course, it should be understood that the moderator device 130 may include additional or fewer components than those illustrated in FIG. 3C and may include more than one of each type of component. According to one embodiment of the present invention, the operating system 137 may cause the processing unit 131 of the moderator device 130 to derive user type data indicative of one or more user types from user input received by the moderator device 130 through the collaborative application 138 via the communication unit 133, for example via a modified user input back channel (UIBC), and transmit the user type data to the receiver 120.

[0059] Now referring to FIG. 3D, the hardware and software components of one or more member devices 140 may include a processing unit 141, a memory 142, a storage device 146, a communication unit 143, a power supply 144, and an input / output (I / O) device 145.

[0060] The processing unit 141 may be one or more processors configured to run the operating system 147, collaboration application 148, and optional overlay image application 149 and perform the tasks and operations of the member device 140 described herein. The memory 142 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read only memory (ROM)), flash memory, or any combination thereof. The communication unit 143 may be any known communication infrastructure that facilitates communication via any known wired or wireless connection. For example, the communication unit 143 may transmit information, e.g., user input data, to the receiver 120 of the collaboration platform 100 via WiFi and receive information, e.g., images, from the receiver 120 via WiFi. The power source 144 may be a battery or may connect the member device 140 to a wall outlet or any other external power source. The storage device 146 may include, but is not limited to, removable and / or non-removable storage devices, such as, for example, magnetic disks, optical disks, or tapes.

[0061] The input device of the I / O device 145 may be one or more devices coupled to or incorporated within the member device 140 for inputting data into the member device 140. For example, the input device of the I / O device 145 may be a touch input device (e.g., a touch pad or touch screen) or an array of location sensors configured to receive user input from a user and generate user input data indicative of the user input. In addition, the input device of the I / O device 145 may be in conjunction with a smart stylet that interacts with the array of location sensors. The output device of the I / O device 145 may be any device coupled to or incorporated within the member device 140 for outputting or otherwise displaying images. Thus, the I / O device 145 may be a touch screen for receiving and displaying images.

[0062] The collaboration application 148 may be stored in the storage device 146 and executed on the processing unit 141. The collaboration application 148 may be a software application and / or software module having one or more sets of instructions suitable for performing the operations of the member device 140 described herein, including facilitating the exchange of information with the receiver 120. For example, the collaboration application 148 may cause the member device 140 to transmit user input data received via an input device of the I / O device 145 to the receiver 120 via the communication unit 143, e.g., via WiFi, for further transmission to the moderator device 130. Furthermore, the collaboration application 148 may cause the member device 140 to receive images from the receiver 120 via the communication unit 133, e.g., via WiFi, for display via an output device of the I / O device 145.

[0063] The optional overlay image application 149 may be stored in the storage device 146 and executed on the processing unit 141. The overlay image application 149 may be a software application and / or software module having one or more sets of instructions suitable for performing the operations of the member device 140 described herein, including facilitating the exchange of information with the receiver 120. When the member device 140 is labeled as a moderator by the moderator device 130, as described above, the overlay image application 149 may operate similarly to the overlay image application 139.

[0064] An operating system 147 may be stored in the storage device 146 and executed on the processing unit 141. The operating system 147 may be suitable for controlling the general operation of the member device 140 and may work with a collaboration application 148 and an optional overlay image application 149 to achieve the functionality of the member device 140 described herein. The member device 140 may also optionally run a graphics library, other operating systems, and / or any other application programs. Of course, it should be understood that the member device 140 may include additional or fewer components than those illustrated in FIG. 3D and may include more than one of each type of component.

[0065] 4A, a block diagram of an exemplary embodiment of a collaboration platform 100 according to the principles of the present invention is provided. As shown in FIG. 4A, user input data may be transmitted from the display 105 to the receiver 120 via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth®. In addition, the user input data and the real image may be communicated between the receiver 120 and the moderator device 130 across a wireless connection, e.g., WiFi. Furthermore, an image overlaid based on the real image and the overlay image may be transmitted from the receiver 120 to the display 105 via a wired connection, e.g., an HDMI® cable.

[0066] Now referring to FIG. 4B, a sequence diagram for using the collaboration platform 100 depicted in FIG. 4A is provided. As described above, the collaboration platform 100 may launch a collaboration application, such as Microsoft Whiteboard available from Microsoft (Redmond, WA) or a third party application such as Google Drive available from Google LLC (Mountain View, Calif.), to display a first real image based on an original image file stored on the moderator device 130, receive user input, modify the original image file stored on the moderator device 130 based on the user input, and display a second real image based on the modified original image. Specifically, as shown in FIG. 4B, a user may provide user input directly to the display 105, such as a touch screen. The first real image may be a math problem already displayed on the display 105, such as from an original image file stored on the moderator device 130, or the display 105 may be blank if the original image file stored on the moderator device 130 is blank at first. The user input may be, for example, a pattern of interaction (e.g., clicking and dragging) with the touch screen of the display 105 that forms the number "3" in red. The shapes that form the number "3" are examples of user inputs, and the color red is an example of a user type of user input. Other possible user types may include, for example, different colors (e.g., gray, black, red, blue, etc.), thickness levels (e.g., thin, medium, thick), or marker or eraser types, etc.

[0067] User input data based on the user input received by the display 105 is then transmitted to the receiver 120 via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth, which then transmits the user input data to the moderator device 130 via a wireless connection, e.g., WiFi. Upon launching the collaboration application, the moderator device 130 modifies the original image file stored in its memory based on the user input data and generates a real image file corresponding to a real image, e.g., a red "3" is superimposed on the math problem. Typically, the real image is then transmitted to the receiver 120 via a wireless connection, e.g., WiFi, which then transmits the real image to the display 105 via a wired connection, e.g., an HDMI cable, to be displayed. Thus, there is an undesirable delay between the time the user provides the user input to the display 105 and when the real image reaches the display 105, i.e., when the red "3" begins to appear on the display 105. As will be understood by one of ordinary skill in the art, the collaborative platform does not wait, for example, until the entire number "3" is drawn before generating an actual image; instead, this process occurs continuously as the user draws the number "3."

[0068] In accordance with the principles of the present invention, the collaboration platform 100 may launch an overlay image generator application to generate an overlay image based on user input provided by the user by the receiver 120, generate an overlayed image based on the overlay image and the actual image received by the moderator device 130, display the overlayed image on the original image on the display 105, and reduce latency of the collaboration platform 100.

[0069] Specifically, as shown in Figures 4A and 4B, the receiver 120 may generate an overlay image based on the user input data, generate an overlay image based on the overlay image and the actual image received from the moderator device 130, and transmit the overlay image to the display 105 via a wired connection, e.g., an HDMI cable, to be displayed over the original image displayed on the display 105, thereby reducing the latency of the collaboration platform 100. In addition, the receiver 120 may determine a user type of the user input by deriving data indicative of the user type from the user input data received from the display 105, e.g., using machine learning, artificial intelligence, or neural networks, as described in further detail below with respect to Figures 7A and 7B. Thus, as the receiver 120 determines the user type, it may generate an overlay image based on both the user input data and the user type.

[0070] 5A, a flow chart is illustrated detailing the data flow and decisions made in implementing the overlaid image generation functionality of receiver 120 of collaboration platform 100. As discussed above, receiver 120 of collaboration platform 100 may be used to generate an overlay image based on user input, and to generate an overlay image based on the overlay image and an actual image received from moderator device 130 such that the overlay image is displayed, thereby reducing latency of collaboration platform 100.

[0071] To begin the process described in FIG. 5A, in step 500, an original image is received by the receiver 120. For example, the original image may be received from the moderator device 130 and may include, for example, a blank screen, a math problem, a photo, etc. In step 501, the receiver 120 sets the original image received from the moderator device 130 as the current image. This may involve decoding the original image and / or placing the original image in a buffer. In step 502, user input data indicative of a user input may be received by the receiver 120 from the display 105, for example, via a USB cable and / or a wireless connection such as Bluetooth. Preferably, the user type of the user input may be optionally set to pre-programmed default settings, for example, a default color (gray), a default thickness (medium line), and a default marker user type, until changed by the user, as described with respect to steps 504-506. If the receiver 120 receives user input data from the display 105 in step 502, the process may proceed to step 503. If the receiver 120 does not receive user input data from the display 105 in step 502, the process may proceed directly to step 508, which is described in more detail below.

[0072] In step 503, the receiver 120 launches a collaboration application to transmit the user input data to the source of the original image, e.g., the moderator device 130, for further processing and analysis. As described above, the moderator device 130 generates a real image based on the user input data received from the receiver 120. In addition, the receiver 120 launches an overlay image generation application to generate an overlay image based on the user input data for immediate display.

[0073] Optionally, in step 504, receiver 120 analyzes the user input data received from display 105 in step 502 to determine whether at least one user type has changed. For example, receiver 120 may compare the spatial location of the user input on display 105 as well as physical contact with display 105 at various times to determine whether the user selected a different user type, e.g., using machine learning, artificial intelligence, or neural networks. If receiver 120 determines that a different user type has not been selected, e.g., the user has not clicked on a different user type icon, in step 505, receiver 120 will continue to use the previous user type, e.g., gray. If receiver 120 determines that a different user type has been selected, e.g., the user selected red, based on the spatial location of the user input and the fact that the user broke contact with display 105 and re-contacted display 105 at that specific spatial location on display 105, in step 506, receiver 120 selects a new user type, e.g., red.

[0074] In step 507, receiver 120 generates a leading edge of an overlay image based on the user input data received in step 502 and the user type selected in step 505 or 506, if a user type was selected in step 505 or 506, as described in further detail with respect to FIG. 5B. Preferably, the overlay image may be generated based on the user input data and a default user type, e.g., default color and / or default line thickness, and thus step 507 may be initiated after step 503, without steps 504-506. The generated overlay image represents the user's actual input provided by the user and may further include predicted user input based on the user's actual input.

[0075] For example, as shown in FIG. 5B, to generate an overlay image based on user input data, optionally user type, in step 511, receiver 120 generates a first portion of an overall overlay image, which represents the user's actual input received by receiver 120 from display 105, e.g., via a USB cable and / or a wireless connection such as Bluetooth. Thus, the first portion of the overlay image, when displayed on display 105 as an overlaid image, will insert what the user actually input on display 105. In step 512, receiver 120 generates a second extended portion of the overall overlay image, which may be a prediction of the user's intended input based on user input data received by receiver 120 from display 105, e.g., via a USB cable and / or a wireless connection such as Bluetooth. For example, using, e.g., extrapolation, machine learning, artificial intelligence, and / or neural networks, receiver 120 may analyze spatial and / or temporal coordinates of the user's input from the user input data and predict the user's intended input, e.g., what will be the user's next input, as described in further detail below. In step 513, receiver 120 generates an overlay image based on the first portion and the second extended portion of the overlay image such that the overlay image will include what the user actually input on display 105 and what the user is predicted to input on display 105.

[0076] Referring again to FIG. 5A, in step 508, receiver 120 may remove a portion of the trailing edge of the overlay image as receiver 120 generates the leading edge of the overlay image. For example, a portion of the overlay image of the overlay image displayed on display 105 may be removed as a function of time or as a function of the spatial amount of the overlay image of the overlay image displayed on display 105 at a given time. For example, each spatial coordinate of the overlay image of the overlay image displayed on display 105 may remain displayed for a predetermined amount of time, e.g., 100-300 milliseconds or more. Thus, each spatial coordinate that makes up the overlay image of the overlay image on display 105 may remain on display 105 for the same amount of time and may be removed after that time has elapsed. Each spatial coordinate of the overlay image is initially displayed on the display 105 at the leading edge of the overlay image of the overlaid image, and as time passes and additional spatial coordinates are displayed, the initial leading edge spatial coordinate terminates at the trailing edge of the overlay image of the overlaid image after, for example, a predetermined amount of time has passed before it is removed. For example, the predetermined amount of time that each spatial coordinate may be displayed may be at least as long as the waiting period for the actual image to be received by and appear thereon. Thus, for a given amount of spatial coordinates displayed on the display 105 over a predetermined period of time, for example, the same amount of spatial coordinates will be removed from the display 105 within the same predetermined period of time.

[0077] According to another aspect of the invention, the portion of the overlay image of the overlaid image displayed on the display 105 may have a maximum spatial distribution, e.g., length and / or amount of spatial coordinates between a leading edge and a trailing edge of the overlay image of the overlaid image, over a given amount of time. Thus, after the spatial coordinates of the overlay image of the overlaid image are initially displayed on the display 105, a predetermined amount of additional spatial coordinates are displayed, such that the initial spatial coordinates are now at the trailing edge of the overlay image of the overlaid image, and the initial spatial coordinates of the overlay image of the overlaid image will be removed from the display 105 when the amount of additional spatial coordinates displayed on the display 105 exceeds a predetermined maximum amount of spatial coordinates allowed on the display 105.

[0078] Thus, if the receiver 120 does not receive user input data from the display 105 in step 502, then in step 508, additional leading edge will not be added to the overlay image such that, for example, when the user removes their stylet / finger from the display 105, the overlay image of the overlaid image displayed on the display 105 is completely replaced by the current image, or additional user input is not provided to the display 105 until, in step 502, additional user input is received from the display 105 by the receiver 120, while a portion of the trailing edge of the overlay image is gradually removed from the trailing edge of the overlay image and replaced with the current actual image received from the moderator device 130.

[0079] In step 509, the receiver 120 generates an overlay image based on the overlay image generated in step 507 and the current image set in step 501. The generated overlay image thus represents the user's actual input provided by the user and may further include predicted user input based on the user's actual input, which is superimposed on the current image. For example, the overlay image may be superimposed on the real image to form an overlay image, as described with respect to FIG. 5C below, which may then be transmitted by the receiver 120 to the display 105. Thus, the latency of the collaboration platform 100 is not perceived on the display 105 because the predicted portion of the overlay image appears similar to the user's input. Furthermore, the current image may be periodically updated as the receiver 120 receives additional images (e.g., real images) from the moderator device 130. For example, the received additional images may be decoded and / or added to a buffer and may become the current image. In this manner, the overlay image generated by the receiver 120 may be superimposed on the updated current image.

[0080] As shown in FIG. 5C, the overlay image may be superimposed on the real image to form an overlaid image. For example, the real image may include a line 515 generated by the moderator device 130 based on user input data corresponding to the user input received by the receiver 120 from the display 105. The line 515 represents what the user actually drew on the display 105, but only includes a majority of what was generated by the moderator device 130 based on the user input data. For example, the user's actual input in real time may be at another point on the display 105, as indicated by the stylet 700. As described above, the overlay image generated by the receiver 120 includes a first portion 516, which represents the user's actual input received by the receiver 120, and a second extended portion 517, which may be a prediction of the user's intended input based on the user input data received by the receiver 120. Additionally, overlay images, e.g., lines 516 and 517, may be superimposed on real image, e.g., line 515, to form overlay images, e.g., lines 515, 516, and 517. As described in more detail below, portions of the overlay images may be removed, e.g., as a function of time, such that as the real image extends, e.g., as a "3" is rendered, line 515 becomes longer, while overlay image lines 516 and 517 may be displayed only toward the extending leading end of overlay image line 515, as shown in FIG. 5C. Additionally, as overlay images may also be generated, based on the speed of user input, overlay images, e.g., lines 516 and 517, may be displayed as longer lines when user input is received by display 105 faster, and as shorter lines when user input is received by display 105 slower.

[0081] 5A, in step 510, the receiver 120 transmits an overlaid image, e.g., a first portion and a second extended portion of the overlaid image that are superimposed on the current image, to the display 105, thereby reducing and / or eliminating latency in the collaborative platform 100. Further, an additional real image corresponding to the additional user input data from the display 105 may be received by the receiver 120 from the moderator device 130 and set as an additional current image, and an additional overlaid image may be generated by the receiver 120 based on the overlay image created by these additional user input data and superimposed on the additional current image.

[0082] 6A-6E, user input provided by a user is illustrated in conjunction with a display of an overlaid image generated by the receiver 120 to illustrate the latency of the real image. As shown in FIG. 6A, the original image displayed on the display 105, e.g., a touch screen, may be blank, and the user may use a stylet 700 to interact with the display 105 by pressing the stylet 700 against the display 105 at point 605. As shown in FIG. 6B, the user drags the stylet 700 on the display 105 from point 605 to point 606. The dragging motion of the stylet 700 by the user, i.e., the user input, may be converted by the display 105 into user input data and transmitted to the receiver 120, which then transmits the user input data to the moderator device 130, which modifies the original image and generates a real image based on the user input data, as described above. An overlay image is then generated by the receiver 120 based on the user input data (optionally user type) and the real image received from the moderator device 130, transmitted to the display 105, and displayed. As explained above, the overlay image may be formed by an overlay image superimposed on the real image, while the overlay image includes a first portion, which represents the user's actual input received by the display 105, and a second extended portion, which may be a prediction of the user's intended input based on the user input data received by the display 105. As shown in FIG. 6B, the real image is still a blank original image, and thus the overlay image appears to include only the overlay image 701 of the overlay image. Thus, latency is reduced on the collaborative platform 100 because the overlay image is displayed almost immediately after the user drags the stylet 700 from point 605 to 606, and is therefore hardly noticed by the user or other observers looking at the display 105.

[0083] The latency of the collaborative application of the collaborative platform 100 is illustrated in FIG. 6C. As shown in FIG. 6C, the user continues to drag the stylet 700 from point 606 to point 607. Meanwhile, the user input can be continuously converted by the display 105 into user input data and transmitted to the receiver 120, which is then continuously transmitted to the moderator device 130 via a wireless connection, e.g., WiFi, for processing. As described above, the moderator device 130 modifies the original image stored in memory based on the user input data and generates a real image representing the user input, e.g., the user's dragging movement of the stylet 700 on the display 105. As shown in FIG. 6C, when the stylet 700 is at point 607, the moderator device 130 has only the processed user input data representing the user's dragging movement of the stylet 700 from point 605 to point 606, and thus generates a real image, e.g., real image 702, representing the user's input. The real image generated by the moderator device 130 is then transmitted to the receiver 120. As explained above, the receiver 120 generates an overlay image, e.g., overlay image 701, including a first portion representing the user's actual input received by the display 105 and a predicted second extended portion representing the user's intended input superimposed on the real image 702. The overlay image is then transmitted to the display 105 via a wired connection, e.g., an HDMI® cable, for display.

[0084] The data flow of the collaborative application requires that user input data be transmitted from the display 105 to the receiver 120 via a wired connection and from the receiver 120 to the modulator device 130 via a wireless connection, and that the real image be transmitted from the modulator device 130 to the receiver 120 via a wireless connection and finally from the receiver 120 to the display 105 via a wired connection. As a result, an undesirable latency of the collaborative platform 100 is observed. This is illustrated in FIG. 6C as the real image 702, which is displayed together with the latency from the overlay image 701. As an illustrative example in FIG. 6C, when the stylet 700 is at point 607, the overlay image 701 appears as a mark from point 605 to the immediate neighbor 607, while the real image 702 has only reached point 606.

[0085] Furthermore, as shown in FIG. 6D, when the stylet 700 is at point 608, the overlay image 701 appears as a mark from point 605 to the immediate neighbor 608, while the real image 702 has only reached point 607. FIG. 6E illustrates the display 105 after a certain time after the latency of the collaborative platform 100, such that the overlay image 701 and the real image 702 extend from point 605 to point 608.

[0086] In addition, the receiver 120 may derive and / or receive information indicating one or more user types such that the generated overlay image is also based on one or more user types. For example, the user may select one or more user types, such as thickness, color, or marker or eraser type, and provide user input according to the selected user type. Thus, as the user begins to draw, for example, the number "3" in red on the display 105, the overlay image will be generated by the receiver 120 and transmitted to the display 105 as an overlaid image of the number "3" in red, such that it will begin to be displayed on the display 105 with a reduced latency.

[0087] 7A-7D, user input provided by a user is illustrated in conjunction with a display of an overlaid image generated by receiver 120 such that the overlaid image includes the user's actual input in addition to the predicted user input generated by receiver 120, superimposed on the actual image. As shown in FIG. 7A, a user may interact with display 105 using stylet 700 by pressing stylet 700 against display 105 at point 705(5,6) and dragging stylet 700 across display 105 from point 705(5,6) to point 706(5,7), point 707(5,8), point 708(5,9), point 709(5,10). Thus, the user's actual input is depicted as line 703, as shown in FIG. 7A. The dragging movement of the twill 700 by the user, i.e., the user input, can be converted by the display 105 into user input data and transmitted to the receiver 120, as described above. The user input data thus includes the user's actual input, e.g., spatial coordinates (5,6), (5,7), (5,8), (5,9), and (5,10). An overlay image is then generated by the receiver 120 based on the user input data (optionally, the user type) and transmitted to the display 105 for display as an overlay image. As described above, the overlay image includes the user's actual input, e.g., line 703, as well as the predicted user input, e.g., line 704 generated by the receiver 120, as shown in FIG. 7B. For example, line 704 may be predicted by receiver 120 based on spatial coordinates (5,6), (5,7), (5,8), (5,9), and (5,10) of the user input data using extrapolation, e.g., linear extrapolation, polynomial extrapolation, conic extrapolation, French curve extrapolation, and / or any other well-known extrapolation techniques, machine learning, artificial intelligence, or neural networks.Based on the spatial coordinates (5,6), (5,7), (5,8), (5,9), and (5,10), the receiver 120 predicts that the user's next input will be to continue dragging the stylet 700 from point 709 (5,10), to point 710 (5,11), to point 711 (5,12), to point 712 (5,13), to point 713 (5,14).

[0088] Additionally, line 704 may be predicted by receiver 120 based on the time coordinates of the user input data using extrapolation, machine learning, artificial intelligence, or neural networks. For example, the user input data received by receiver 120 may include data indicating that point 705 was touched by stylet 700 at T1, point 706 at T2, point 707 at T3, point 708 at T4, and point 709 at T5, and receiver 120 may determine the velocity of stylet 700 based on T1-T5. Thus, receiver 120 would predict that point 710 will be touched by stylet 700 at T6, point 711 at T7, point 712 at T8, and point 713 at T9, such that the velocity between T6-T9 corresponds to the velocity of T1-T5. Thus, points 710-713 of line 704 will be displayed on display 105 with a velocity corresponding to a velocity based on T1-T5 such that points 710, 711, 712, and 713 of line 704 will appear on display 105 at the same time as the user drags stylet 700 to points 710, 711, 712, and 713 in real time, thereby eliminating any latency on collaborative platform 100. Additionally, receiver 120 may determine the acceleration of stylet 700 based on T1-T5 such that the acceleration between T6-T9 corresponds to the acceleration of T1-T5. Thus, points 710-713 of line 704 will be displayed on display 105 with modified velocities corresponding to accelerations based on T1-T5 such that points 710, 711, 712, and 713 of line 704 will appear on display 105 at the same time as the user drags stylet 700 to points 710, 711, 712, and 713 in real time, thereby eliminating any latency on collaborative platform 100.

[0089] Using extrapolation, machine learning, artificial intelligence, or neural networks, receiver 120 may predict complex curved lines by predicting finite line segments that form a curve and predicting the angle of each finite line segment and the change in angle between adjacent line segments. For example, receiver 120 may detect a first angle of a first line segment of the user's actual input, detect a second angle of a second line segment of the user's actual input, and determine the change in angle between the first angle and the second angle. Based on the first angle, the second angle, and the change in angle of the user's actual input, receiver 120 may predict the curve of the user's next input of the finite line segment. For example, if receiver 120 detects that the user's actual input is a sequence of finite line segments that form a curve with a known change in angle between each adjacent line segment, receiver 120 will generate an overlay image having a predicted extended portion with the same curvature. Additionally, the receiver 120 may detect a rate of change of the angle between adjacent finite line segments of the user's actual input and predict the user's next input based on the detected rate of change of the angle between adjacent finite line segments.

[0090] As shown in FIG. 7C , a user may use the stylet 700 to interact with the display 105 by pressing the stylet 700 against the display 105 at point 716(2,5) and dragging the stylet 700 on the display 105 from point 716(2,5) to point 717(5,6) to point 718(8,8). Thus, the user's actual input is depicted as a line 714 as shown in FIG. 7C . The dragging motion of the stylet 700 by the user, i.e., the user input, can be converted by the display 105 into user input data and transmitted to the receiver 120 as described above. Thus, the user input data includes the user's actual input, e.g., a first line segment from spatial coordinate (2,5) to spatial coordinate (5,6) having a first angle and a second line segment from spatial coordinate (5,6) to spatial coordinate (8,8). An overlay image is then generated by receiver 120 based on the user input data (optionally user type) and transmitted to display 105 to be displayed as an overlay image. As explained above, the overlay image includes the user's actual input, e.g., line 714, as well as predicted user input, e.g., line 715, generated by receiver 120, as shown in Figure 7D. For example, line 715 may be predicted by receiver 120 using extrapolation, machine learning, artificial intelligence, or neural networks based on the spatial coordinates (2,5), (5,6), and (8,8) of the user input data. Based on the first angle of the first line segment from spatial coordinate (2,5) to spatial coordinate (5,6) and the second angle of the second line segment from spatial coordinate (5,6) to spatial coordinate (8,8), the receiver 120 predicts that the user's next input will continue to drag the stylet 700 from point 718 (8,8), to point 719 (11,11), to point 720 (14,15). The angles of the line segments from point 718 to point 719 and from point 719 to point 720 will correspond to the rate of change between the first angle of the line segment from point 716 to point 717 and the second angle of the line segment from point 717 to point 718.

[0091] As described above, line 715 may also be predicted by receiver 120 based on the time coordinates of the user input data using extrapolation, machine learning, artificial intelligence, or neural networks. For example, the user input data received by receiver 120 may include data indicating that point 716 was touched by stylet 700 at T1, point 717 at T2, and point 718 at T3, and receiver 120 may determine the velocity of stylet 700 based on T1-T3. Thus, receiver 120 would predict that point 719 is touched by stylet 700 at T4 and point 720 at T5, such that the velocity between T3-T5 corresponds to the velocity of T1-T3. Thus, points 719 and 720 of line 715 will be displayed on display 105 with a velocity corresponding to a velocity based on T1-T3 such that points 719 and 720 of line 715 will appear on display 105 at the same time as the user drags stylet 700 from point 719 to 720 in real time, thereby eliminating any latency on collaborative platform 100. Additionally, receiver 120 may determine the acceleration of stylet 700 based on T1-T3 such that the acceleration between T3-T5 corresponds to the acceleration of T1-T3. Thus, points 719 and 720 of line 715 will be displayed on display 105 with a modified velocity corresponding to the acceleration based on T1-T3 such that points 719 and 720 of line 715 will appear on display 105 at the same time as the user drags stylet 700 from point 719 to 720 in real time, thereby eliminating any latency on collaborative platform 100.

[0092] 8A and 8B, an exemplary method of collecting user type data according to an aspect of the present invention is provided. FIG. 8A is a screenshot of the display 105 at a first time, and FIG. 8B is a screenshot of the display 105 at a second time. As shown in FIG. 8A and 8B, the interface displayed on the display 105 may include user-friendly icons in a ribbon at the top of the screen representing selectable user types, including, but not limited to, a marker icon 601, a thickness icon 602, an eraser icon 603, and a color icon 604. For example, in response to clicking on the thickness icon 602, a drop-down menu may appear with additional sub-icons for selecting between thickness levels such as "thin line", "medium line", and "thick line". Furthermore, for example, in response to clicking on the color icon 604, a drop-down menu may appear with additional sub-icons for selecting between different colors such as "gray", "black", "blue", "yellow", etc. Preferably, the user type of the user input may then be set to pre-programmed default settings, e.g., default color (gray), default thickness (medium line), and default marker user type, until changed by the user.

[0093] As described above, the receiver 120 may receive user input data from the display 105 via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth, and may determine from the user input data one or more user types of the user input. For example, using, e.g., machine learning, artificial intelligence, and / or neural networks, the receiver 120 may analyze and / or process the user input data to determine the user type. Using machine learning, artificial intelligence, and / or neural networks, the receiver 120 may determine the user type based on observing a pattern of the user's movement relative to the display 105 and / or the user's actions, e.g., the type of mark subsequently rendered.

[0094] 8A, the user has drawn a line extending from point 605 to point 606 to point 607, for example, by touching the display 105 and moving from point 605 to point 606 to point 607 without breaking contact with the display 105. As shown in FIG. 8A, a marker icon 601 has previously been selected, for example, by touching any point within the perimeter of the point on the display 105 that corresponds to the marker icon 601. Based on machine learning, artificial intelligence, or neural networks, the receiver 120 can identify the interface of the display 105 and correlate a specific action by the user (e.g., clicking the point on the display 105 where the marker icon 601 resides) with a specific user type. For example, if the marker icon 601 is observed to be clicked, and shortly thereafter the user input data indicates that the clicking of the marker icon 601 was followed by a dragging movement of the stylet by the user from point 605 to point 606 to point 607 resulting in a mark extending from point 605 to point 606 to point 607, the receiver 120 will learn that by clicking the point on the display 105 where the marker icon 601 resides, a marker user type is selected, which allows the user to draw a line. Thus, the receiver 120 will associate the spatial region of the marker icon 601 with the ability to draw solid lines in a drawing. Using machine learning to compare multiple user inputs made at different points in time, the receiver 120 can infer the various icons of any interface and their respective functions. Thus, the receiver 120 may include a database by which to compare user actions on the display 105 and determine the selected user type given a specific interface.

[0095] As shown in FIG. 8B, at a second time, the receiver 120 receives user input data indicating that the user has discontinued contact with the display 105 and then, through machine learning, contacted the display 105 at a point on the display 105 associated with the eraser icon 603, which is associated with the functionality of erasing. Thus, in response to clicking the eraser icon 603, the receiver 120 determines that an eraser user type has been selected, as shown in FIG. 8B, and generates an overlay image of an eraser mark from point 607 to point 606 in response to the user contacting the display 105 at point 607 and dragging the stylet from point 607 to point 606. As will be appreciated by those skilled in the art, by analyzing the user input data received from the display 105 and determining the selected user type, the receiver 120 generates an overlay image based on the user type of the user input, not just the user input, and accurately displays an overlay image that corresponds to the user's selected user type and user input.

[0096] Referring now to FIG. 9A, a block diagram of another exemplary embodiment of a collaboration platform 100' according to the principles of the present invention is provided. As shown in FIG. 9A, user input data may be transmitted from the display 105' to the receiver 120' via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth®. In addition, user input data and real images may be communicated between the receiver 120' and the moderator device 130' across a wireless connection, e.g., WiFi. Furthermore, overlaid images may be transmitted from the receiver 120' to the display 105' via a wired connection, e.g., an HDMI® cable. As shown in FIG. 9A, data user indicating the user type of input may be transmitted from the moderator device 130' to the receiver 120' via a wireless connection, e.g., a defined TCP port.

[0097] Referring now to Figure 9B, a sequence diagram for using the collaboration platform 100' depicted in Figure 9A is provided. As described above with reference to Figure 4B, the collaboration platform 100' of Figure 9A also launches a collaboration application to display a first image based on an original image file stored on the moderator device 130', receive user input, modify the original image file stored on the moderator device 130' based on the user input, and display a second image based on the modified original image.

[0098] Like the collaboration platform 100 of FIG. 4A, the collaboration platform 100' may generate an overlay image based on the user input provided by the user by the receiver 120', generate an overlayed image based on the overlay image, display the overlayed image on the original image on the display 105', and launch an overlay image generator application to reduce the latency of the collaboration platform 100'. The collaboration platform 100' differs from the collaboration platform 100 in that the receiver 120' may receive data indicating a user type directly from the moderator device 130' via a wireless connection, e.g., a defined TCP port, in addition to the user input data received from the display 105' via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth. In this embodiment, the receiver 120' does not need to derive information regarding the selected user type of the user input from the user input data received from the display 105. For example, as described above with respect to FIG. 3C, the moderator device 130' may include an overlay image application 139 to process and analyze the user input data received from the display 105' through the receiver 120', determine a selected user type from the user input data, and transmit data indicative of the selected user type to the receiver 120' via a defined TCP port. Thus, the receiver 120' generates an overlay image based on both the user input data and the user type data, generates an overlaid image based on the overlay image, and transmits the overlaid image to the display 105' via a wired connection, e.g., an HDMI cable, thereby reducing the latency of the collaboration platform 100'.

[0099] Referring now to FIG. 10A, a block diagram of another exemplary embodiment of a collaboration platform 100'' according to the principles of the present invention is provided. As shown in FIG. 10A, user input data may be transmitted from the display 105'' to the receiver 120'' via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth®. In addition, user input data and real images may be communicated between the receiver 120'' and the moderator device 130'' across a wireless connection, e.g., WiFi. Furthermore, overlaid images may be transmitted from the receiver 120'' to the display 105'' via a wired connection, e.g., an HDMI® cable. As shown in FIG. 10A, data indicating a user type of user input may be transmitted from the operating system of the moderator device 130'' to the receiver 120'' via a modified user input back channel (UIBC) extension. The UIBC extension will generally be used to transmit user input data from the receiver to the moderator device. Here, however, the UIBC extension is modified to allow the transmission of data from the moderator device 130'' to the receiver 120''.

[0100] 10B, a sequence diagram for using the collaboration platform 100″ depicted in FIG. 10A is provided. As described above with reference to FIGS. 4B and 9B, the collaboration platform 100″ of FIG. 10A also launches a collaboration application to display a first image based on an original image file stored on the moderator device 130″, receive user input, modify the original image file stored on the moderator device 130″ based on the user input, and display a second image based on the modified original image.

[0101] 4A, collaboration platform 100'' may generate an overlay image based on user input provided by a user via receiver 120'', generate an overlayed image based on the overlay image, display the overlayed image over the original image on display 105'', and launch an overlay image generator application to reduce latency of collaboration platform 100''. Collaboration platform 100'' differs from collaboration platform 100 in that receiver 120'' may receive data indicative of a user type directly from the operating system of moderator device 130'' via the UIBC extension described above, in addition to user input data received from display 105'' via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth.

[0102] In this embodiment, the receiver 120'' does not need to derive information regarding the selected user type of the user input from the user input data received from the display 105. For example, as described above with respect to FIG. 3C, the operating system 137 of the moderator device 130'' may process and analyze the user input data received from the display 105'' through the receiver 120'', determine the selected user type from the user input data, and transmit data indicating the selected user type to the receiver 120'' via the UIBC extension. Thus, the receiver 120'' generates an overlay image based on both the user input data and the user type data, generates an overlaid image based on the overlay image, and transmits the overlaid image to the display 105'' via a wired connection, e.g., an HDMI cable, to be displayed over the original image displayed on the display 105'', thereby reducing the latency of the collaboration platform 100''.

[0103] 11, a flow chart is illustrated detailing the data flow and decisions made in implementing the overlaid image generation functionality of the receiver 120' of the collaboration platform 100' or the receiver 120'' of the collaboration platform 100''. As mentioned above, the receiver 120' of the collaboration platform 100' and the receiver 120'' of the collaboration platform 100'' may be used to generate an overlay image based on user input, such that the overlay image is displayed while the real image is generated by the moderator device 130', 130'', thereby reducing the latency of the collaboration platform 100', 100''.

[0104] To begin the process depicted in FIG. 11, in step 1101, user input data corresponding to a user input is received by the receiver 120′, 120″ from the display 105′, 105″, for example, via a USB cable and / or a wireless connection such as Bluetooth. In step 1102, the receiver 120′, 120″, upon launching a collaborative application, transmits the user input data to the source of the original image, for example, the moderator device 130′, 130″, for further processing and analysis. For example, the moderator device 130′, 130″ may derive data indicating at least one user type of the user input. Thus, in step 1103, user type data is received by the receiver 120′ from an application of the moderator device 130′, for example, via a defined TCP port, or by the receiver 120″ from the moderator device 130″, for example, via a UIBC extension.

[0105] In step 1104, the receiver 120', 120'' generates an overlay image based on the user input data received in step 1101 and the user type data received in step 1103. The overlay image may be generated based on the user input data and a default user type until a new user type is received in step 1103. Preferably, the user input user type may then be set to pre-programmed default settings, e.g., default color (gray), default thickness (medium line), and default marker user type, until changed by the user. In step 1106, the receiver 120', 120'' receives a real image generated by the moderator device 130', 130'' based on the user input data received from the display 105', 105''. In step 1106, the receiver 120', 120'' generates an overlayed image based on the overlay image and the real image. As explained above, the overlaid image may be formed by an overlay image superimposed on the real image, where the overlay image includes a first portion, which represents the user's actual input received by the receiver 120', 120'', and a second extended portion, which may be a prediction of the user's intended input based on the user input data received by the receiver 120', 120''. In step 1107, the receiver 120', 120'' transmits the overlaid image, for example, to the display 105', 105'', to be displayed on the original image, thereby reducing the latency of the collaborative platform 100', 100''.

[0106] 12A, a block diagram of another exemplary embodiment of a collaborative platform 100''' according to the principles of the present invention is provided. As shown in FIG. 12A, user input data may be transmitted from the display 105''' to the receiver 120''' via a wired connection, e.g., a USB cable, and / or a wireless connection such as Bluetooth®. The receiver 120''' may be capable of implementing the functionality of the moderator device described herein. For example, the receiver 120''' may generate a real image based on the user input data received from the display 105''', and further generate an overlay image including a predicted portion based on the user input data, as well as an overlay image and an overlay image based on the real image. The overlay image may be transmitted from the receiver 120''' to the display 105''' via a wired connection, e.g., an HDMI® cable.

[0107] 12B, a sequence diagram for using the collaborative platform 100'''' depicted in FIG. 12A is provided. The collaborative platform 100'''' of FIG. 12A may launch a collaborative application to display a first image based on an original image file stored on the receiver 120''', receive user input, modify the original image file stored on the receiver 120'''' based on the user input, and display a second image based on the modified original image.

[0108] 4A , collaboration platform 100''' may generate an overlay image based on user input provided by a user through receiver 120''', including a predicted portion based on the user input data, generate an overlayed image based on the overlay image, and display the overlayed image over the original image on display 105''', invoking an overlay image generator application to reduce latency of collaboration platform 100'''. Collaboration platform 100''' differs from collaboration platform 100 in that receiver 120''' may function as a moderator device as described herein and generate a modified real image based on user input data received from display 150''' without the need to transmit the user input data to an external moderator device.

[0109] Thus, receiver 120''' generates a modified real image based on the user input, the user input data, and optionally, the user type data, generates an overlay image based on the user input data, and optionally, the user type data, generates an overlayed image based on the overlay image and the real image, and transmits the overlayed image to display 105''' via a wired connection, e.g., an HDMI (registered trademark) cable, to be displayed over the original image displayed on display 105''', thereby reducing the latency of collaborative platform 100'''.

[0110] The collaborative platform described herein for generating overlaid images for display would reduce latency due to the need to transmit data across a wireless network, e.g., between the receiver, moderator, and member devices. As would be understood by one skilled in the art, additional sources of delay include processor and application delays. For example, a computing device for receiving user input, e.g., a touchscreen display, would be limited in its processing time of the user input to generate user input data for transmission to the receiver. In accordance with the principles of the present invention, extrapolation, artificial intelligence, machine learning, and / or neural networks may be implemented to predict user input as the user interacts with the touchscreen, such that an overlay image generator application of the receiver may generate an overlaid image based on the predicted user input, rather than waiting for user input data from the touchscreen and / or moderator device (which may suffer from application delays in processing user input data), thereby further reducing the latency of the collaborative platform.

[0111] It should be understood that any of the computer operations described herein above may be implemented, at least in part, as computer readable instructions stored on a computer readable memory. Of course, it should be understood that the embodiments described herein are illustrative, and that the components can be arranged, substituted, combined and designed in a wide variety of different configurations, all of which are contemplated and fall within the scope of the present disclosure.

[0112] The foregoing description of the exemplary embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed embodiments. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

1. 1. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, first user input data on a second device indicative of a first user input at a first time; transmitting, by the first device, the first user input data to the third device; determining, by the first device, an overlay image indicative of a predicted second user input based on the first user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; receiving, by the first device, second user input data on the second device indicative of an actual second user input at a second time; A method comprising:

2. 2. The method of claim 1, wherein determining, by the first device, the overlay image includes determining, by the first device, a first portion of the overlay image indicative of the first user input on the second device based on the first user input data.

3. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, determining, by the first device, the overlay image includes determining, by the first device, a first portion of the overlay image indicative of the user input on the second device based on the user input data; The method, wherein determining, by the first device, the overlay image includes predicting, by the first device, an extended portion of the overlay image based on the user input data.

4. 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on at least one of spatial coordinates or temporal coordinates of the user input data.

5. 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on a velocity of the user input data.

6. 4. The method of claim 3, wherein the extended portion of the overlay image comprises a curved portion comprising a plurality of finite line segments, and predicting, by the first device, the extended portion of the overlay image includes predicting the curved portion based on an angle of each finite line segment of the plurality of finite line segments.

7. 4. The method of claim 3, wherein predicting, by the first device, the extended portion of the overlay image based on the user input data includes predicting, by the first device, the extended portion of the overlay image based on at least one of extrapolation, machine learning, artificial intelligence, or neural networks.

8. The method of claim 3 , wherein determining, by the first device, the overlay image includes determining, by the first device, the overlay image comprising the first portion and an extending portion of the overlay image.

9. The method of claim 1 , wherein a portion of the overlay image is displayed on the second device for a predetermined period of time.

10. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, a portion of the overlay image is displayed on the second device for a predetermined period of time; The method, wherein the predetermined period of time is at least as long as a latency period on the collaborative platform.

11. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, The method, wherein the overlay image comprises a leading edge and a trailing edge, such that as a number of leading edge spatial coordinates increases on the second device, a portion of the trailing edge spatial coordinates are removed from the second device depending on the length of latency on the collaborative platform.

12. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, The method of claim 1, wherein the overlay image comprises a maximum amount of spatial coordinates such that when additional spatial coordinates beyond the maximum amount of spatial coordinates are displayed, initial display spatial coordinates are removed from the overlay image.

13. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, The method, wherein the overlay image has a leading edge, a trailing edge, and a maximum spatial length such that as the leading edge extends, the trailing edge is removed to maintain the maximum spatial length of the overlay image in the overlaid image displayed on the second device.

14. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, The method, wherein the overlay image comprises a leading edge and a trailing edge such that as the leading edge extends onto the second device at a rate, the trailing edge is removed from the second device at the rate.

15. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; Including, A method, wherein the overlay image comprises a leading end and a trailing end, such that as a number of spatial coordinates of the leading end increase on the second device, a portion of the spatial coordinates of the trailing end are removed from the second device depending on the rate at which the spatial coordinates increase.

16. 2. The method of claim 1, further comprising determining, by the first device, an input type corresponding to the first user input on the second device, the input type comprising at least one of a thickness, a color, or a marker or eraser type.

17. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; determining, by the first device, an input type corresponding to the user input on the second device; the input type comprises at least one of thickness, color, or marker or eraser type; The method, wherein determining, by the first device, the input type includes determining, by the first device, the input type based on the user input data and machine learning.

18. The method of claim 1 , further comprising receiving, by the first device, data from the third device indicating an input type corresponding to the first user input.

19. 20. The method of claim 18, wherein receiving, by the first device, data indicative of the input type includes receiving, by the first device, data indicative of the input type from an application running on the third device over a defined TCP port.

20. A method for reducing latency on a collaborative platform, the method comprising: receiving, by the first device, a first image from a third device; receiving, by the first device, user input data on a second device indicative of a user input; transmitting, by the first device, the user input data to the third device; determining, by the first device, an overlay image based on the user input data; determining, by the first device, an overlaid image based on the overlay image and the first image; transmitting, by the first device, the overlaid image to the second device and displaying the overlaid image on the second device; receiving, by the first device, data from the third device indicating an input type corresponding to the user input; Including, The method, wherein receiving, by the first device, data indicative of the input type includes receiving, by the first device, data indicative of the input type from an operating system running on the third device via a User Input Backchannel (UIBC) extension.

Images