A method, apparatus, system and first electronic device for synchronizing with a whiteboard

Abstract

This application discloses a synchronization method, apparatus, system, and first electronic device for a collaborative whiteboard. The method is applied to the first electronic device and includes: receiving vector data sent by the second electronic device through a first channel between the two devices, and drawing handwriting in a transparent top-level view based on the vector data; receiving visual data sent by the second electronic device through a second channel between the two devices, the visual data including image frames generated based on the screen content of the collaborative whiteboard; displaying the received visual data in a bottom-level view; and, in response to displaying the visual data in the bottom-level view, cleaning up handwriting related to the screen content in the transparent top-level view based on the visual data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a synchronization method, apparatus, system, and first electronic device for a collaborative whiteboard. Background Technology

[0002] With the rapid development of mobile internet technology, the demand for real-time collaborative interaction in scenarios such as remote work, online education, and sports tactical analysis is increasing. As a core interactive tool, collaborative whiteboards can support multiple users in different physical locations to perform operations such as drawing, writing, and inserting pictures on a shared virtual canvas in real time, and view the operation results of other users in real time. It has become an indispensable application system in the above scenarios.

[0003] However, existing collaborative whiteboards struggle to simultaneously achieve "smooth writing" (low latency) and "smooth visual consistency" (high fidelity), resulting in a poor user experience. Summary of the Invention

[0004] In view of the above problems, this application provides a synchronization method, device, system, and first electronic device for a collaborative whiteboard, so as to achieve a balance between the smoothness of writing on the collaborative whiteboard and the fidelity of screen synchronization, thereby improving the user experience. The specific solution is as follows:

[0005] The first aspect of this application provides a synchronization method for a collaborative whiteboard, applied to a first electronic device, comprising:

[0006] The system receives vector data sent by the second electronic device via a first channel and draws handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard.

[0007] Visual data transmitted by the second electronic device is received via a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than that of the second channel.

[0008] The received visual data is displayed in a bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view;

[0009] In response to displaying the visual data in the underlying view, the handwriting related to the screen content in the transparent top-level view is cleaned up based on the visual data.

[0010] In one possible implementation, the first channel is a point-to-point connection channel established between the first electronic device and the second electronic device, and the second channel is a connection channel established between the first electronic device and the second electronic device through a remote server.

[0011] In one possible implementation, before receiving vector data transmitted by the second electronic device via the first channel between the two electronic devices, the method further includes:

[0012] Monitor the link status of the first channel between the device and the second electronic device;

[0013] In response to the link status indicating that the first channel is available, vector data transmitted by the second electronic device is received through the first channel;

[0014] In response to the link status indicating that the first channel is unavailable, vector data transmitted by the second electronic device is received via the second channel.

[0015] In one possible implementation, receiving vector data transmitted by the second electronic device via the second channel includes:

[0016] The second channel receives vector data encapsulated in the video encoded stream corresponding to the visual data.

[0017] In one possible implementation, the second channel includes a first sub-channel and a second sub-channel; the first sub-channel is used to receive the visual data; receiving vector data transmitted by the second electronic device through the second channel includes:

[0018] Vector data sent by the second electronic device is received through the second sub-channel.

[0019] In one possible implementation, the synchronization method of the collaborative whiteboard further includes:

[0020] The first time stamp generated by the vector data on the second electronic device is received through the first channel;

[0021] The visual data also includes: a second timestamp generated by the image content on the second electronic device;

[0022] The response to displaying the visual data in the bottom layer view, and the cleaning up of handwriting related to the screen content in the transparent top layer view based on the visual data, includes:

[0023] In response to displaying the visual data in the underlying view, a second timestamp of the screen content being generated on the second electronic device is obtained from the visual data;

[0024] Clean up the handwriting in the transparent top-level view that corresponds to vector data with the first timestamp being earlier than or equal to the second timestamp.

[0025] In one possible implementation, receiving visual data transmitted by the second electronic device via a second channel between the second electronic device and the second electronic device includes:

[0026] Visual data transmitted by the second electronic device at a variable frame rate is received via the second channel between the second electronic device and the second electronic device.

[0027] The variable frame rate is dynamically adjusted according to the rate of change of the collaborative whiteboard's image.

[0028] Another aspect of this application provides a synchronization device for a collaborative whiteboard, comprising:

[0029] A first receiving module is configured to receive vector data transmitted by a second electronic device via a first channel between the second electronic device and the second electronic device; the vector data is generated by the second electronic device in response to user operations on a collaborative whiteboard.

[0030] The first drawing module is used to draw strokes in a transparent top-level view based on the vector data;

[0031] The second receiving module is configured to receive visual data sent by the second electronic device through a second channel between the second electronic device and the second electronic device, the visual data including image frames generated based on the screen content of the collaborative whiteboard; the transmission delay of the first channel is less than the transmission delay of the second channel;

[0032] The display module is used to display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view;

[0033] A cleanup module is used to clean up handwriting related to the screen content in the transparent top-level view in response to the display of the visual data in the bottom-level view.

[0034] A third aspect of this application provides a first electronic device, comprising:

[0035] Memory is used to store computer programs;

[0036] The processor is configured to execute the computer program to enable the first electronic device to perform the following method steps:

[0037] The system receives vector data sent by the second electronic device via a first channel and draws handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard.

[0038] Visual data transmitted by the second electronic device is received via a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than that of the second channel.

[0039] The received visual data is displayed in a bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view;

[0040] In response to displaying the visual data in the underlying view, the handwriting related to the screen content in the transparent top-level view is cleaned up based on the visual data.

[0041] A fourth aspect of this application provides a synchronization system for a collaborative whiteboard, comprising: a first electronic device and a second electronic device;

[0042] The first electronic device and the second electronic device are connected by a communication link and have a parallel and independent first channel and a second channel. The transmission delay of the first channel is less than the transmission delay of the second channel.

[0043] The second electronic device is configured to generate vector data in response to user operations on the collaborative whiteboard, send the vector data to the first electronic device through the first channel, and generate visual data containing image frames based on the real-time screen content of the collaborative whiteboard, and send the visual data to the first electronic device through the second channel.

[0044] The first electronic device is configured to receive the vector data through the first channel and draw handwriting in a transparent top-level view based on the vector data; receive the visual data through the second channel and display the visual data in a bottom-level view that serves as the display base for the transparent top-level view; and, in response to the display of the visual data in the bottom-level view, clean up handwriting related to the content of the collaborative whiteboard screen in the transparent top-level view based on the visual data.

[0045] In this application, the low-latency first channel prioritizes the transmission of lightweight, high-frequency vector data. Since its transmission latency is optimized to an extremely low level, it can achieve instantaneous transmission of vector data. After receiving the vector data, the first electronic device immediately draws the corresponding handwriting in real time in the transparent top-level view, realizing instantaneous feedback between user operation and handwriting display, achieving a smooth writing effect, significantly improving writing fluency, and meeting the core needs of users for smooth interaction in remote collaboration scenarios.

[0046] At the same time, the high-fidelity second channel synchronously transmits the complete visual data captured and generated by the second electronic device. This visual data serves as the visual truth value of the collaborative whiteboard image. After being transmitted to the first electronic device, it is displayed in the bottom layer view. The bottom layer view serves as the display base for the transparent top layer view. The visual data displayed in the bottom layer view can accurately correct the vector strokes drawn in the transparent top layer view. Even if the vector data has slight deviations due to network fluctuations or differences in device performance during transmission and rendering, it can be quickly corrected by the visual truth value of the bottom layer view, ensuring that the whiteboard image displayed by the first electronic device and the second electronic device is completely consistent, thus thoroughly guaranteeing the consistency of the image in multi-device collaboration.

[0047] In addition, since there is a slight time difference between the vector handwriting drawing in the first channel and the visual data display in the second channel, the superposition of the two can easily produce double handwriting visual ghosting. The cleaning operation can promptly remove historical handwriting that has been covered by the underlying visual data in the transparent top layer view, which not only eliminates ghosting and improves the user's visual experience, but also completely eliminates the problem of accumulated vector data deviation.

[0048] At the same time, cleaning up redundant handwriting can reduce device memory usage, prevent data accumulation from affecting the smoothness of application operation, and ensure the long-term stable operation of collaborative whiteboard applications.

[0049] In summary, by relying on the collaborative transmission of two parallel channels, coupled with the division of labor and cooperation of dual-layer views and handwriting cleanup operations, this application can simultaneously achieve writing fluency and screen consistency, further optimizing the overall collaborative interaction experience. This enables the collaborative whiteboard to better adapt to various real-time collaborative scenarios such as remote work, online education, and sports tactical analysis, significantly improving its applicability and practicality. Attached Figure Description

[0050] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0051] Figure 1 A flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 1 of this application;

[0052] Figure 2 This is a flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 3 of this application.

[0053] Figure 3 A flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 6 of this application;

[0054] Figure 4 A flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 7 of this application;

[0055] Figure 5 A schematic diagram of the structure of a first electronic device provided in this application. Detailed Implementation

[0056] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0057] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0058] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0059] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Reference Figure 1 This is a flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 1 of this application. This method can be applied to a first electronic device, such as... Figure 1 As shown, the method may include, but is not limited to, the following steps:

[0061] Step S101: Receive vector data sent by the second electronic device through a first channel between the two electronic devices, and draw handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user operations on the collaborative whiteboard.

[0062] In this embodiment, both the first and second electronic devices need to have an application installed that provides collaborative whiteboard functionality, or a system with built-in collaborative whiteboard functionality that can be called normally (e.g., some customized operating systems integrate specific collaborative whiteboard function modules). This is the basic hardware and software environment guarantee that the devices can participate in collaborative operations and realize data interaction. Only when the relevant application is installed or the corresponding function module is available can the device have the ability to parse and generate the vector data required for the collaborative whiteboard.

[0063] For devices with a collaborative whiteboard application or collaborative whiteboard functionality installed, the relevant application or function must be launched and running normally. If the application is not launched, the device cannot receive and send vector data generated by collaborative whiteboard operations, and therefore cannot establish a communication link for collaboration. For example, on a mobile phone or computer, it is necessary to ensure that the collaborative whiteboard application is actively running in the background or foreground to prepare for subsequent collaborative operations.

[0064] The first and second electronic devices can complete identity verification through collaborative whiteboard applications. Authentication methods can be diverse, such as login using an account and password, authorization login using third-party accounts (such as WeChat or QQ accounts), and biometric login (fingerprint, facial recognition). Identity verification ensures the legitimacy of the devices and users participating in collaborative operations, guaranteeing the security of collaborative operations and the reliability of data exchange. For example, in an enterprise's internal collaborative whiteboard application, employees must log in using an account and password assigned by the company to verify their identity and prevent unauthorized access to the collaborative operation space.

[0065] After identity verification, both the first and second electronic devices must successfully enter the same collaborative whiteboard operating space, i.e., the collaborative room. The collaborative room is a dedicated operating space specifically designed by the collaborative whiteboard application to enable real-time collaboration among multiple users, similar to a virtual "shared meeting room." Within the collaborative room, devices can interact and synchronize data in real time, enabling collaborative writing, annotation, and discussion. Only electronic devices that have entered the same collaborative room (and meet the aforementioned application-related conditions) can establish corresponding communication links (i.e., the first channel and the second channel) to achieve vector data transmission and collaborative operations. Devices that have not entered the collaborative room, even if they have the collaborative whiteboard application installed, cannot participate in collaborative operations within the current collaborative room and cannot establish communication links with other devices for this collaboration.

[0066] Two parallel and distinct communication links can be established between sub-devices, referred to as the first channel and the second channel, respectively.

[0067] The first channel and the second channel are parallel and independent communication links. The core difference between them lies in the transmission delay, specifically, the transmission delay of the first channel is less than that of the second channel.

[0068] The first channel can be used to transmit lightweight, high-frequency vector data. Based on the real-time transmission requirements of this type of data, its transmission latency is optimized to an extremely low level, enabling instantaneous transmission and real-time rendering of vector data.

[0069] The second channel can be used to transmit large amounts of complete visual data (image frames). Because this type of data is large, it needs to go through multiple stages of processing such as acquisition, encoding, transmission, and decoding during transmission, so its transmission delay is naturally higher than that of the first channel.

[0070] Users can trigger vector data generation through interactive methods such as finger touch and stylus writing on a collaborative whiteboard application on a second electronic device (such as a tablet or mobile phone with a touchscreen). Specifically, the collaborative whiteboard application running on the second electronic device listens for touch events in real time. This method is applicable to electronic devices on various systems, including Android, iOS, and Windows, and can capture user operation-related data by listening for touch events.

[0071] Of course, in addition to listening to touch events, iOS electronic devices can also call the underlying PencilKit framework and register callbacks (such as canvasViewDrawingDidChange).

[0072] Both of the above methods are designed to extract newly added handwriting point sequences, pressure values, and tool types (pen / eraser) in real time, while simultaneously recognizing the corresponding gesture commands of user operations, and combining this information to generate vector data.

[0073] Vector data may include, but is not limited to, at least one of the following: pressure values, tool types, and gesture commands, as well as a sequence of coordinate points of handwriting.

[0074] The handwriting coordinate sequence records the coordinates of each key position in the whiteboard coordinate system during user operations (such as writing or drawing). For example, when a user writes a Chinese character, the coordinates of each point the stylus passes through throughout the entire process, from the beginning to the end of the stroke, are recorded sequentially, forming a coordinate sequence. Using these coordinate points, the receiving end can accurately depict the user's writing trajectory and reconstruct the complete handwriting.

[0075] The pressure value reflects the amount of force applied by the user to the second electronic device during operation. Different pressure values ​​will result in different line thicknesses in the handwriting. For example, when the user writes with force, the line will be thicker; while when writing lightly, the line will be thinner. Based on the received pressure value information, the receiving end can dynamically adjust the line thickness to ensure that the reproduced handwriting matches the user's operation at the sending end, enhancing the realism and expressiveness of the writing.

[0076] Tool type information identifies the tool the user is currently using. Common tools include brushes, erasers, line tools, and circle tools. Different tools have different drawing characteristics and functions. For example, the brush tool is used for freehand writing and drawing, while the eraser tool is used to erase existing strokes. Based on the tool type information, the receiving end invokes the corresponding drawing algorithm and functions to ensure that the effects of the user's operation using different tools are accurately reproduced.

[0077] Gesture commands are instructions generated by users through specific hand gestures, adding rich interactive functionality to collaborative whiteboard applications. Gesture commands can include, but are not limited to:

[0078] Zoom gestures. When a user simultaneously spreads or contracts two fingers on the screen, a zoom command is generated. This command includes the direction of zoom (enlarge or shrink) and the zoom ratio. Upon receiving this command, the receiving device will zoom in or out of the whiteboard content accordingly, allowing the user to view details or the overall layout.

[0079] Rotation gesture. If a user uses two fingers to rotate the screen, a rotation command is generated. The command records the rotation angle and direction (clockwise or counterclockwise). The receiving end uses this information to rotate the selected content (such as a drawn graphic) on the whiteboard, enabling flexible content adjustment.

[0080] Movement gestures. When a user swipes a single finger across the screen, a movement command is generated, indicating the direction and distance the whiteboard content should move on the screen. The receiving end moves the whiteboard display area according to this command, allowing the user to browse different parts of the whiteboard.

[0081] Step S102: Receive visual data sent by the second electronic device through a second channel between the two electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard.

[0082] In this embodiment, the collaborative whiteboard application of the second electronic device can call the graphical interface (such as the snapshot interface or the ReplayKit framework) of the operating system of the second electronic device to trigger a screen capture operation according to preset rules.

[0083] Preset rules may include, but are not limited to, any of the following:

[0084] First, periodic capture, for example, a preset capture frequency of 15 frames per second, to ensure continuous capture of changes in the collaborative whiteboard screen; second, trigger-based capture, which immediately triggers the capture operation the moment the user finishes their handwriting operation, ensuring that the complete operation result screen is captured.

[0085] After the capture is completed, the collaborative whiteboard application of the second electronic device can convert the captured current complete screen of the collaborative whiteboard into a video frame in YUV or RGB format. This video frame is the visual data, which contains all the current visual content of the collaborative whiteboard, including the user's completed handwriting, inserted pictures, the display effects of the operation tools, and the screen state after the gesture operation, etc., to achieve WYSIWYG screen capture.

[0086] Once the visual data has been collected, the collaborative whiteboard application sends it to the first electronic device via a second channel to ensure stable transmission of the visual data.

[0087] Step S103: Display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view.

[0088] The collaborative whiteboard application of the first electronic device adopts a two-layer view architecture, consisting of a bottom layer view (such as UIImageView or MetalView) and a transparent top layer view (such as PKCanvasView).

[0089] The bottom layer view is the basic display view of the collaborative whiteboard application. It is an opaque video rendering view, mainly used to carry and display visual data (video frames). Its display content is a baseline version of the collaborative whiteboard screen.

[0090] The transparent top-level view is a drawing view that is overlaid on the bottom-level view. It is fully transparent and is used to draw the strokes corresponding to the vector data received from the first channel in real time without affecting the display of the bottom-level view.

[0091] The bottom view is responsible for providing a complete and consistent screen baseline, while the transparent top view is responsible for providing low-latency instantaneous handwriting feedback. The two can be superimposed to form a complete display screen for the collaborative whiteboard.

[0092] Visual data can be regarded as the visual truth of the collaborative whiteboard image, which contains all the complete visual content of the current whiteboard. No matter what deviations occur in the vector strokes drawn by the first electronic device in the transparent top layer view due to transmission, rendering and other factors, they will eventually be corrected by the visual data displayed in the bottom layer view, ensuring that the collaborative whiteboard image seen by the first electronic device and the second electronic device is completely consistent, and guaranteeing the high fidelity of the image.

[0093] Step S104: In response to displaying the visual data in the bottom layer view, clean up the handwriting related to the screen content in the transparent top layer view based on the visual data.

[0094] The first electronic device employs a display architecture that overlays a bottom-level view and a transparent top-level view. The transparent top-level view is used to draw low-latency vector handwriting in real time, while the bottom-level view is used to display high-fidelity visual data (visual truth). Because there is a slight time difference between the drawing of the vector handwriting and the display of the visual data, and both contain handwriting content generated by user operations, if the handwriting in the transparent top-level view is not cleaned up, the top-level vector handwriting will overlap with the handwriting in the bottom-level visual data, resulting in a visual ghosting effect of "double handwriting." This severely impacts the user's visual experience and prevents clear display of the collaborative whiteboard.

[0095] Besides eliminating visual ghosting and ensuring a good visual experience, the cleanup operation is crucial for ensuring consistency in collaborative visuals across multiple devices. During prolonged collaborative interactions, vector data may experience minor deviations during transmission and rendering due to factors such as network fluctuations and differences in device performance. While individual deviations may be difficult to detect, their accumulation can lead to significant discrepancies between the strokes drawn in the transparent top-level view and the original operation screen of the second electronic device, thus disrupting visual consistency across multiple devices. The visual data displayed in the bottom-level view, as the visual truth, completely and accurately reflects the current state of the collaborative whiteboard. By cleaning up strokes covered by this visual data in the transparent top-level view, the accumulated deviations during vector data transmission and rendering can be completely eliminated, ensuring that the whiteboard images displayed on the first and second electronic devices are completely consistent.

[0096] Meanwhile, the cleanup operation can effectively optimize device performance and ensure smooth collaborative interaction. Vector strokes drawn in the transparent top-level view continuously occupy device memory. If not cleaned up for a long time, historical stroke data will accumulate, leading to excessive device memory usage and affecting the smooth operation of the collaborative whiteboard application. Through the cleanup operation, historical vector strokes covered by visual data can be cleared in a timely manner, eliminating the need for the device to store this redundant data for a long time, effectively reducing memory usage, ensuring the continuous and stable operation of the collaborative whiteboard application, and further improving the user's collaborative interaction experience.

[0097] In this embodiment, the low-latency first channel prioritizes the transmission of lightweight, high-frequency vector data. Since its transmission latency is optimized to an extremely low level, it can achieve instantaneous transmission of vector data. After receiving the vector data, the first electronic device immediately draws the corresponding handwriting in real time in the transparent top-level view, realizing instantaneous feedback between user operation and handwriting display, achieving a smooth writing effect, significantly improving writing fluency, and meeting the core needs of users for smooth interaction in remote collaboration scenarios.

[0098] At the same time, the high-fidelity second channel synchronously transmits the complete visual data captured and generated by the second electronic device. This visual data serves as the visual truth value of the collaborative whiteboard image. After being transmitted to the first electronic device, it is displayed in the bottom layer view. The bottom layer view serves as the display base for the transparent top layer view. The visual data displayed in the bottom layer view can accurately correct the vector strokes drawn in the transparent top layer view. Even if the vector data has slight deviations due to network fluctuations or differences in device performance during transmission and rendering, it can be quickly corrected by the visual truth value of the bottom layer view, ensuring that the whiteboard image displayed by the first electronic device and the second electronic device is completely consistent, thus thoroughly guaranteeing the consistency of the image in multi-device collaboration.

[0099] In addition, since there is a slight time difference between the vector handwriting drawing in the first channel and the visual data display in the second channel, the superposition of the two can easily produce double handwriting visual ghosting. The cleaning operation can promptly remove historical handwriting that has been covered by the underlying visual data in the transparent top layer view, which not only eliminates ghosting and improves the user's visual experience, but also completely eliminates the problem of accumulated vector data deviation.

[0100] At the same time, cleaning up redundant handwriting can reduce device memory usage, prevent data accumulation from affecting the smoothness of application operation, and ensure the long-term stable operation of collaborative whiteboard applications.

[0101] In summary, by relying on the collaborative transmission of two parallel channels, coupled with the division of labor and cooperation of dual-layer views and handwriting cleanup operations, this application can simultaneously achieve writing fluency and screen consistency, further optimizing the overall collaborative interaction experience. This enables the collaborative whiteboard to better adapt to various real-time collaborative scenarios such as remote work, online education, and sports tactical analysis, significantly improving its applicability and practicality.

[0102] As another optional embodiment of this application, this embodiment provides a synchronization method for a collaborative whiteboard. This embodiment is mainly an implementation of the first channel and the second channel. Specifically, the first channel can be a point-to-point connection channel established between the first electronic device and the second electronic device.

[0103] In this embodiment, the first channel does not rely on a remote server and directly enables direct communication between two devices, which can be adapted to collaborative scenarios within the same local area network (such as multi-device collaboration in an office, teacher-student collaboration in a classroom, etc.). Its specific establishment process may include:

[0104] This channel relies on the mDNS multicast domain name protocol to complete the automatic discovery and matching connection of local area network devices. This protocol does not require the deployment of traditional DNS servers. It can realize mutual identification of devices by relying on the local area network broadcast mechanism, accurately obtain basic information such as the IP address, device name and application running status of the other party, and lay the preconditions for point-to-point direct connection.

[0105] When the channel is established, the two electronic devices first start the collaborative whiteboard application simultaneously and enter the same collaborative room. The application automatically triggers the mDNS protocol LAN scanning mechanism, and each device actively broadcasts collaborative identity information carrying the collaborative room ID and the device's unique identifier to the LAN.

[0106] Secondly, after the two devices scan and identify each other's broadcast information and verify that they belong to the same collaborative room, they automatically create a TCP or UDP socket long connection, ultimately forming a stable and reliable point-to-point dedicated first channel.

[0107] Finally, the channel maintains a real-time connection status monitoring mechanism. Once a connection interruption is detected, such as a device leaving the local area network or an application crashing, the automatic reconnection logic is immediately triggered to ensure that the first channel remains online, meeting the core requirement of uninterrupted instantaneous transmission of vector data.

[0108] This point-to-point connection channel features low latency, high bandwidth, and no public network traffic consumption. Because it eliminates the need for remote server relay, the data transmission path is minimized, and transmission latency can be controlled at the millisecond level, perfectly matching the "high-frequency, lightweight, and real-time" transmission requirements of vector data, further improving writing fluency. The high bandwidth supports simultaneous transmission of multiple sets of vector data, avoiding data congestion. Simultaneously, direct connection within the local area network eliminates the need for public network traffic consumption, reducing user costs and making it particularly suitable for collaborative scenarios within a multi-user local area network (such as conference rooms and classrooms), enhancing the convenience of collaboration.

[0109] The second channel relies on a cloud server for data relay and interaction. It can adapt to cross-regional and wide-area network remote collaboration scenarios in different cities and network environments, perfectly meeting the needs of remote office work and cross-regional online education for remote collaborative whiteboard use. Its specific setup process can include:

[0110] The collaborative whiteboard application pre-integrates a mature RTC real-time communication software development kit. Relying on the core underlying communication capabilities of the RTC SDK, such as anti-packet loss processing, dynamic network adaptation, and audio and video encoding and decoding optimization, it specifically addresses the inherent transmission problems commonly found in wide area network scenarios, such as network fluctuations, uneven transmission latency, and poor network penetration of devices, thus laying a solid foundation for the stable transmission of visual big data.

[0111] When the channel is established, the two electronic devices first start the collaborative whiteboard application and complete the compliance authentication. Then, they send an application to the remote cloud server to join the exclusive cloud collaborative room through the integrated RTC SDK. The application message carries key verification information such as the unique collaborative room ID, the device's unique identifier, and the real-time network operation status.

[0112] Secondly, the remote cloud server performs compliance verification and identity matching on the received application information. After the verification is successful, it assigns a dedicated independent communication link to the two electronic devices, thus formally establishing the second channel with the cloud server as the transit hub.

[0113] Finally, the second channel maintains real-time link monitoring, relying on the RTC SDK to continuously and dynamically sense network parameters such as real-time network bandwidth and transmission latency, and adaptively adjusts the encoding format and transmission rate of visual data to avoid problems such as screen stuttering, screen tearing, and data loss caused by wide area network transmission from the source, ensuring continuous and stable transmission of visual data.

[0114] The second channel features strong packet loss resistance, high penetration, and support for large-scale concurrency. Packet loss resistance, achieved through the SDK's retransmission mechanism and redundant coding technology, compensates for data loss during wide area network transmission, ensuring the integrity of visual data. High penetration overcomes firewall restrictions between different networks (such as home broadband, office networks, and 4G / 5G), enabling stable connections across network devices. Support for large-scale concurrency adapts to multi-user collaborative scenarios (such as online classrooms with dozens or even hundreds of participants, and large-scale remote conferences), meeting the collaborative needs of various scenarios.

[0115] As another optional embodiment of this application, refer to Figure 2 This is a flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 3 of this application. Figure 2 As shown, the method may include, but is not limited to, the following steps:

[0116] Step S201: Monitor the link status of the first channel between the device and the second electronic device.

[0117] This embodiment can adopt a multi-dimensional independent optional monitoring mode. Each monitoring dimension is an independent and effective basis for judging the link status. The link status can be identified as long as any monitoring judgment condition is met, without the need for all monitoring indicators to be checked and met simultaneously.

[0118] The link monitoring process for the first channel may specifically include at least one of the following:

[0119] The two ends of the device periodically send heartbeat detection messages to each other to verify and provide feedback on the basic connectivity status of the link in real time.

[0120] Real-time sampling and monitoring of the connection status of the Socket physical connection.

[0121] Continuously collect and statistically analyze RTT round-trip latency data.

[0122] Step S202: In response to the link status indicating that the first channel is available, receive vector data sent by the second electronic device through the first channel.

[0123] Based on heartbeat detection and latency monitoring, when the first channel Socket connection remains in a normal connectivity state and the real-time RTT round-trip latency is consistently below the preset threshold of 100ms, it indicates that the first channel of the low-latency point-to-point direct connection is in a healthy and usable state.

[0124] At this point, the original normal collaborative transmission strategy can be maintained, with the first channel fixed as the sole main transmission channel for vector data. It continuously receives various vector data generated by the second electronic device in response to the user's whiteboard writing, drawing, and gesture operations, maintaining the millisecond-level instantaneous transmission of lightweight, high-frequency vector data. This continues the core interactive effect of low-latency writing and responsiveness in Example 1, ensuring that the smoothness of collaborative writing in stable local area network scenarios is not compromised.

[0125] The specific generation method and data composition of the vector data are completely consistent with those in Example 1, and will not be repeated here.

[0126] Step S203: In response to the link status indicating that the first channel is unavailable, receive vector data sent by the second electronic device through the second channel.

[0127] When the heartbeat detection detects that the first channel Socket connection is abnormally disconnected or fails to reconnect, or the real-time RTT round-trip latency continues to exceed the preset threshold of 100ms, or the network jitter is so severe that it cannot meet the real-time writing transmission requirements, the first channel link is immediately determined to be unavailable and cannot continue to undertake the low-latency vector data transmission task.

[0128] At this point, the automatic switching logic of the primary and backup channels can be triggered, seamlessly switching the transmission path of vector data from the first channel to the second channel. The second channel, which relies on the cloud RTC architecture and has strong resistance to packet loss and penetration, temporarily undertakes the backup transmission of vector data, ensuring that even if the first channel fails, the handwriting vector data can still be transmitted continuously and without interruption, avoiding problems such as disconnection of handwriting, drawing lag, and unresponsive operation from the source, and ensuring uninterrupted collaborative interaction.

[0129] Step S204: Draw handwriting in the transparent top-level view based on the vector data.

[0130] Regardless of whether the vector data is transmitted through the primary channel used normally or the secondary channel used as a fallback, after the first electronic device receives the complete vector data, it uniformly follows the drawing logic established in Example 1 to reproduce the corresponding handwriting, graphic annotations, and gesture operation effects in real time in the fully transparent top-level view.

[0131] The specific execution rules and view overlay logic of this step's handwriting rendering are completely consistent with step S101 of Example 1, except that the vector data transmission channel is different. The rendering effect is not affected by the channel switching, ensuring a consistent and imperceptible visual experience for the user's writing.

[0132] Step S205: Receive visual data sent by the second electronic device through a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than the transmission delay of the second channel.

[0133] Step S206: Display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view.

[0134] Step S207: In response to displaying the visual data in the bottom layer view, clean up the handwriting related to the screen content in the transparent top layer view based on the visual data.

[0135] For detailed procedures of steps S204-S207, please refer to the relevant descriptions in steps S101-S104 of Example 1, which will not be repeated here.

[0136] In this embodiment, when the local area network environment is good, the low-latency first point-to-point channel can be selected as the main transmission path for vector data by default, continuously ensuring a smooth and low-latency interactive experience for users. Once any abnormal fault is detected, such as local area network environment deterioration, first channel heartbeat disconnection, Socket connection disconnection, or RTT round-trip latency exceeding the standard, the system can automatically and seamlessly switch the vector data transmission path to the cloud-based second channel based on the RTC architecture for backup transmission without manual intervention or leaving the collaborative room. This ensures that even if the local area network environment deteriorates, users can still continuously complete bidirectional synchronous interaction of vector data and visual data through this cloud link (i.e., the second channel). Combined with the original dual-view rendering, image truth correction, and handwriting cleanup core logic, it continues to operate stably without affecting the handwriting drawing effect and the uniformity of the multi-device image. This effectively achieves the goal of never dropping the collaborative whiteboard interaction, greatly improving the network adaptability, fault tolerance performance, and operational stability of the overall collaborative system. It is fully adaptable to various complex collaborative usage scenarios such as network fluctuations, long distances, and poor local area network signals, continuously ensuring high-quality, smooth, and uninterrupted operation of various collaborative whiteboard tasks.

[0137] As another optional embodiment of this application, the synchronization method for a collaborative whiteboard provided in Embodiment 4 of this application is mainly an implementation of the above-mentioned method of receiving vector data sent by the second electronic device through the second channel. Specifically, it may include, but is not limited to, the following steps:

[0138] Step S2031: Receive vector data encapsulated in the video encoded stream corresponding to the visual data through the second channel.

[0139] When the first channel fails and vector data is switched to the second channel for transmission, the second electronic device first performs structured serialization encoding on various types of vector data, such as the real-time generated handwriting coordinate point sequence, pressure value, tool type, and gesture command, and converts them into a custom private field data format that can be embedded in the video transmission link.

[0140] Subsequently, the collaborative whiteboard application can call the RTC SDK encoding interface to embed the private fields of the serialized vector data into the preset redundant data frame segments or frame header extension positions of the YUV or RGB format visual data video encoding stream, so that the vector data and visual image frame data are merged into the same integrated composite video encoding stream without the need to open up an additional independent data transmission channel.

[0141] After the integrated encapsulation and encoding are completed, the composite video encoded stream is synchronously packaged and transmitted through the second channel relayed by the cloud server. It is sent, scheduled, and transmitted synchronously with the visual data video encoded stream throughout the entire process, and there is no timing deviation or network scheduling difference between the two data transmissions.

[0142] Correspondingly, after the first electronic device receives the integrated composite video encoding stream through the second channel, it synchronously performs data separation and parsing operations in the pre-decoding and rendering stage of the video. First, it extracts and decodes the original complete vector data from the preset private field position of the composite video encoding stream, and then sends the extracted pure visual image frame data into the bottom layer view for rendering and display. The restored standard vector data is then used to draw handwriting in real time in the transparent top layer view according to the predetermined drawing logic of Embodiments 1 and 3.

[0143] In this embodiment, by encapsulating vector data in the visual data video encoding stream for co-transmission, problems such as out-of-order data packets, asynchronous arrival times, bandwidth congestion, and difficulties in cross-link data alignment that are prone to occur in the dual-path independent transmission of the second channel in a wide area network environment are effectively avoided. Even after the first channel fails and is switched over, the timing of vector handwriting drawing and visual display can still be accurately matched, effectively alleviating secondary problems such as handwriting delay, screen misalignment, and synchronization stuttering after the switch. While simplifying the second channel transmission link scheduling logic, it further improves the collaborative synchronization accuracy and operational stability after the primary and backup channels are switched over. Together with the automatic switching mechanism in Embodiment 3, it can stably achieve the collaborative operation goal of the collaborative whiteboard never going offline.

[0144] As another optional embodiment of this application, a synchronization method for a collaborative whiteboard is provided in embodiment 5 of this application. In this embodiment, the second channel may include a first sub-channel and a second sub-channel.

[0145] Both the first and second sub-channels are built on the cloud-based RTC relay architecture of the second channel. They share the same cloud-based collaborative room identity verification and network penetration capabilities, but their data transmission links, scheduling strategies, and transmission priorities are independent of each other and do not interfere with each other, thus achieving dedicated transmission of visual big data and vector lightweight data classification without each other occupying resources.

[0146] The first sub-channel is a video media transmission sub-channel, which can be used to continuously receive various visual data sent by the second electronic device, ensuring the stable encoding, transmission and decoding rendering of the complete image frames of the collaborative whiteboard throughout the process.

[0147] The second sub-channel can be an independent dedicated data sub-channel for Data Stream, which is dedicated to the transmission of vector data under backup conditions, and does not carry visual image frame data, so as to realize the physical separation of the two types of core data and their respective transmission functions.

[0148] This embodiment is mainly one implementation of receiving vector data sent by the second electronic device through the second channel, and may specifically include, but is not limited to, the following steps:

[0149] Step S2032: Receive vector data sent by the second electronic device through the second sub-channel (e.g., Data Stream channel).

[0150] After the primary / backup switch logic is triggered by the failure of the first channel, the collaborative whiteboard application can use the media stream and data stream separation creation capability natively supported by the RTC SDK to simultaneously and in parallel create the first sub-channel video media stream link and the second sub-channel Data Stream dedicated data link on the basis of the established cloud second channel. The two sub-channels are activated simultaneously, work in parallel, and are independently scheduled, without the need for additional cloud identity verification and room matching.

[0151] The second electronic device can directly package, encrypt, and transmit vector raw data such as real-time generated handwriting coordinate point sequences, pressure values, tool types, and gesture commands through a dedicated second sub-channel without embedding or encapsulating them into a video encoding stream. The transmission is prioritized according to the dedicated scheduling rules for high-frequency, lightweight data, and is not affected or squeezed by the visual big data encoding rate, bandwidth fluctuations, or video encoding and decoding operations of the first sub-channel. At the same time, the second electronic device transmits conventional visual data independently through the first sub-channel. The two types of data are sent synchronously, independently scheduled, and do not interfere with each other.

[0152] Correspondingly, the first electronic device synchronously and in parallel listens to the data of two sub-channels. After receiving the complete vector data separately through the second sub-channel, it directly decodes and restores the original handwriting operation data without performing video stream data stripping and splitting operations. After decoding, it immediately renders and draws the handwriting in real time in the transparent top-level view according to the established drawing logic of Embodiments 1 and 3. At the same time, the first electronic device independently receives visual data through the first sub-channel and completes the screen display in the bottom-level view, and completes collaborative screen correction and synchronization in conjunction with the subsequent handwriting cleaning steps.

[0153] In this embodiment, by splitting the second channel into a dual-path splitting transmission architecture consisting of a first sub-channel and a second sub-channel, compared to the integrated encapsulation transmission method, it is possible to achieve complete physical separation of visual data and vector data transmission. This effectively avoids the situation where video encoding and decoding operations drag down the timeliness of vector data transmission and the mutual influence of abnormal single-type data transmission under the integrated encapsulation mode. Each type of data is adapted to a dedicated transmission scheduling strategy, specifically matching its own data transmission characteristics. After the first channel is switched over as a backup, the real-time performance of vector handwriting drawing and the integrity of visual display are continuously guaranteed. This effectively improves the drawing delay and screen synchronization deviation caused by data coupling transmission. While strengthening the independence and anti-interference capability of the two types of data transmission, it further enhances the stability and reliability of the main and backup channel switching and the overall operation of the collaborative whiteboard under complex wide area network conditions. In conjunction with the automatic switching mechanism in Embodiment 3, it continuously stabilizes the uninterrupted collaborative operation effect of the collaborative whiteboard.

[0154] As another optional embodiment of this application, refer to Figure 3 This is a flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 6 of this application. Figure 3 As shown, the method may include, but is not limited to, the following steps:

[0155] Step S301: Receive vector data sent by the second electronic device and a first timestamp generated on the second electronic device by the vector data through a first channel between the two electronic devices, and draw handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user operations on the collaborative whiteboard.

[0156] In this embodiment, when the second electronic device generates vector data corresponding to each writing, drawing, and gesture operation in real time, it can simultaneously collect high-precision time information from the device's local system to generate a first timestamp for the vector data corresponding to a single frame and a single handwriting. This first timestamp accurately records the original generation time of each set of vector handwriting at the sending end, and is synchronously packaged and encrypted along with the corresponding vector data, and sent to the first electronic device through the low-latency first channel.

[0157] The first timestamp can use a millisecond-level high-precision time encoding format to accurately correspond to the starting generation node of each stroke drawing, without any timestamp duplication, disorder, or delayed assignment. After receiving the vector data and the corresponding first timestamp, the first electronic device synchronously binds and stores the vector data and the corresponding time sequence identifier, and renders the strokes in real time in the transparent top-level view according to the established drawing logic. The correspondence between the strokes and the first timestamp is preserved throughout the process, providing accurate data for subsequent time sequence comparison and cleanup, without affecting the original low-latency writing and responsive interactive effect.

[0158] Step S302: Receive visual data sent by the second electronic device through a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard and a second timestamp generated on the second electronic device by the screen content. The transmission delay of the first channel is less than the transmission delay of the second channel.

[0159] In this embodiment, after each collaborative whiteboard image capture and visual data encoding operation, the second electronic device can synchronously collect high-precision time information from the local system of the current device to generate a second timestamp that uniquely matches the complete visual image frame. The second timestamp accurately records the moment when the visual true value image is captured and solidified, and is synchronously packaged with the visual image frame data and sent to the first electronic device through the second channel.

[0160] The second timestamp and the first timestamp can use the same time base encoding rules to ensure that the time sequence comparison of the two types of timestamps is accurate and effective, and there is no problem of time base misalignment.

[0161] As the global true value of the visual data, its corresponding second timestamp represents the latest cutoff time of the unified synchronization of multiple devices on the collaborative whiteboard, which defines the precise time sequence boundary for subsequent handwriting cleanup. The basic process and encoding / decoding logic of the second channel for transmitting visual data are completely consistent with Example 1.

[0162] Step S303: Display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view.

[0163] In this embodiment, the specific execution logic of rendering and displaying visual data in the underlying view, the relationship between the superposition of the two-layer view, and the function of correcting the true value of the image are completely consistent with step S103 in embodiment 1. After the first electronic device receives the visual data and the corresponding second timestamp, it synchronously completes the rendering and solidification of the underlying view image and synchronously binds and stores the correspondence between the visual data and the second timestamp. There is no need to adjust the view architecture and rendering rules, ensuring the integrity and consistency of the underlying base image.

[0164] Step S304: In response to displaying the visual data in the underlying view, obtain a second timestamp from the visual data on which the screen content was generated on the second electronic device.

[0165] After the visual data screen is rendered and displayed in the bottom view, the first electronic device can immediately retrieve the second timestamp data that has been pre-bound and stored, extract the time sequence cutoff node corresponding to the latest synchronized screen, and use it as the sole time sequence judgment standard for handwriting cleanup.

[0166] Step S304 can be executed synchronously and instantaneously after the video decoding and rendering are completed, without additional delay or time consumption, and without affecting the smoothness of collaborative interaction. Its core function is to define a precise timing threshold for the handwriting cleanup operation, replacing the original unfounded overall image matching cleanup method, and giving handwriting cleanup a clear and quantifiable execution judgment standard.

[0167] Step S305: Clean up the handwriting in the transparent top-level view where the first timestamp is earlier than or equal to the second timestamp.

[0168] The first electronic device can compare and filter the first timestamp corresponding to all drawn strokes in the transparent top-level view with the extracted second timestamp one by one, and only clean up the historical vector strokes whose first timestamp is earlier than or equal to the second timestamp. The operation content corresponding to this part of the strokes has been completely synchronized and solidified in the visual truth screen of the bottom-level view. These are redundant strokes that have been synchronized and do not need to be repeatedly superimposed and displayed.

[0169] For the latest real-time handwriting with a first timestamp later than the second timestamp, it is determined to be a newly added handwriting that has not yet been synchronized and solidified and needs to be displayed in real time. It is then completely preserved without any cleaning.

[0170] This time-series differentiated cleaning method completely abandons the traditional one-size-fits-all cleaning model and accurately distinguishes redundant and invalid handwriting from valid real-time handwriting.

[0171] Steps S304-S305 are a specific implementation of step S104 in Example 1.

[0172] In this embodiment, a dual-time-series verification mechanism is added, consisting of a first time stamp of vector data and a second time stamp of visual data. This enables precise and targeted erasure of handwriting in the transparent top-level view, effectively solving problems such as over-erasure, untimely erasure, and overlapping of old and new handwriting that are common in traditional overall erasure methods. It completely eliminates the risk of handwriting ghosting and the accumulation of vector data deviations from the root cause of the time sequence. This not only continuously preserves the low-latency real-time feedback effect of the latest handwriting, ensuring that the smoothness of writing does not diminish, but also accurately eradicates redundant historical handwriting that has already been synchronized, maintaining a high degree of consistency in multi-device collaborative visuals. Without increasing the computing power burden on devices or affecting the real-time performance of collaborative interaction, it further improves the operational stability and synchronization accuracy of the collaborative whiteboard under long-term, continuous, and high-frequency collaborative interaction conditions. This makes it suitable for core use cases of long-term, high-frequency continuous collaboration, such as high-intensity remote work, routine online teaching, and professional sports tactical review.

[0173] As another optional embodiment of this application, refer to Figure 4 This is a flowchart illustrating a synchronization method for a collaborative whiteboard provided in Embodiment 7 of this application. This embodiment is mainly an implementation of step S102 in Embodiment 1 above.

[0174] When transmitting visual data through the second channel of a collaborative whiteboard, a fixed frame rate encoding transmission mode is commonly used. Regardless of whether the collaborative whiteboard is currently in a static, inactive state, in a state of small-scale writing and drawing, or in a state of large-scale screen refresh, image frame data is continuously acquired, encoded, and transmitted at a uniform fixed frame rate. This fixed frame rate transmission method has significant technical drawbacks. When the whiteboard is stationary for a long time without any user operation, the continuous high-frequency transmission of repetitive and unchanging visual image frames will result in the ineffective waste of public network bandwidth resources and device encoding and decoding computing resources. Especially in scenarios with large-scale multi-user collaboration and limited 4G / 5G mobile networks, it can easily lead to problems such as network bandwidth congestion, data transmission congestion, and soaring device power consumption.

[0175] In high-dynamic operation scenarios such as large-scale whiteboard refreshes, batch image insertion, and overall page switching, a fixed frame rate cannot match the demand for rapid screen changes, resulting in visual image lag, multi-device screen synchronization gaps, and refresh stuttering. It fails to simultaneously meet the dual requirements of efficient bandwidth utilization and high-fidelity screen synchronization in high-dynamic scenarios. To address these specific technical issues, this embodiment 7, based on the core infrastructure of dual-channel transmission, dual-layer view rendering, and precise handwriting cleanup in embodiment 1, makes no changes to the low-latency vector data transmission logic, top-level and bottom-level view display architecture, or core handwriting cleanup steps. It only specifically optimizes the transmission frame rate control mechanism for the second-channel visual data, introducing a core strategy of dynamic image change rate perception and variable frame rate adaptive parameter adjustment. The transmission frame rate is dynamically adjusted according to the real-time screen changes of the collaborative whiteboard, achieving the technical effects of static bandwidth saving, dynamic image quality preservation, and stable synchronization across all operating conditions. Specifically, this may include, but is not limited to, the following detailed implementation steps:

[0176] Step S1021: Receive visual data transmitted by the second electronic device at a variable frame rate through the second channel between the second electronic device and the second electronic device.

[0177] The variable frame rate can be dynamically adjusted according to the rate of change of the collaborative whiteboard's image.

[0178] The specific criteria for determining the rate of change in the image and the corresponding rules for adjusting variable frame rates can be refined as follows:

[0179] First, the whiteboard is adapted and adjusted for static, unchanging conditions. When the screen change rate monitoring module detects no pixel changes on the collaborative whiteboard canvas for a preset duration (uniformly preset to 3 seconds), and no user actions such as writing, drawing, gesture zooming, image insertion, or page switching, it determines that the collaborative whiteboard is in a static, inactive state. At this time, a variable frame rate degradation strategy is triggered simultaneously, dynamically reducing the visual data transmission frame rate to an ultra-low frame rate of 1fps. In special scenarios, visual data acquisition and transmission can be temporarily stopped directly, with only the first channel vector data standby transmission link remaining.

[0180] Under this condition, there is no need to continuously transmit the same whiteboard image frames, which minimizes the amount of data transmitted through the second channel, greatly saves public network bandwidth, reduces the encoding computing power consumption of the second electronic device and the decoding and rendering power consumption of the first electronic device, and avoids unnecessary resource occupation.

[0181] Second, the whiteboard adapts and adjusts for minor dynamic writing conditions. When the screen change rate monitoring module detects slight operations such as single-stroke writing, small-line drawing, or local annotation, and the canvas only shows continuous changes in a small area of ​​pixels, with the screen change rate within a preset low threshold range, the collaborative whiteboard is determined to be in a normal minor dynamic collaborative working condition. At this time, a variable frame rate adaptation strategy is triggered simultaneously to maintain the visual data transmission frame rate at the standard normal frame rate of 15fps, matching the needs of normal collaborative scenarios such as daily remote office annotation, classroom knowledge point annotation, and simple tactical marking. This balances the visual screen synchronization and continuity during normal writing with the reasonable utilization of bandwidth resources, avoiding screen refresh lag and excessive bandwidth consumption.

[0182] Third, the whiteboard adapts and adjusts for large-scale refresh operations. When the screen change rate monitoring module detects that the user is performing large-scale refresh operations such as batch image insertion, full-page canvas switching, global scaling and rotation, or large-area graphic drawing, resulting in rapid changes in a large range of pixel positions on the canvas and batch additions and removals of layer elements, the screen change rate triggers a preset high threshold range, indicating that the collaborative whiteboard is in a high-dynamic, high-refresh-rate condition. At this time, a variable frame rate instantaneous upscaling strategy is triggered simultaneously, instantly increasing the visual data transmission frame rate to a high frame rate of 30fps. This allows for short-term, continuous high-frame-rate acquisition and transmission of complete visual image frames, accurately capturing the details of the entire process of large-scale screen changes on the whiteboard. This avoids problems such as visual screen update delays, multi-device screen synchronization misalignment, refresh stuttering, and screen flickering under high-dynamic operations, ensuring high-fidelity consistency of the screen under high-dynamic operation scenarios.

[0183] After the second electronic device completes the dynamic acquisition, encoding and packaging of variable frame rate visual data, it is stably transmitted to the first electronic device through the second channel relying on the cloud RTC architecture. The first electronic device, as the receiving end, does not need to be adapted to complex frame rate decoding configuration. It can automatically be compatible with visual image frames of different frame rates based on the original underlying view rendering logic, and normally complete a series of subsequent operations such as the baseline display of the underlying view screen and the subsequent cleaning of the handwriting in the transparent top layer view. The variable frame rate adjustment process does not affect the display effect of the dual-layer view overlay, does not interfere with the low-latency writing experience of the first channel vector data, and is seamlessly compatible with the core synchronization logic of Example 1.

[0184] In this embodiment, a variable frame rate visual data transmission mechanism based on dynamic perception of screen change rate addresses the dilemma of "bandwidth waste" and "dynamic synchronization lag" in collaborative whiteboard visual data transmission compared to the traditional fixed frame rate transmission mode. It provides differentiated adaptation to three core collaborative operating conditions: static whiteboard, routine small-scale operations, and large-scale refreshes. During the whiteboard's standby phase, ultra-low frame rate or a stop-transmission mode conserves bandwidth and device computing power, adapting to resource-constrained scenarios such as mobile networks and multi-user concurrency.

[0185] During regular collaborative writing, a standard balanced frame rate is used to balance smooth interaction and resource utilization, adapting to the needs of daily office and teaching use. During high-dynamic screen operation, instantaneous upsampling ensures screen synchronization accuracy and display continuity, eliminating synchronization stuttering during large operations.

[0186] This embodiment retains the core advantages of all the aforementioned embodiments, such as low-latency writing, high-fidelity synchronization, automatic channel switching, and precise handwriting cleanup. It further enhances the network adaptability and resource utilization efficiency of the collaborative whiteboard, reduces application power consumption and traffic consumption, and fully adapts to various remote collaborative whiteboard usage scenarios with different network conditions, different operational intensities, and different collaborative scales, significantly optimizing the overall user experience for long-term continuous collaborative operation.

[0187] The synchronization device for the collaborative whiteboard provided in this application will be described below. The synchronization device for the collaborative whiteboard described below can be referred to in correspondence with the synchronization method for the collaborative whiteboard described above.

[0188] The synchronization device for a collaborative whiteboard may include:

[0189] A first receiving module is configured to receive vector data sent by a second electronic device via a first channel between the second electronic device and the second electronic device; the vector data is generated by the second electronic device in response to user operations on a collaborative whiteboard.

[0190] The first drawing module is used to draw strokes in a transparent top-level view based on the vector data.

[0191] The second receiving module is configured to receive visual data sent by the second electronic device through a second channel between the second electronic device and the second electronic device. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than the transmission delay of the second channel.

[0192] The display module is used to display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view.

[0193] A cleanup module is used to clean up handwriting related to the screen content in the transparent top-level view in response to the display of the visual data in the bottom-level view.

[0194] In this embodiment, the first channel can be a point-to-point connection channel established between the first electronic device and the second electronic device, and the second channel can be a connection channel established between the first electronic device and the second electronic device through a remote server.

[0195] In this embodiment, the synchronization device for the collaborative whiteboard may further include:

[0196] The monitoring module is used to monitor the link status of the first channel between the second electronic device and the second electronic device.

[0197] The first receiving module can be specifically used to receive vector data sent by the second electronic device through the first channel in response to the link state indicating that the first channel is available;

[0198] The second receiving module can be specifically used to receive vector data sent by the second electronic device through the second channel in response to the link status indicating that the first channel is unavailable.

[0199] The second receiving module receives vector data sent by the second electronic device through the second channel, specifically including:

[0200] The second channel receives vector data encapsulated in the video encoded stream corresponding to the visual data.

[0201] The second channel may include a first sub-channel and a second sub-channel; the first sub-channel may be used to receive the visual data; the second receiving module receives vector data sent by the second electronic device through the second channel, specifically including:

[0202] Vector data sent by the second electronic device is received through the second sub-channel.

[0203] In this embodiment, the first receiving module can also be used for:

[0204] The first time stamp generated by the vector data on the second electronic device is received through the first channel.

[0205] The visual data may further include a second timestamp generated on the second electronic device for the image content.

[0206] The cleanup module can be used specifically for:

[0207] In response to displaying the visual data in the underlying view, a second timestamp of the screen content being generated on the second electronic device is obtained from the visual data;

[0208] Clean up the handwriting in the transparent top-level view that corresponds to vector data with the first timestamp being earlier than or equal to the second timestamp.

[0209] In this embodiment, the second receiving module can specifically be used for:

[0210] Visual data transmitted by the second electronic device at a variable frame rate is received via the second channel between the second electronic device and the second electronic device.

[0211] The variable frame rate is dynamically adjusted according to the rate of change of the collaborative whiteboard's image.

[0212] In another embodiment of this application, a first electronic device is provided, referring to Figure 5 The first electronic device may include:

[0213] The memory 100 is used to store computer programs.

[0214] Processor 200 is configured to execute the computer program to enable the first electronic device to perform the following method steps:

[0215] The system receives vector data sent by the second electronic device via a first channel and draws handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard.

[0216] Visual data transmitted by the second electronic device is received via a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than that of the second channel.

[0217] The received visual data is displayed in a bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view;

[0218] In response to displaying the visual data in the underlying view, the handwriting related to the screen content in the transparent top-level view is cleaned up based on the visual data.

[0219] In another embodiment of this application, a synchronization system for a collaborative whiteboard is provided, which may include:

[0220] First electronic device and second electronic device;

[0221] The first electronic device and the second electronic device are connected by a communication link and have a parallel and independent first channel and a second channel. The transmission delay of the first channel is less than the transmission delay of the second channel.

[0222] The second electronic device is configured to generate vector data in response to user operations on the collaborative whiteboard, send the vector data to the first electronic device through the first channel, and generate visual data containing image frames based on the real-time screen content of the collaborative whiteboard, and send the visual data to the first electronic device through the second channel.

[0223] The first electronic device is configured to receive the vector data through the first channel and draw handwriting in a transparent top-level view based on the vector data; receive the visual data through the second channel and display the visual data in a bottom-level view that serves as the display base for the transparent top-level view; and, in response to the display of the visual data in the bottom-level view, clean up handwriting related to the content of the collaborative whiteboard screen in the transparent top-level view based on the visual data.

[0224] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0225] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0226] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0227] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A synchronization method for a collaborative whiteboard, applied to a first electronic device, characterized in that, include: Through a first channel between the device and the second electronic device, vector data sent by the second electronic device is received, and handwriting is drawn in a transparent top-level view based on the vector data; The vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard; Visual data transmitted by the second electronic device is received via a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than that of the second channel. The received visual data is displayed in a bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view; In response to displaying the visual data in the underlying view, the handwriting related to the screen content in the transparent top-level view is cleaned up based on the visual data.

2. The synchronization method for the collaborative whiteboard according to claim 1, characterized in that, The first channel is a point-to-point connection channel established between the first electronic device and the second electronic device, and the second channel is a connection channel established between the first electronic device and the second electronic device through a remote server.

3. The synchronization method for the collaborative whiteboard according to claim 1, characterized in that, Before receiving vector data sent by the second electronic device through the first channel between the two electronic devices, the method further includes: Monitor the link status of the first channel between the device and the second electronic device; In response to the link status indicating that the first channel is available, vector data transmitted by the second electronic device is received through the first channel; In response to the link status indicating that the first channel is unavailable, vector data transmitted by the second electronic device is received via the second channel.

4. The synchronization method for the collaborative whiteboard according to claim 3, characterized in that, Receiving vector data sent by the second electronic device through the second channel includes: The second channel receives vector data encapsulated in the video encoded stream corresponding to the visual data.

5. The synchronization method for the collaborative whiteboard according to claim 3, characterized in that, The second channel includes a first sub-channel and a second sub-channel; the first sub-channel is used to receive the visual data; Receiving vector data sent by the second electronic device through the second channel includes: Vector data sent by the second electronic device is received through the second sub-channel.

6. The synchronization method for the collaborative whiteboard according to claim 1, characterized in that, The synchronization method for the collaborative whiteboard also includes: The first time stamp generated by the vector data on the second electronic device is received through the first channel; The visual data also includes: a second timestamp generated by the image content on the second electronic device; The response to displaying the visual data in the bottom layer view, and the cleaning up of handwriting related to the screen content in the transparent top layer view based on the visual data, includes: In response to displaying the visual data in the underlying view, a second timestamp of the screen content being generated on the second electronic device is obtained from the visual data; Clean up the handwriting in the transparent top-level view that corresponds to vector data with the first timestamp being earlier than or equal to the second timestamp.

7. The synchronization method for the collaborative whiteboard according to claim 1, characterized in that, Receiving visual data transmitted by the second electronic device through a second channel between the two devices includes: Visual data transmitted by the second electronic device at a variable frame rate is received via the second channel between the second electronic device and the second electronic device. The variable frame rate is dynamically adjusted according to the rate of change of the collaborative whiteboard's image.

8. A synchronization device for a collaborative whiteboard, characterized in that, include: The first receiving module is configured to receive vector data sent by the second electronic device through a first channel between the second electronic device and the second electronic device; The vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard; The first drawing module is used to draw strokes in a transparent top-level view based on the vector data; The second receiving module is configured to receive visual data sent by the second electronic device through a second channel between the second electronic device and the second electronic device, the visual data including image frames generated based on the screen content of the collaborative whiteboard; the transmission delay of the first channel is less than the transmission delay of the second channel; The display module is used to display the received visual data in the bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view; A cleanup module is used to clean up handwriting related to the screen content in the transparent top-level view in response to the display of the visual data in the bottom-level view.

9. A first electronic device, characterized in that, include: Memory is used to store computer programs; The processor is configured to execute the computer program to enable the first electronic device to perform the following method steps: The system receives vector data sent by the second electronic device via a first channel and draws handwriting in a transparent top-level view based on the vector data; the vector data is generated by the second electronic device in response to user actions on the collaborative whiteboard. Visual data transmitted by the second electronic device is received via a second channel between the first and second electronic devices. The visual data includes image frames generated based on the screen content of the collaborative whiteboard. The transmission delay of the first channel is less than that of the second channel. The received visual data is displayed in a bottom layer view, wherein the bottom layer view constitutes the display base of the transparent top layer view; In response to displaying the visual data in the underlying view, the handwriting related to the screen content in the transparent top-level view is cleaned up based on the visual data.

10. A synchronization system for a collaborative whiteboard, characterized in that, include: First electronic device and second electronic device; The first electronic device and the second electronic device are connected by a communication link and have a parallel and independent first channel and a second channel. The transmission delay of the first channel is less than the transmission delay of the second channel. The second electronic device is configured to generate vector data in response to user operations on the collaborative whiteboard, send the vector data to the first electronic device through the first channel, and generate visual data containing image frames based on the real-time screen content of the collaborative whiteboard, and send the visual data to the first electronic device through the second channel. The first electronic device is configured to receive the vector data through the first channel and draw handwriting in a transparent top-level view based on the vector data; The visual data is received through the second channel and displayed in the bottom view, which serves as the display base for the transparent top-level view. In response to the display of the visual data in the underlying view, the system cleans up the handwriting related to the content of the collaborative whiteboard screen in the transparent top-level view based on the visual data.

Images