A multi-person painting method and system capable of realizing real-time synchronization and low-delay transmission

Abstract

The application discloses a multi-person drawing method and system capable of realizing real-time synchronization and low-delay transmission. The method comprises the following steps: a sending end collects touch points when a user continuously draws strokes, and divides the drawing segments according to the direction angle change value, the pressure difference value or the brush parameter switching result; the reconstructable error value is calculated according to the trajectory, pressure, brush and layer characteristics of each drawing segment, so as to distinguish low-error segments and high-error segments. The fast display data is generated for the low-error segments, and the fast display data and the supplementary detail data are generated for the high-error segments, and the data is sent to a service end through a real-time sending queue and a detail sending queue respectively. The service end assigns an operation serial number and forwards the data, and a receiving end temporarily renders in sequence, restores the corresponding layer tile state after the supplementary detail data arrives, and re-renders the affected segments, so that the synchronization delay is reduced and the picture consistency is improved.

Description

Technical Field

[0001] This application relates to the field of multi-person painting data transmission control technology, specifically a multi-person painting method and system that can achieve real-time synchronization and low-latency transmission. Background Technology

[0002] With the widespread adoption of tablets, styluses, and online collaboration tools, multi-user online painting is increasingly being used in digital creation, remote teaching, and design reviews. Existing multi-user painting systems typically involve the client collecting touch points, pressure values, brush parameters, and layer information generated during the user's drawing process, and then synchronizing this data with other participating terminals via the server, enabling multiple users to collaboratively draw on the same canvas. However, in practical applications, continuous strokes often involve a large number of sampling points, and complex brushes also involve changes in parameters such as texture, transparency, scattering, and blending modes. Transmitting the complete drawing data in real time can easily consume high bandwidth and cause transmission queuing and display delays during network fluctuations or when multiple users are drawing concurrently.

[0003] In existing technologies, to reduce transmission pressure, handwriting data is typically processed by reducing the sampling frequency, compressing trajectory data, or transmitting only key points. While these methods can reduce the amount of data, for handwriting with significant curvature changes, obvious pressure abrupt changes, or complex brush effects, the receiving end is prone to problems such as trajectory deviation, inconsistent stroke thickness, and texture distortion when reconstructing from simplified data. Furthermore, when multiple users are drawing in the same or adjacent canvas areas, the lack of reliable operation sequence control and local state recovery mechanisms can lead to inconsistent display results on different terminals. Therefore, existing multi-user drawing synchronization technologies still struggle to simultaneously achieve low-latency display, bandwidth usage control, and final image consistency. Summary of the Invention

[0004] The purpose of this application is to provide a multi-user painting method and system that can achieve real-time synchronization and low-latency transmission, so as to solve the problems mentioned in the background art.

[0005] According to one aspect of this application, a multi-person drawing method that enables real-time synchronization and low-latency transmission is provided, comprising the following steps: The sending end collects touch points while the user draws continuously, and divides the collected touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. For each drawn segment, the reconstructable error value is calculated based on the trajectory, pressure, brush and layer features of the drawn segment, and compared with the reconstructable error threshold to obtain a low error segment or a high error segment. For low-error segments, fast display data is generated; for high-error segments, fast display data and supplementary detailed data are generated. The sending end adds the fast display data to the real-time sending queue and the supplementary detailed data to the detailed sending queue before sending it to the server. The server assigns operation sequence numbers to drawing operations and forwards them for quick display of data and supplementary detailed data; The receiving end performs temporary rendering based on the operation sequence number and fragment number, according to the quick display data, and saves the layer tile state before rendering. After receiving the supplementary detail data, it searches for the layer tile state based on the handwriting identifier, fragment number, layer identifier, and tile range, restores the affected tiles to the layer tile state, and then re-renders the corresponding drawing fragment according to the supplementary detail data.

[0006] Preferably, dividing the collected touch points into drawing segments includes: when the change in direction angle between adjacent touch points exceeds the direction segmentation threshold, or the pressure difference exceeds the pressure segmentation threshold, or the brush parameters are switched, the current touch point is determined as the end point of the previous drawing segment and the start point of the next drawing segment; when the number of touch points contained in a drawing segment reaches a preset upper limit, the touch points collected when the preset upper limit is reached are determined as the end point of the drawing segment and the start point of the next drawing segment.

[0007] Preferably, the reconstructable error value is obtained by combining the normalized value of the average curvature of the fragment, the normalized value of the average distance between adjacent touch points, the normalized value of the pressure change rate, the normalized value of the effective radius of the brush, the normalized value of the transparency change range, the brush texture sensitivity coefficient, and the layer blending influence coefficient according to their corresponding weights. The reconstructable error threshold is adjusted based on the number of data packets in the real-time transmission queue that have not been acknowledged by the server and the average round-trip time of the most recent transmissions. When the number of data packets that have not been acknowledged by the server exceeds the queue length threshold, or the average round-trip time exceeds the delay threshold, the reconstructable error threshold is adjusted to a threshold greater than the current value. When neither of these thresholds is exceeded, the reconstructable error threshold is restored to the default value or adjusted to a threshold less than the current value.

[0008] Preferably, the quick display data includes start point coordinates and pressure value, end point coordinates and pressure value, intermediate key touch points and their pressure values, brush identifier, brush version identifier, deterministic rendering seed, layer identifier, handwriting identifier, fragment number, tile range, and quick data identifier. The receiving end identifies the quick display data based on the quick data identifier, determines the temporary rendering order according to the operation sequence number and fragment number, interpolates intermediate trajectory points based on the start point coordinates, end point coordinates, intermediate key touch points and their pressure values, and performs temporary rendering in conjunction with the brush identifier, brush version identifier, deterministic rendering seed, and layer identifier.

[0009] Preferably, the supplementary detail data includes the intermediate sampling point coordinate sequence, pressure value sequence, speed change information between touch points, local parameters, handwriting identifier, segment number, layer identifier, tile range, and detail data identifier within the drawn segment; the receiving end identifies the supplementary detail data based on the detail data identifier, and uses the intermediate sampling point coordinate sequence, pressure value sequence, speed change information, and local parameters as input for re-rendering the corresponding drawn segment.

[0010] Preferably, the sending end first sends the fast display data in the real-time sending queue; when there is no data to be sent in the real-time sending queue, the sending end retrieves supplementary detail data from the detail sending queue; the detail sending queue is arranged in descending order according to the reconstructable error value corresponding to the supplementary detail data and sent in this order.

[0011] Preferably, after receiving the fast display data and supplementary detail data, the server retains the association between the operation sequence number and the handwriting identifier, fragment number, layer identifier, and tile range. For the fast display data, if two drawing operations are located in different layer partitions or affect different tile ranges, the server forwards them in parallel. If two drawing operations affect the same tile range within the same layer, the server determines the forwarding order according to the operation sequence number.

[0012] Preferably, the server receives the currently viewed canvas area and zoom level from the receiver, matches the tile range of the supplementary detail data to be forwarded with the currently viewed canvas area, and obtains a visual matching result. The server generates a forwarding order based on the reconstructable error value, visual matching result, zoom level, length of the sending queue facing the receiver, most recent round-trip time, number of packet loss retransmissions, and confirmed operation sequence number, and forwards the supplementary detail data to the receiver according to the forwarding order.

[0013] Preferably, after the receiving end successfully finds the layer tile status, it restores the affected tile to the layer tile status and re-renders the corresponding drawing fragment according to the supplementary detail data; when there is a subsequent drawing fragment that has been temporarily rendered and affects the same affected tile after the corresponding drawing fragment, the receiving end re-renders the subsequent drawing fragment according to the operation sequence number and fragment number.

[0014] In another aspect, this application also provides a multi-user painting system capable of real-time synchronization and low-latency transmission, comprising: The system comprises a transmitter, a server, and a receiver; wherein the transmitter includes a touch point acquisition and segmentation module, an error assessment module, and a data generation and transmission module. The touch point acquisition and segmentation module is used to acquire touch points when the user draws continuously, and divide the acquired touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. The error evaluation module is used to calculate the reconstructable error value for each drawn segment based on the trajectory, pressure, brush and layer features of the drawn segment, and compare the reconstructable error value with the reconstructable error threshold to obtain a low error segment or a high error segment. The data generation and sending module is used to generate fast display data for low error segments, generate fast display data and supplementary detail data for high error segments, add the fast display data to the real-time sending queue, and add the supplementary detail data to the detail sending queue before sending it to the server. The server includes a synchronization forwarding module, which is used to assign operation sequence numbers to drawing operations and forward the fast display data and the supplementary detail data. The receiving end includes a temporary rendering module, a tile state saving module, and a detail re-rendering module; The temporary rendering module is used to perform temporary rendering based on the operation sequence number and fragment number, according to the fast display data. The tile state saving module is used to save the layer tile state before the temporary rendering module performs temporary rendering. The detailed re-rendering module is used to, after receiving supplementary detailed data, find the layer tile state based on the handwriting identifier, fragment number, layer identifier, and tile range in the supplementary detailed data, restore the affected tiles to the layer tile state, and re-render the corresponding drawing fragment according to the supplementary detailed data.

[0015] This application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-user painting method described above, which enables real-time synchronization and low-latency transmission.

[0016] In another aspect, this application provides a storage medium having stored computer program instructions thereon, which can be executed by a processor to implement the multi-user drawing method described above, which enables real-time synchronization and low-latency transmission.

[0017] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-person painting method described above, which enables real-time synchronization and low-latency transmission.

[0018] This application divides continuous handwriting into segments at the sending end and calculates reconstructable error values ​​by combining trajectory, pressure, brush, and layer features. This enables the system to distinguish between drawing segments that can be approximately reconstructed from key points and those requiring additional details, thus avoiding the indiscriminate real-time transmission of all touch sampling data. For low-error segments, only fast display data is sent; for high-error segments, supplementary detail data is sent. Separate scheduling using real-time and detail sending queues allows the receiving end to prioritize obtaining the approximate shape of the handwriting, reducing remote display waiting time. The server assigns operation sequence numbers to drawing operations and forwards supplementary detail data based on tile range, visible area, scaling ratio, and network status, which helps maintain a stable drawing order within the area during concurrent drawing by multiple users. The receiving end saves the layer tile state before temporary rendering. Upon receiving supplementary detail data, it only restores and re-renders the affected tiles, reducing the overhead of redrawing the entire canvas and improving the image restoration accuracy and display consistency across terminals in complex brush scenes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a multi-person drawing method that enables real-time synchronization and low-latency transmission, provided as an embodiment of this application.

[0021] Figure 2 This is a schematic diagram illustrating the process of generating and sending quick display data and supplementary detailed data according to embodiments of this disclosure.

[0022] Figure 3 This is a schematic diagram of the server-side forwarding and sequence control process provided according to an embodiment of this disclosure.

[0023] Figure 4 This is a schematic diagram illustrating the process of re-rendering the corresponding drawing segment at the receiving end according to an embodiment of this disclosure.

[0024] Figure 5 This is a schematic diagram of a multi-user painting system that enables real-time synchronization and low-latency transmission, as provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] According to embodiments of this disclosure, a multi-user painting method is provided that enables real-time synchronization and low-latency transmission. This method is applicable to scenarios where multiple users simultaneously create digital paintings on the same canvas via network connection. Each user runs a painting client on their respective terminal device, and the server handles the unified scheduling and synchronous forwarding of the painting data. The terminal devices include tablets, smartphones, personal computers, and other devices with touch or handwriting input capabilities. The touch sampling frequency, screen resolution, hardware performance, and available network bandwidth of each device may differ. This method divides the handwriting generated during the painting process into segments and assesses the degree of reconstruction, distinguishes the transmission priority of different segments, and corrects local tile areas at the receiving end. This achieves low-latency preview and consistent restoration of the final image for multi-user collaborative painting without transmitting bitmaps.

[0028] The above scenario involves a sender, a server, and a receiver. It can be understood that the sender and receiver can be interchanged, or the client can be both a sender and a receiver. The sender includes an input control module, a brush system, a layer system, a rendering and display module, a real-time communication module, and a drawing fragment analysis unit. The receiver client includes a structured data scheduling rendering module, a rendering and display module, and a local re-rendering cache unit. The server includes a collaborative synchronization control module and a data storage module. The canvas is logically divided into multiple rectangular tile areas, each tile corresponding to a fixed rectangular area on the canvas. The tile size can be configured according to the canvas resolution and terminal performance. The tile is the basic operation unit for subsequent local rendering, state caching, and supplementary correction.

[0029] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the multi-user painting method described in this application, which enables real-time synchronization and low-latency transmission. It should be noted that these embodiments are merely illustrative of this application and not intended to limit its scope of protection. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0030] like Figure 1 As shown in the figure, this application discloses a multi-person painting method that can achieve real-time synchronization and low-latency transmission, including the following method steps: S1, the sending end collects touch points when the user draws continuously, and divides the collected touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. S2, For each drawn segment, calculate the reconstructable error value based on the trajectory, pressure, brush and layer features of the drawn segment, and compare it with the reconstructable error threshold to obtain a low error segment or a high error segment. S3 generates fast display data for low-error segments and generates fast display data and supplementary detail data for high-error segments; the sending end adds the fast display data to the real-time sending queue and the supplementary detail data to the detail sending queue before sending it to the server. S4, the server assigns operation sequence numbers to the drawing operations and forwards them to quickly display data and supplement detailed data; S5, the receiving end temporarily renders according to the operation sequence number and fragment number based on the fast display data, and saves the layer tile state before rendering; based on the received supplementary detail data, it searches for the layer tile state according to the handwriting identifier, fragment number, layer identifier and tile range, restores the affected tiles to the layer tile state, and then re-renders the corresponding drawing fragment according to the supplementary detail data.

[0031] In some embodiments, for step S1, the sending client continuously collects touch points through the input control module during the user's continuous drawing of handwriting. Each touch point contains at least coordinate position and corresponding pressure value. When generating speed change information, each touch point also contains a collection timestamp or a sampling time interval between adjacent touch points. The drawing segment analysis unit performs segmentation processing on the collected touch points during the touch point collection process, without waiting for the entire stroke to be drawn.

[0032] Specifically, the segmentation rules include: the drawing segment analysis unit sequentially checks the change in direction angle, pressure difference, and whether brush parameters have changed between adjacent touch points; specifically, when the change in direction angle between adjacent touch points exceeds the direction segmentation threshold, the current touch point is determined as the end point of the previous drawing segment and also as the starting point of the next drawing segment; when the pressure difference exceeds the pressure segmentation threshold, the same segmentation process is performed; when brush parameters change, segmentation is also performed at the switching point, and any one of the above three conditions is met to trigger segmentation.

[0033] Furthermore, when the number of touch points contained in a drawing segment reaches a preset upper limit, regardless of whether the above conditions are met, the drawing segment analysis unit will determine the touch points collected when the preset upper limit is reached as the end point of the drawing segment and the starting point of the next drawing segment. The purpose is to control the amount of data in a single segment and avoid excessively long single segments that lead to coarse granularity in subsequent transmission and processing. The direction segmentation threshold and pressure segmentation threshold can be configured to system preset values ​​or automatically determined according to the characteristics of the current brush type. For brushes with rich texture details and sensitive to trajectory changes, the segmentation threshold can be set appropriately small to obtain finer segment division, while it can be appropriately enlarged for smooth and simple brushes.

[0034] In some embodiments, for step S2, for each formed drawing segment, the drawing segment analysis unit calculates a reconstructable error value based on the trajectory, pressure, brush, and layer features of the drawing segment; the reconstructable error value represents the degree of deviation that may occur between the rendering result of the receiving client reconstructing the handwriting of the segment by interpolation based on only a few key touch points and the result of the sending end rendering using all touch points; the value ranges from 0 to 1, and the larger the value, the less accurate the reconstruction by interpolation is, and the more additional data is needed for correction.

[0035] The reconstructable error value is obtained by weighting the following seven factors according to their corresponding weights:

[0036] in, The average curvature normalization value of the segment is calculated by taking the average of the changes in the directional angle between the vectors formed by the adjacent touch points within the drawn segment, and then dividing it by the preset curvature normalization reference value. This value reflects the curvature of the handwriting trajectory. The more obvious the curvature, the greater the deviation of the receiver's interpolation from the actual trajectory. The average spacing between adjacent touch points is normalized by dividing the mean distance between adjacent touch points within the segment by a preset spacing normalization benchmark. A larger spacing means a longer interpolation span and more missing information in the middle. The normalized value of the pressure change rate is obtained by dividing the maximum absolute value of the pressure difference between adjacent touch points within the segment by the preset pressure change normalization benchmark. The more drastic the pressure change, the greater the deviation in the receiver's reproduction of the stroke thickness and transparency. The brush effective radius normalization value is obtained by dividing the current brush size setting by the preset radius normalization reference value. The larger the brush radius, the wider the range of pixels affected by the same trajectory deviation. The normalized value for the transparency variation is obtained by dividing the difference between the maximum and minimum transparency values ​​within the segment by the preset normalized transparency variation benchmark. The five normalized values ​​are obtained by dividing the corresponding original feature value by its respective preset normalization benchmark value, where each normalization benchmark value is a preset constant. If any normalized value, when divided by its corresponding original feature value, is greater than one, the normalized value is set to 1; if the result is less than zero, the normalized value is set to 0. , , , and All are limited to the range of 0-1; This is the brush texture sensitivity coefficient, determined by the brush system based on brush parameters, and ranges from 0 to 1. For brushes containing texture maps, scattered point distributions, graininess, or wet edge effects, the texture sensitivity coefficient is higher; for rounded, smooth brushes, the coefficient is lower. This coefficient is automatically generated by the brush system during brush creation or modification based on the type and value of the brush parameters. Specifically, it is determined from the pre-established mapping relationship between brush parameters and brush texture sensitivity coefficients, ensuring that the same brush parameters correspond to the same... value; This is the layer blending influence coefficient, determined by the blending mode of the layer containing the drawn fragment, and its value ranges from 0 to 1. For blending modes sensitive to color overlay, such as Multiply and Overlay, the layer blending influence coefficient has a higher value; for normal blending modes, the value is lower. This coefficient can be preset by the system for each blending mode, specifically determined from the preset mapping relationship between blending modes and layer blending influence coefficients, ensuring that the same blending mode corresponds to the same... value; Among them, the weight is to Each value is a positive number between 0 and 1, and the sum of all weights is 1. Weight values ​​can be preset according to the characteristics of the brush type; different brush types can be configured with different weight combinations. Optionally, weight values ​​can also be calibrated by statistically analyzing the interpolation and restoration deviations of various brushes under different drawing conditions, and the calibrated weight sets are stored in a preset mapping relationship between brush types and weight sets, so that the same brush type corresponds to a specific weight set under the same configuration. to .

[0037] In one embodiment, the drawing segment analysis unit compares the reconstructable error value calculated for each drawing segment with a reconstructable error threshold: when the reconstructable error value does not exceed the reconstructable error threshold, the drawing segment is determined to be a low-error segment; when the reconstructable error value exceeds the reconstructable error threshold, it is determined to be a high-error segment.

[0038] Optionally, the reconstructable error threshold is adjusted based on the number of data packets in the real-time transmission queue that have not been acknowledged by the server and the average round-trip time of the most recent transmissions. When the number of data packets that have not been acknowledged by the server exceeds the queue length threshold, or the average round-trip time exceeds the latency threshold, the sending client adjusts the reconstructable error threshold to a threshold greater than the current value, so that more fragments are classified as low-error fragments, reducing the amount of supplementary detail data that needs to be generated and sent, thereby alleviating queue backlog under network congestion.

[0039] When the number of unacknowledged data packets and the average round-trip time do not exceed the corresponding thresholds, the sending client restores the reconstructible error threshold to its default value or adjusts it to a threshold lower than the current value, enabling more fragments to obtain support from supplementary detailed data. The default value of the reconstructible error threshold, the queue length threshold, and the latency threshold can all be configured to system preset values, and the adjustment result of the reconstructible error threshold is limited to a preset range between 0 and 1. Adaptive adjustment allows the client to actively shrink the generation range of supplementary data when the network is poor, concentrating limited bandwidth to ensure the transmission of fast-displaying data, and then expanding the coverage of supplementary data after the network recovers, thereby balancing real-time performance and reconstruction accuracy under different network conditions.

[0040] In some embodiments, for step S3, based on the determination results of low-error segments and high-error segments, the sending client generates transmission data of the corresponding type for each drawn segment.

[0041] like Figure 2 As shown, Figure 2 This is a schematic diagram of the generation and transmission process of fast display data and supplementary detail data according to an embodiment of the present disclosure, wherein in S201: for low error segments, the sending client only generates fast display data.

[0042] Its content includes: the starting coordinates and pressure value of the drawing segment, the ending coordinates and pressure value, the intermediate key touch points and their pressure values, the brush identifier, the brush version identifier, the deterministic rendering seed, the layer identifier, the handwriting identifier, the segment number, the tile range, and the quick data identifier. Among them, the handwriting identifier is used to identify a continuous drawing operation, the segment number identifies the drawing segment within the handwriting, the tile range identifies the canvas tile area affected by the segment, and the quick data identifier is used by the receiving client to distinguish data types; the intermediate key touch points are a small number of representative points selected by the drawing segment analysis unit from the segment touch points that can reflect changes in trajectory direction, and the selection criteria are touch points with large changes in direction angle or at trajectory turning points.

[0043] In S202: For high-error segments, the sending client simultaneously generates fast display data and supplementary detail data. The fast display data is identical to that of low-error segments; the supplementary detail data includes: the coordinate sequence of intermediate sampling points within the drawn segment, the pressure value sequence, speed change information between touch points, local parameters, handwriting identifiers, segment number, layer identifier, tile range, and detail data identifiers. The intermediate sampling point coordinate sequence is denser than the key touch points in the fast display data, containing more intermediate trajectory position information; the speed change information is determined based on the coordinate positions of adjacent touch points and the acquisition timestamp or sampling time interval, recording the speed differences between adjacent touch points for the receiving end to adjust the stroke dynamics during re-rendering; local parameters are segment-level parameters needed to correct interpolation results, such as point-by-point changes in brush parameters within the segment; the detail data identifiers allow the receiving client to identify the data as supplementary detail data, and the supplementary detail data is associated with the corresponding fast display data through the handwriting identifier and segment number.

[0044] In S203: The sending client adds the fast display data to the real-time sending queue and adds the supplementary detailed data to the detailed sending queue.

[0045] When sending data, the real-time communication module first sends the fast display data in the real-time sending queue. When there is no data to be sent in the real-time sending queue, the sending client retrieves supplementary detail data from the detail sending queue and sends it. The detail sending queue is arranged in descending order according to the reconstructable error value corresponding to the supplementary detail data. The arrangement result is used as the sending order of the supplementary detail data, that is, the higher the reconstructable error value of the segment, the higher the priority of its supplementary detail data.

[0046] The reason for adopting this queued and priority-based transmission method is that the receiving end of low-error segments can obtain a rendering effect that is close to that of the sending end simply by interpolation, without the need to generate supplementary detail data for them, or to add their supplementary detail data to the detail transmission queue; delaying their supplementary data has little impact on the image; while high-error segments have obvious trajectory curvature, pressure abrupt changes, or brush texture changes, and interpolation alone can easily produce perceptible deviations in terms of edges, transparency overlays, or texture representation. Prioritizing the transmission of their supplementary data can concentrate the limited real-time bandwidth on the parts that have the greatest impact on image reproduction.

[0047] In addition, the division of drawing segments and data generation are completed segment by segment during the handwriting drawing process. The sending client does not need to wait for the entire stroke to be drawn before it can start sending the quickly displayed data of the formed segments to the server. Thus, the remote device can gradually see each segment of the handwriting while the sending user is still drawing.

[0048] In some embodiments, for step S4, such as Figure 3 As shown, Figure 3This is a schematic diagram of the server forwarding and sequence control process according to an embodiment of the present disclosure, wherein in S301: the server assigns an operation sequence number to the drawing operation.

[0049] After receiving the fast display data and supplementary detail data sent by each sending client, the collaborative synchronization control module assigns an operation sequence number to each drawing operation. This sequence number is globally incremented across all user operations to determine the final rendering order of different user drawing operations. Each drawing segment within the same stroke retains the segment number generated by the sending end. The server retains the association between the operation sequence number and the handwriting identifier, segment number, layer identifier, and tile range for subsequent forwarding scheduling and receiver positioning.

[0050] In S302: Forwarding fast display data and supplementary detail data. Specifically, for forwarding fast display data, if two drawing operations are located in different layer partitions or affect different tile ranges, the server forwards their fast display data in parallel without waiting for each other. If two drawing operations affect the same tile range within the same layer, the server determines the forwarding order according to the operation sequence number, ensuring consistent rendering coverage results for the area on each receiving end.

[0051] For supplementary detail data, the server generates a forwarding order based on multiple status information and forwards the data sequentially. Specifically, the server receives the currently viewed canvas area and zoom level from the receiving client, matches the tile range of the supplementary detail data to be forwarded with this canvas area to obtain a visual matching result, i.e., determining whether the tile corresponding to the data to be forwarded is within the range currently being viewed by the receiving client. Based on the above, the server generates a forwarding order according to the reconstructable error value, visual matching result, zoom level, length of the sending queue to the receiving client, most recent round-trip time, number of packet loss retransmissions, and confirmed operation sequence number, and forwards the supplementary detail data to the receiving client according to this forwarding order.

[0052] In one embodiment, when generating the forwarding order, the server evaluates the comprehensive priority of each piece of supplementary detail data to be forwarded: the higher the reconstructable error value, the greater the deviation of the segment when reconstructed by interpolation at the receiving end, and the higher the priority; if it is within the visible range of the receiving end and the scaling ratio is larger, the rendering details of that area are more easily perceived by the user, and the priority is correspondingly increased; the longer the sending queue to that receiving end, the greater the round-trip time, and the more frequent the packet loss retransmission, the more congested the channel is, and the priority differentiation becomes more important, with low-priority supplementary detail data being further delayed to ensure that high-priority data passes first. The above comprehensive priority is determined according to the server's preset sorting rules or preset priority scoring rules, so that the same reconstructable error value, visual matching result, scaling ratio, sending queue length, most recent round-trip time, number of packet loss retransmissions, and confirmed operation sequence number correspond to the determined forwarding order.

[0053] In some embodiments, for step S5, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the process of re-rendering the corresponding drawing segment by the receiving end according to the embodiments of this disclosure. In S401: After receiving the fast display data, the receiving end client identifies it as fast display data according to the fast data identifier, determines the temporary rendering order according to the operation sequence number and segment number, and saves the layer tile state before rendering.

[0054] Before performing temporary rendering, the receiving client first determines the tile to be rendered based on the layer identifier and tile range in the quick display data. If the local re-rendering cache unit does not save the current rendering state of the tile to be rendered corresponding to the handwriting identifier, fragment number, layer identifier, and tile range of the current quick display data, then a copy of the current rendering state of the tile is made and stored in the local re-rendering cache unit as the layer tile state. The layer tile state is then associated with and saved with the handwriting identifier, fragment number, layer identifier, and tile range. If the cache capacity of the local re-rendering cache unit reaches its limit, the earliest saved layer tile state is released to make room for the new state.

[0055] The cache capacity limit can be configured according to the available memory of the terminal device; the purpose of saving the layer tile state is that once subsequent supplementary detailed data arrives, the receiving end can find the layer tile state of the corresponding segment before temporary rendering based on the handwriting identifier, segment number, layer identifier and tile range, restore the tile to the state before temporary rendering, and then re-render based on more complete data.

[0056] After saving the layer tile state, the receiving client generates intermediate trajectory points using an interpolation algorithm based on the start-point coordinates, end-point coordinates, intermediate key touch points, and their pressure values ​​from the quickly displayed data. The interpolation algorithm can employ cubic spline interpolation. Specifically, using each key touch point as a model point, a piecewise cubic polynomial curve is constructed to ensure the trajectory satisfies the position and first derivative continuity conditions at each key point, thus generating a smooth transition between key points. After generating the intermediate trajectory points, the receiving client, combining the brush identifier, brush version identifier, deterministic rendering seed, and layer identifier, calls the rendering display module to perform temporary rendering on the corresponding layer. This allows the remote user to immediately see the location and approximate shape of the handwriting being drawn by another user.

[0057] After receiving the supplementary detail data, the receiving client identifies its data type based on the detail data identifier and extracts the handwriting identifier, fragment number, layer identifier, and tile range from it. Based on this, it searches for the corresponding layer tile state in the local re-rendering cache unit.

[0058] In S402: After receiving the supplementary detail data, the receiving client identifies its data type based on the detail data identifier, and extracts the handwriting identifier, fragment number, layer identifier and tile range from it, and then looks up the corresponding layer tile state in the local re-rendering cache unit.

[0059] In S403: Restore the affected tiles to the state of the layer tiles, and then re-render the corresponding drawing fragment according to the supplementary detail data.

[0060] After a successful search, the receiving client restores the affected tile to the tile state of that layer. Then, based on the handwriting identifier and fragment number, it associates the supplementary detail data with the corresponding quick display data. It uses the start point coordinates and pressure value, end point coordinates and pressure value, brush identifier, brush version identifier, and deterministic rendering seed from the corresponding quick display data, and combines them with the intermediate sampling point coordinate sequence, pressure value sequence, speed change information, and local parameters from the supplementary detail data as inputs for re-rendering the corresponding drawing fragment. The re-rendering is then completed by combining the brush parameters corresponding to the brush identifier and the deterministic rendering seed.

[0061] Because the supplemental detail data contains denser sampling points and more complete parameters, the re-rendered handwriting is closer to the original result from the sending end in terms of trajectory accuracy, pressure transition, and texture representation. Furthermore, when there are subsequent drawing fragments that have already been temporarily rendered and affect the same affected tile after the drawing fragment, the receiving client re-renders these subsequent drawing fragments sequentially according to the operation sequence number and fragment number. Specifically, for subsequent drawing fragments that have received supplemental detail data and completed corrections, they are re-rendered according to their supplemental detail data; for subsequent drawing fragments that have not yet received supplemental detail data, they are re-rendered according to their fast display data to ensure the correct overlay relationship of fragments within the same tile area. The above re-rendering process only involves the affected tile area.

[0062] In one embodiment, if the receiving client does not find the corresponding layer tile state in the local re-rendering cache unit, for example, because the tile state has been released due to cache capacity limitations, the receiving client marks the corresponding drawing fragment as pending refresh. When the affected tile is triggered by user operation to update the screen, or when the system periodic consistency check is triggered, the receiving client re-renders the affected tile according to the complete structured record containing supplementary detailed data. The complete structured record is either locally saved by the receiving client or obtained by the server after continuing to send a list of missing fragment numbers. When re-rendering the affected tile, the receiving client renders the relevant complete structured records affecting the affected tile in the order of operation sequence number and fragment number.

[0063] After completing the above processing, the receiving client sends feedback to the server on the highest operation sequence number received, the range of segment numbers that have been corrected, and the list of missing segment numbers. Based on this feedback, the server determines whether to continue sending the corresponding supplementary detail data to the receiving client. For example, if the receiving end has obtained a complete structured record of a segment through subsequent historical compressed packets, the server will not resend the supplementary detail data of that segment, thereby reducing the bandwidth consumption of redundant transmission.

[0064] In one embodiment, regarding the global brush dataset and deterministic rendering, the server maintains a global brush dataset that stores the set of brush parameters used by all users during the collaboration process, along with their corresponding brush version identifiers. The brush parameter set includes parameters such as brush size, opacity, texture density, dot pitch, dithering, wet edge parameters, and blending intensity. The sending client records the brush identifier, brush version identifier, and deterministic rendering seed in both the quick display data and supplementary detail data.

[0065] After receiving the drawing data, the receiving client retrieves the corresponding brush parameter set from the global brush data set based on the brush identifier and brush version identifier. Using a deterministic rendering seed as input for random number generation, it performs deterministic rendering on brushes containing texture, scatter, or grain effects. The deterministic rendering seed is a random value generated by the sending client during drawing. The receiving end can reproduce the same random effect as the sending end by using the same seed value, thus avoiding texture and scatter differences caused by different devices generating different random numbers. When the sending client modifies the brush parameters, it generates a new brush version identifier while retaining the original brush version identifier in the global brush data set, ensuring that strokes drawn using the old version can still be correctly rendered with the original parameters on all receiving ends.

[0066] Regarding conflict control for concurrent operations, in one optional implementation, the system assigns an independent layer partition to each user participating in the collaboration. When each user draws at the same canvas position, they actually operate on the layers within their respective partitions, and the drawing content of different users does not overlap with each other. Users can choose to show or hide the layer partitions of other users to reduce visual interference during the creation process. The creator of the collaborative canvas has administrator privileges and can temporarily take over the layer partitions of other users or remove specific users from the collaborative canvas.

[0067] Regarding the storage and incremental loading of historical data, the server packages and compresses the accumulated structured drawing data according to the preset number of operations. When packaging, the quick display data and supplementary detail data of the same drawing fragment are merged into a complete structured record. This complete structured record includes at least the operation sequence number and fragment number used to determine the drawing order, as well as the layer identifier and tile range used to determine the affected area. When a new device joins, it reports the latest local operation sequence number and the list of existing brush versions. Based on this, the server returns the missing incremental data package. The new device renders the incremental data according to the operation sequence number based on the existing local snapshot, thus obtaining the current canvas state.

[0068] In this embodiment, the sending client assesses the reconstructability of a drawn fragment based on its trajectory, pressure, brush, and layer features. Supplementary detail data is generated only for high-error fragments and prioritized for transmission based on their reconstructability error values. The server, considering visual matching results, network status, and reconstructability error values, schedules the forwarding order of the supplementary detail data. The receiving end displays the handwriting immediately upon receiving the fast display data, and only performs local re-rendering corrections on affected tiles when subsequent supplementary detail data arrives. This reduces the latency of handwriting display on remote devices, improves remote restoration accuracy in complex brush scenarios, ensures reasonable allocation of bandwidth and terminal computing resources, and guarantees image consistency in scenarios with multiple users and high concurrency drawing.

[0069] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0070] Please see Figure 5 , Figure 5 This application provides a multi-user drawing system capable of real-time synchronization and low-latency transmission. The system embodiment is similar to... Figure 1 Corresponding to the illustrated method embodiments, this system can be specifically applied to various electronic devices. The system specifically includes: The system comprises a transmitter, a server, and a receiver; wherein the transmitter includes a touch point acquisition and segmentation module, an error assessment module, and a data generation and transmission module. The touch point acquisition and segmentation module is used to acquire touch points when the user draws continuously, and divide the acquired touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. The error evaluation module is used to calculate the reconstructable error value for each drawn segment based on the trajectory, pressure, brush and layer features of the drawn segment, and compare the reconstructable error value with the reconstructable error threshold to obtain a low error segment or a high error segment. The data generation and sending module is used to generate fast display data for low error segments, generate fast display data and supplementary detail data for high error segments, add the fast display data to the real-time sending queue, and add the supplementary detail data to the detail sending queue before sending it to the server. The server includes a synchronization forwarding module, which is used to assign operation sequence numbers to drawing operations and forward the fast display data and the supplementary detail data. The receiving end includes a temporary rendering module, a tile state saving module, and a detail re-rendering module; The temporary rendering module is used to perform temporary rendering based on the operation sequence number and fragment number, according to the fast display data. The tile state saving module is used to save the layer tile state before the temporary rendering module performs temporary rendering. The detailed re-rendering module is used to, after receiving supplementary detailed data, find the layer tile state based on the handwriting identifier, fragment number, layer identifier, and tile range in the supplementary detailed data, restore the affected tiles to the layer tile state, and re-render the corresponding drawing fragment according to the supplementary detailed data.

[0071] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0072] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0073] Figure 6 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input section 806, an output section 807, a communication section 809, and an input / output (I / O) interface 805 are also connected to the bus 804.

[0074] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a central processing unit (CPU) 801, it performs the functions defined in the methods of this application.

[0075] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0076] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0077] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-user painting method capable of real-time synchronization and low-latency transmission, characterized in that, include: The sending end collects touch points while the user draws continuously, and divides the collected touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. For each drawn segment, a reconstructable error value is calculated based on the segment's trajectory, pressure, brush, and layer features, and compared with a reconstructable error threshold to determine whether the segment is low-error or high-error. The reconstructable error value is obtained by combining the segment's average curvature normalized value, the average distance between adjacent touch points normalized value, the pressure change rate normalized value, the brush effective radius normalized value, the transparency change amplitude normalized value, the brush texture sensitivity coefficient, and the layer blending influence coefficient according to their respective weights. Fast display data is generated for low-error segments, and fast display data and supplementary detail data are generated for high-error segments. The sending end adds the fast display data to the real-time sending queue and the supplementary detail data to the detail sending queue before sending it to the server. The server assigns operation sequence numbers to drawing operations and forwards them for quick display of data and supplementary detailed data; The receiving end performs temporary rendering based on the operation sequence number and fragment number, according to the quick display data, and saves the layer tile state before rendering. After receiving the supplementary detail data, it searches for the layer tile state based on the handwriting identifier, fragment number, layer identifier, and tile range, restores the affected tiles to the layer tile state, and then re-renders the corresponding drawing fragment according to the supplementary detail data.

2. The multi-person painting method according to claim 1, which enables real-time synchronization and low-latency transmission, is characterized in that... The step of dividing the collected touch points into drawing segments includes: when the change in direction angle between adjacent touch points exceeds the direction segmentation threshold, or the pressure difference exceeds the pressure segmentation threshold, or the brush parameters are switched, the current touch point is determined as the end point of the previous drawing segment and the start point of the next drawing segment; when the number of touch points contained in a drawing segment reaches a preset upper limit, the touch point collected when the preset upper limit is reached is determined as the end point of the drawing segment and the start point of the next drawing segment.

3. The multi-person painting method according to claim 2, which enables real-time synchronization and low-latency transmission, is characterized in that... The reconstructable error threshold is adjusted based on the number of data packets in the real-time transmission queue that have not been acknowledged by the server and the average round-trip time of the most recent transmissions. When the number of data packets that have not been acknowledged by the server exceeds the queue length threshold, or the average round-trip time exceeds the delay threshold, the reconstructable error threshold is adjusted to a threshold greater than the current value. When neither of these thresholds is exceeded, the reconstructable error threshold is restored to the default value or adjusted to a threshold less than the current value.

4. A multi-person painting method according to claim 3, characterized in that, The quick display data includes the starting point coordinates and pressure value, the ending point coordinates and pressure value, intermediate key touch points and their pressure values, brush identifier, brush version identifier, deterministic rendering seed, layer identifier, handwriting identifier, fragment number, tile range, and quick data identifier. The receiving end identifies the quick display data based on the quick data identifier, determines the temporary rendering order according to the operation sequence number and fragment number, interpolates intermediate trajectory points based on the starting point coordinates, ending point coordinates, intermediate key touch points and their pressure values, and performs temporary rendering in conjunction with the brush identifier, brush version identifier, deterministic rendering seed and layer identifier.

5. A multi-person painting method according to claim 4, characterized in that, The supplementary detail data includes the coordinate sequence of intermediate sampling points within the drawn segment, the pressure value sequence, the speed change information between touch points, local parameters, handwriting identifiers, segment numbers, layer identifiers, tile ranges, and detail data identifiers. The receiving end identifies the supplementary detail data based on the detail data identifiers and uses the coordinate sequence of intermediate sampling points, the pressure value sequence, the speed change information, and the local parameters as inputs for re-rendering the corresponding drawn segment.

6. A multi-person painting method according to claim 5, characterized in that, The sending end first sends the fast display data in the real-time sending queue; when there is no data to be sent in the real-time sending queue, the sending end retrieves supplementary detail data from the detail sending queue; the detail sending queue is arranged in descending order according to the reconstructable error value corresponding to the supplementary detail data and sent in this order.

7. A multi-person painting method according to claim 6, characterized in that, After receiving the quick display data and supplementary detail data, the server retains the association between the operation sequence number and the handwriting identifier, fragment number, layer identifier, and tile range. For quick display data, if two drawing operations are located in different layer partitions or affect different tile ranges, the server forwards them in parallel. If two drawing operations affect the same tile range within the same layer, the server determines the forwarding order according to the operation sequence number.

8. A multi-person painting method according to claim 7, characterized in that, The server receives the currently viewed canvas area and zoom level from the receiver, matches the tile range of the supplementary detail data to be forwarded with the currently viewed canvas area, and obtains a visual matching result. The server generates a forwarding order based on the reconstructable error value, visual matching result, zoom level, length of the sending queue facing the receiver, most recent round-trip time, number of packet loss retransmissions, and confirmed operation sequence number, and forwards the supplementary detail data to the receiver according to the forwarding order.

9. A multi-person painting method according to claim 8, characterized in that, After the receiving end successfully finds the layer tile status, it restores the affected tile to the layer tile status and re-renders the corresponding drawing fragment according to the supplementary detail data; when there is a subsequent drawing fragment that has been temporarily rendered and affects the same affected tile after the corresponding drawing fragment, the receiving end re-renders the subsequent drawing fragment according to the operation sequence number and fragment number.

10. A multi-user painting system capable of real-time synchronization and low-latency transmission, characterized in that, include: Sender, server, and receiver; The transmitting end includes a touch point acquisition and segmentation module, an error assessment module, and a data generation and transmission module; The touch point acquisition and segmentation module is used to acquire touch points when the user draws continuously, and divide the acquired touch points into drawing segments according to the change in the direction angle of adjacent touch points, the pressure difference, or the brush parameter switching results. The error evaluation module is used to calculate a reconstructable error value for each drawn segment based on its trajectory, pressure, brush, and layer features, and compare the reconstructable error value with a reconstructable error threshold to obtain a low-error segment or a high-error segment. The reconstructable error value is obtained by combining the segment's average curvature normalized value, the average distance between adjacent touch points normalized value, the pressure change rate normalized value, the brush's effective radius normalized value, the transparency change amplitude normalized value, the brush texture sensitivity coefficient, and the layer blending influence coefficient according to their corresponding weights. The data generation and sending module is used to generate fast display data for low error segments, generate fast display data and supplementary detail data for high error segments, add the fast display data to the real-time sending queue, and add the supplementary detail data to the detail sending queue before sending it to the server. The server includes a synchronization forwarding module, which is used to assign operation sequence numbers to drawing operations and forward the fast display data and the supplementary detail data. The receiving end includes a temporary rendering module, a tile state saving module, and a detail re-rendering module; The temporary rendering module is used to perform temporary rendering based on the operation sequence number and fragment number, according to the fast display data. The tile state saving module is used to save the layer tile state before the temporary rendering module performs temporary rendering. The detailed re-rendering module is used to, after receiving supplementary detailed data, find the layer tile state based on the handwriting identifier, fragment number, layer identifier, and tile range in the supplementary detailed data, restore the affected tiles to the layer tile state, and re-render the corresponding drawing fragment according to the supplementary detailed data.

Images