Screen synchronization method, electronic equipment, readable storage medium and program product

Abstract

The embodiment of the invention provides a screen synchronization method, electronic equipment, a readable storage medium and a program product. The method relates to the technical field of screen synchronization, and comprises the following steps: cutting a current frame of a mother screen server to obtain a plurality of current sub-images; performing Hash coding on the plurality of current sub-graphs to obtain a plurality of coded current sub-graphs; comparing the difference between each coded current sub-graph and the corresponding coded historical sub-graph, and determining the coded current sub-graph with the difference as a target sub-graph; the plurality of historical sub-graphs are obtained by cutting historical frames of the mother screen server, and the plurality of historical sub-graphs are in one-to-one correspondence with the plurality of current sub-graphs according to a preset cutting sequence; and transmitting the target sub-image to a sub-screen server synchronized with the mother screen server, wherein the sub-screen server is used for updating the image frame based on the target sub-image. According to the invention, the technical problem that the screen synchronization mode is highly limited by the network communication capability in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of screen synchronization technology, and in particular to a screen synchronization method, electronic device, readable storage medium, and program product. Background Technology

[0002] Screen synchronization is an application feature that can greatly improve communication efficiency in various scenarios. Through real-time screen synchronization, the content to be expressed can be quickly presented to the target audience. To ensure high-definition real-time image transmission, existing screen synchronization generally has high network bandwidth requirements. When the network is not smooth, screen stuttering is very likely to occur. In addition, there is a low bandwidth threshold limitation. When the network conditions are below a certain level, synchronization will be directly interrupted.

[0003] The relevant technology encodes and transmits the entire screen image. Even if only a small part of the area changes, the entire frame data still needs to be sent. This is severely limited by network bandwidth and is difficult to adapt to application environments with poor network conditions. It is also prone to repeated interruptions in screen synchronization due to insufficient network transmission capacity, which affects the user experience. Summary of the Invention

[0004] This application provides a screen synchronization method, electronic device, readable storage medium, and program product to alleviate or solve the technical problem that screen synchronization methods in related technologies are severely limited by network communication capabilities.

[0005] In a first aspect, embodiments of this application provide a screen synchronization method, including: The current frame of the parent screen server is cropped to obtain multiple current sub-images; Hash-encode multiple current subgraphs to obtain multiple encoded current subgraphs; Compare the differences between each current subgraph after encoding and the corresponding historical subgraph after encoding, and determine the current subgraph after encoding with differences as the target subgraph; multiple historical subgraphs are obtained by cropping the historical frames of the master screen server, and multiple historical subgraphs correspond one-to-one with multiple current subgraphs according to a predetermined cropping order. The target sub-image is transmitted to the child screen server, which is synchronized with the parent screen server. The child screen server is used to update image frames based on the target sub-image.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0009] Based on the screen synchronization method of the first aspect described above, this application has at least the following beneficial effects or advantages: Since the entire screen frame doesn't need to be processed and transmitted, both the parent and child screen servers consume less computing and storage resources when processing data. By transmitting only the differing target sub-images instead of the entire screen frame, the amount of data that needs to be transmitted is significantly reduced. This reduces bandwidth usage and improves data transmission efficiency during network transmission, making it more suitable for applications with poor network environments. By reducing the amount of data transmitted and optimizing the update process, screen synchronization latency can be reduced, resulting in smoother remote display.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 A flowchart of a screen synchronization method according to an embodiment of this application is shown; Figure 2 A sub-diagram schematic diagram of the screen synchronization method according to an embodiment of this application is shown; Figure 3 A schematic flowchart of a screen synchronization method according to an embodiment of this application is shown; Figure 4 A schematic block diagram of a screen synchronization device according to an embodiment of this application is shown; Figure 5 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0013] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0014] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0015] The following terms will be used in the following text: Screen synchronization allows one party's screen content to be projected or shared with another party in real time, enabling information sharing and communication in scenarios such as remote collaboration, teaching, presentations, or meetings, thereby improving communication efficiency and collaboration quality.

[0016] MD5 (Message-Digest Algorithm 5) is a widely used cryptographic hash function that can compress messages of arbitrary length into a 128-bit (16-byte) hash value, typically represented as a 32-bit hexadecimal number. The hash values ​​generated by the MD5 algorithm are highly unique; even slight changes in the input data will result in significantly different output hash values, making it suitable for verifying data integrity and consistency.

[0017] UDP (User Datagram Protocol) is a connectionless network protocol that sends data directly without establishing a connection before communication, making it suitable for scenarios with high real-time requirements. UDP does not guarantee the order, integrity, or arrival of data packets, and will not retransmit lost packets. UDP can send data to multiple targets simultaneously (such as live video streaming and LAN discovery services).

[0018] Screen synchronization technology is highly dependent on network quality. A smooth, real-time screen transmission relies on a high-bandwidth, low-latency, and low-packet-loss network environment, while poor network conditions significantly impact user experience. Current technologies still struggle to avoid image quality degradation or interruptions in weak network environments. Insufficient bandwidth can cause stuttering, reduced resolution, or interruptions; high latency can lead to desynchronization between operation and display; and packet loss can cause screen tearing and other issues. Current technologies have not yet reduced their dependence on the network to achieve more stable adaptability in weak network conditions.

[0019] It should be noted that the application scenarios or examples provided in this application embodiment are for ease of understanding, and this application embodiment does not specifically limit the application of the technical solution. Furthermore, the user information (including but not limited to user device information, user personal information, user interaction information on the main screen server and sub-screen server, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, display data of the main screen server and sub-screen server, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 A flowchart of a screen synchronization method according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101 to S104.

[0022] Step S101: Crop the current frame of the parent screen server to obtain multiple current sub-images; Step S102: Hash-encode multiple current subgraphs to obtain multiple encoded current subgraphs; Step S103: Compare the differences between each current sub-image after encoding and the corresponding historical sub-image after encoding, and determine the current sub-image after encoding with differences as the target sub-image; the multiple historical sub-images are obtained by cropping the historical frames of the parent screen server, and the multiple historical sub-images correspond one-to-one with the multiple current sub-images according to the predetermined cropping order; Step S104: Transmit the target sub-image to the sub-screen server that is synchronized with the parent screen server. The sub-screen server is used to update image frames based on the target sub-image.

[0023] For example, the execution entity of this application embodiment may be a processing unit integrated in the master screen server. The processing unit may run a dedicated software module through a CPU (Central Processing Unit) / GPU (Graphics Processing Unit) to process image cropping, hash calculation and difference detection in real time.

[0024] For example, the aforementioned master screen server and child screen server can be applied to online meeting tools, cloud collaboration platforms, remote teaching, remote surgical assistance, live interactive sessions, and so on. They can synchronize the speaker's screen-shared content, which can be PPT presentations, documents, etc., transmitting only the changing areas, such as cursor movement on the display screen.

[0025] In the embodiments provided in this application, screen synchronization is achieved by updating the image on the sub-screen server by comparing the differences between the current screen frame and historical screen frames. The current screen frame on the parent screen server is cropped into multiple smaller sub-images, thus obtaining the current sub-image. These current sub-images are then hash-encoded to generate multiple encoded current sub-images. Hash encoding is a method of converting data into a fixed-length string for quickly comparing data consistency. The encoded current sub-images are compared with their corresponding historical sub-images. The historical sub-images are obtained by similarly cropping historical frames on the parent screen server and correspond one-to-one with the current sub-images according to a predetermined cropping order. By comparison, it is determined which current sub-images differ from the historical sub-images, and these differing sub-images are marked as target sub-images.

[0026] Figure 2 A sub-diagram schematic diagram of the screen synchronization method according to an embodiment of this application is shown, such as... Figure 2 As shown, the identified target sub-images are transmitted to the sub-screen servers that are synchronized with the parent screen server. After receiving the target sub-images, the sub-screen servers update the image frames based on these sub-images, thereby achieving screen content synchronization.

[0027] By transmitting only the subgraphs that differ (the target subgraphs) instead of the entire screen frame, the amount of data that needs to be transmitted can be significantly reduced, thereby reducing bandwidth requirements and improving transmission efficiency, which helps to achieve an efficient and low-latency screen synchronization method.

[0028] It's important to note that hash encoding compresses image data into fixed-length hash values. Comparison only requires checking if these hash values ​​match, resulting in extremely low computational complexity. In contrast, current pixel-level image comparison techniques require comparing every pixel value of the two sub-images one by one. The computational load is directly proportional to the number of pixels in the image, and the computation time increases significantly with higher image resolution. For example, for a 1080p sub-image, pixel-level comparison requires processing approximately 2 million pixels, while hash comparison only requires comparing fixed-length hash values ​​(such as 128-bit or 256-bit), improving computational efficiency by several orders of magnitude. Hash encoding comparison simplifies complex image data into easily comparable hash values, greatly improving efficiency and reducing resource consumption while maintaining comparison accuracy. It is particularly suitable for screen synchronization scenarios requiring frequent image difference detection.

[0029] For example, the hash encoding described above can be of various types, such as MD5 (Message Digest Algorithm 5), Perceptual Hash (pHash), and SHA-256 (Secure Hash Algorithm 256-bit). Preferably, the MD5 algorithm is used, as it can generate a 128-bit (32-character) hash value and is computationally fast. Perceptual Hash generates hashes based on the visual content of the image, is robust to scaling, rotation, and brightness changes, and is suitable for visual similarity comparison of screen image frames. It can ignore minor pixel variations (such as JPEG compression noise) and focus on semantic changes. The SHA-256 algorithm generates a 256-bit (64-character) hash value, offering stronger collision resistance and making it suitable for screen synchronization methods with high security sensitivity requirements. In terms of computational speed, the MD5 algorithm is superior to the Perceptual Hash algorithm, which in turn is superior to the SHA-256 algorithm. Regarding algorithm security and collision resistance, the SHA-256 algorithm is superior to the Perceptual Hash algorithm, which is superior to the MD5 algorithm. You can make specific settings according to the real-time requirements and image security requirements of screen synchronization. Select MD5 for transmission speed requirements, pHash for visual robustness requirements, and SHA-256 for high security requirements.

[0030] For example, data can be transmitted between the parent screen server and the child screen servers using UDP. UDP does not require connection establishment, unlike TCP's three-way handshake and acknowledgment mechanism; data reaches the receiving end directly, making it suitable for millisecond-level response requirements for screen synchronization. The same data can be sent to multiple child screen servers simultaneously, avoiding the point-to-point copying load of TCP in classroom live streaming scenarios.

[0031] In the embodiments provided in this application, for image cropping control, the number of horizontal and vertical cropping operations can be set, and the size of the generated sub-images can be used as a reference for image cropping size. For cropping frequency control, the time interval between cropping operations can be set, affecting the frequency of cropping operations and serving as a reference for the frequency of cropping actions; by serializing sub-images through image numbering, the minimum number of changed sub-images can be filtered, reducing the amount of sub-image data that needs to be updated and lowering the bandwidth usage for data transmission.

[0032] Since the embodiments provided in this application can adjust the dynamic cropping method according to communication resources, in order to solve the problem that the cropping methods of the preceding and following frames are different due to changes in communication resources, thus affecting the comparison results of the preceding and following frames, the method further includes: If the current cropping size of the current frame is different from the historical cropping size of the historical frame, the historical frame is re-cropped using the current cropping size to obtain multiple updated sub-images; Multiple update subgraphs are hash-encoded to obtain multiple encoded update subgraphs. Compare the differences between each encoded current subgraph and its corresponding encoded updated subgraph, and identify the encoded current subgraph with the differences as the target subgraph.

[0033] In the embodiments provided in this application, when changes in communication resources cause inconsistencies in the cropping size between the current frame and historical frames, the historical frames are re-cropped using the cropping size of the current frame, generating multiple updated sub-graphs with the same cropping method as the current frame. These updated sub-graphs are hash-encoded to obtain encoded updated sub-graphs. The hash values ​​of the encoded current sub-graph are compared with the encoded updated sub-graphs at the corresponding positions. If a mismatch exists, the current sub-graph is determined to be the target sub-graph. In this way, it is ensured that sub-graphs of consecutive frames are compared under the same cropping rules, avoiding misjudgments caused by different cropping methods.

[0034] According to the embodiments provided in this application, the method may further include the following: When the channel resources between the mother screen server and the child screen server change, the historical cropping size is updated based on the channel resources to obtain the current cropping size.

[0035] In the embodiments provided in this application, when changes in communication resources cause inconsistencies in the cropping size between the current frame and historical frames, the cropping strategy can be dynamically adjusted during network fluctuations. The historical cropping size is updated according to changes in channel resources to determine the current cropping size that adapts to the current transmission capabilities. Through this processing, the master screen server can dynamically adjust its image cropping method according to real-time changes in channel resources, achieving seamless network adaptation and preventing screen synchronization interruptions due to network fluctuations.

[0036] For example, to support dynamic cropping and resizing adjustments and ensure consistent frame alignment, the master screen server needs to store the original historical frames, not just the encoded hash values ​​or sub-image data, while also optimizing storage overhead. When the cropping size changes (e.g., from 64×64 to 128×128), the image needs to be re-blocked based on the original image, but the encoded sub-images (e.g., MD5 hashes) cannot reconstruct the pixel data. Directly storing the original frames avoids image quality loss due to multiple compression / decompressions, ensuring accurate alignment of the re-cropped sub-images with the current frame. To reduce storage pressure, historical frames can be compressed, using either lossless or slightly lossy compression methods. The compression method is not limited and should be set according to specific needs.

[0037] The embodiments provided in this application control the overall amount of image data generated by adding settings for image cropping size and cropping operation frequency before the image cropping stage. Simultaneously, during the processing of the cropped sub-images, the smallest difference image file is generated through minimum ratio adjustment and output for screen synchronization.

[0038] According to the embodiments provided in this application, the method may further include the following steps: The cropping frequency is determined based on the channel resources between the master screen server and the child screen server, and the display type that represents the image change rate of the master screen server. Perform cropping processing on the video frames in the master screen server according to the cropping frequency.

[0039] In the embodiments provided in this application, the channel resources between the master screen server and the child screen server, the image change rate of the master screen content, and different display types such as static documents, dynamic videos, or highly interactive applications are considered. By comprehensively analyzing the channel resources and display types, the optimal cropping processing frequency is calculated, and the video frames of the master screen are cropped in real time according to this frequency.

[0040] The aforementioned dynamic adjustment of cropping frequency enhances the adaptability and efficiency of screen synchronization. It balances image quality and transmission performance based on actual network conditions and content changes: increasing the cropping frequency to ensure real-time synchronization when network conditions are good and content changes frequently; and decreasing the frequency to reduce bandwidth consumption when network resources are scarce or content is relatively static. This adaptive mechanism not only optimizes network resource utilization but also ensures a smooth screen synchronization experience under various environmental conditions, effectively solving the problem of instability in traditional fixed-frequency cropping methods in changing network environments.

[0041] For example, the aforementioned channel resources may include various factors, such as bandwidth, network latency, packet loss rate, and uplink / downlink asymmetry. The bandwidth refers to the maximum amount of data that the channel can transmit per unit time, such as 100 Mbps. Bandwidth directly affects the setting of the clipping frequency. High bandwidth allows for high-frequency clipping (e.g., 60 fps) and large sub-images (256 × 256 pixels), ensuring smooth screen synchronization. Low bandwidth requires lowering the frequency (e.g., 5 fps) and reducing the sub-image size (32 × 32 pixels) to avoid data accumulation leading to stuttering. For example, a 4G network (10 Mbps) may only support 15 fps synchronization, while 5G (1 Gbps) can easily achieve 60 fps.

[0042] Network latency is the round-trip time for data to travel from send to receive. High latency forces a reduction in clipping frequency; otherwise, user actions and screen responses will become disconnected. Low latency, on the other hand, supports real-time, high-frequency clipping. For example, remote surgery requires latency of <50ms and needs to maintain a high clipping frequency to ensure operational synchronization.

[0043] Packet loss rate indicates the proportion of data packets lost during transmission. A high packet loss rate (>5%) requires reducing the pruning frequency and compensating for losses with redundant data. A low packet loss rate (<1%) allows for high-frequency pruning, relying directly on differential transmission; for example, in Wi-Fi packet loss, dynamic frequency reduction can prevent screen artifacts.

[0044] On many networks, uplink bandwidth is much lower than downlink bandwidth. Uplink bandwidth affects the transmission from the parent screen to the child screen, limiting the amount of data the parent screen can send to target sub-images that differ from the parent screen's. Downlink bandwidth affects the child screen's receiving capability. When uplink bandwidth is insufficient, the parent screen needs to reduce the frequency and compress the sub-image size. When downlink bandwidth is insufficient, the cropping frequency needs to be further reduced to avoid data accumulation on the child screen server.

[0045] According to the embodiments provided in this application, comparing the differences between each encoded current subgraph and the corresponding encoded historical subgraph to determine the encoded current subgraph with differences as the target subgraph may include the following steps: The first pointer points to any first current subgraph among the multiple current subgraphs after encoding, and the second pointer points to the first historical subgraph among the multiple historical subgraphs after encoding, with the first current subgraph corresponding to the first historical subgraph; If the first hash value of the encoded first current subgraph does not match the second hash value of the encoded first historical subgraph, the first current subgraph is determined to be the target subgraph. The method further includes the following steps: pointing the first pointer to the second current subgraph after the encoded first current subgraph; pointing the second pointer to the second historical subgraph after the encoded first historical subgraph; Until the first pointer is null and the second pointer is null, execute the action of sending the target subgraph to the subscreen server.

[0046] In the embodiments provided in this application, a first pointer and a second pointer are used to point to the encoded current subgraph and historical subgraph respectively (the two correspond one-to-one in a predetermined order). The first hash value of the current subgraph is compared with the second hash value of the corresponding first historical subgraph. If the hash values ​​do not match, the first current subgraph is determined as the target subgraph. Then the pointers move sequentially to continue comparing the next group of subgraphs until all subgraphs have been compared (i.e., both pointers point to null). After all the target subgraphs are determined, the sending process to the subscreen server is executed.

[0047] By employing pointer traversal and hash value comparison, efficient detection of sub-image differences and localization of target sub-images are achieved. Sequential pointer traversal ensures all sub-images are compared in an orderly manner, avoiding omissions of differing regions. Based on the fast matching characteristic of hash values, pixel-by-pixel comparison is unnecessary, significantly reducing computational load and improving difference detection efficiency. Transmitting only the target sub-image with mismatched hash values ​​further reduces data transmission volume, optimizes network resource utilization, and effectively reduces server computational load and bandwidth consumption while maintaining real-time screen synchronization.

[0048] According to the embodiments provided in this application, after step S104: transmitting the target sub-image to the sub-screen server synchronized with the parent screen server, the method may further include the following steps: When there are multiple sub-screen servers, obtain the interactive sub-graphs sent by each of the multiple sub-screen servers. The interactive sub-graph is the sub-graph in the image frame of the corresponding sub-screen server that contains the interactive operation part. When there are differences in the interaction subgraphs corresponding to multiple sub-screen servers, the interaction subgraphs corresponding to the multiple sub-screen servers are merged to obtain merged data. Based on the merged data, update the image frames following the current frame in the master screen server.

[0049] In the embodiments provided in this application, when multiple sub-screen servers exist, interactive sub-graphs are obtained from each sub-screen server. An interactive sub-graph refers to the sub-graph within an image frame of a sub-screen server where the user has performed interactive operations, such as clicking or dragging. Since different sub-screen servers are operated independently by the user, the interactive sub-graphs generated by their interaction processing may differ. The parent screen server detects whether there are differences in the interactive sub-graphs from different sub-screen servers. If differences are found, the parent screen server merges these interactive sub-graphs to generate merged data. The merged data may include a combination of all interactive sub-graphs, or selectively merge sub-graphs according to specific rules (such as timestamps, user priorities, etc.). Based on the merged data, the image frames after the current frame in the parent screen server are updated, integrating the interactive information from different sub-screen servers to reflect the interactive operations of all users.

[0050] Through the above processing, multiple users can collaborate on different sub-screen servers. By merging the interactive operations of different users, the effect of multiple people operating the same screen can be achieved. This embodiment of the application optimizes network bandwidth in bidirectional screen interaction by having the sub-screen server transmit only the interactive sub-graph (i.e., the local area operated by the user), rather than the complete image frame. When multiple sub-screen users operate, each sub-screen server extracts only the sub-graph area corresponding to the interactive operation and encodes and transmits it. The parent screen server receives and merges the differing interactive data to update subsequent frames. This significantly reduces the amount of data transmitted in traditional full-frame transmission, thereby reducing network bandwidth requirements while ensuring the real-time performance of bidirectional interaction. It is particularly suitable for bandwidth-constrained remote collaboration scenarios, such as multi-user online editing in mobile network environments. It achieves bidirectional synchronization of screen content and avoids network congestion caused by excessive data transmission, improving the system's availability under complex network conditions.

[0051] For example, the cropping size and cropping frequency in the sub-screen server can be set differently from those in the parent screen server, and the bandwidth can be allocated on demand according to both uplink and downlink traffic to avoid bandwidth bottlenecks.

[0052] For example, for the detection method of the above-mentioned interactive subgraph, dynamic tracking technology can be used to mark the coordinates of the subgraph to be uploaded in real time by listening to input events (such as mouse trajectory) or detecting pixel changes.

[0053] According to the embodiments provided in this application, merging the interactive subgraphs corresponding to multiple sub-screen servers to obtain merged data can include the following methods: When multiple sub-screen servers correspond to the same sub-graph in multiple current sub-graphs, the interaction sub-graphs corresponding to the multiple sub-screen servers are merged based on a predetermined merging strategy to obtain merged data. The interaction sub-graphs are marked with timestamps indicating that the interaction occurred in the corresponding sub-screen server. The predetermined merging strategy includes: using the sub-graph with the latest timestamp from the interaction sub-graphs corresponding to multiple sub-screen servers as the merged data; or, The interactive subgraphs corresponding to multiple sub-screen servers are overlaid as layers to obtain merged data.

[0054] In the embodiments provided in this application, the process of merging interaction subgraphs corresponding to multiple sub-screen servers to generate merged data can be implemented using one of the following two predetermined merging strategies: For a timestamp-based merging strategy, each interaction subgraph is marked with a timestamp, indicating the specific time of interaction on the corresponding sub-screen server. During merging, the parent screen server selects the interaction subgraph with the latest timestamp as the merged data, and the latest interaction operation is given priority and reflected in the final merged data. The timestamp-based merging strategy ensures that the latest interaction operations are updated in a timely manner, improving data real-time performance.

[0055] The layer overlay merging strategy involves overlaying the interactive subgraphs from multiple sub-screen servers, stacking them together according to a specific order and rules to form merged data. This preserves information from all interactive subgraphs and is suitable for scenarios requiring comprehensive consideration of all interactive operations. Through these two merging strategies, the master screen server can effectively integrate interactive information from different sub-screen servers, generating merged data reflecting all user interactions. This merged data can then be used to update image frames in the master screen server, enabling bidirectional screen interaction. The layer overlay merging strategy comprehensively considers all interactive operations, retains more interactive information, and enhances the richness of the interaction.

[0056] The two merging strategies described above can be selected based on different application scenarios and needs. This merging mechanism achieves multiple optimizations in multi-terminal interaction scenarios, including operation conflict resolution, information integrity preservation, and transmission efficiency, through a dual strategy of timestamp and layer overlay. It is suitable for remote collaborative systems with high requirements for operation timing and content integrity.

[0057] In addition to the above embodiments and optional embodiments, this application also provides an optional implementation method. Figure 3 A schematic flowchart of a screen synchronization method according to an embodiment of this application is shown, such as... Figure 3 As shown, the following is a detailed explanation.

[0058] When the process starts, the "begin" parameter is used as the starting point for the screen synchronization process, corresponding to the initial state of the parent screen server or child screen server starting the image synchronization task. Before data transmission for screen synchronization, the image cropping size and processing frequency are set. Initial configuration can be performed using a dynamic cropping method that adjusts communication resources. Based on network bandwidth, screen content type, etc., the current cropping size is preset, determining how many smaller sub-images to split a complete screen frame into. At the same time, screen frame comparison and synchronization frequencies are defined, determining the cycle for checking screen content changes and performing sub-image cropping.

[0059] Next, image cropping is performed. The complete screen frame is cut into several sub-image blocks according to the cropping size to prepare for subsequent difference detection. The cropped sub-images are stored in the image sequence in an encoded form, and the pointer is set to point to the current sub-image of the first image. The traversal process is initialized, and the difference is compared with the historical sub-images obtained based on the historical frames one by one, starting from the first sub-image.

[0060] From the stored previous version image sequence (historical frame data), find the sub-image (historical frame sub-image) with the same quality at the current comparison position according to the predetermined cropping order, and use it as the hash comparison benchmark. Locate the current sub-image in the current frame cropped sub-image sequence using a pointer, convert the two into MD5 hash values ​​for comparison. If they do not match, it is determined that the target sub-image needs to be updated synchronously; if they match, skip it and continue to the next one for comparison.

[0061] If the comparison is inconsistent, the modification status of the entire current frame is set to 1, and the sub-image with the difference (target sub-image) is sent to the sub-screen server using the UDP protocol. This achieves the transmission of only the difference area and reduces bandwidth consumption. This corresponds to the step of transmitting the target sub-image to the sub-screen server. If the comparison is consistent, it is determined whether the pointer has reached the end of the image sequence. If not, the comparison continues in a loop. When the end of the image sequence is reached, it is recorded as the pointer pointing to null, and the comparison of the entire frame ends.

[0062] When the pointer reaches the end and all sub-images match, the modification status flag for the current frame is set to 0, indicating that the entire frame has not changed, and the process temporarily ends. The flag of the current frame is checked. If the flag is 1, it indicates that at least one sub-image has changed, and the image sequence needs to be updated. At this time, the old historical frame sub-image sequence is updated, and the current frame sub-image sequence is used as the new historical frame data storage to prepare for the next round of comparison, forming a synchronous closed loop.

[0063] Through the above optional implementation methods, the amount of image data to be transmitted during the screen synchronization image processing can be controlled overall by flexibly configuring the image cropping size and cropping operation frequency in the screen synchronization service. Simultaneously, in the sub-image comparison processing, image number serialization is used to achieve the minimum number of modified sub-images to be filtered, reducing the amount of image data to be transmitted at the source. The combination of these two methods enables a high degree of flexibility in adjusting the service under different network conditions and usage requirements during screen synchronization applications, providing a screen synchronization display technology that can flexibly adapt to various bandwidth conditions.

[0064] Corresponding to the application scenarios and methods provided in the embodiments of this application, Figure 4 A schematic block diagram of a screen synchronization device according to an embodiment of this application is shown, such as... Figure 4 As shown in the illustration, this application also provides a screen synchronization device, including: The cropping module 401 is used to crop the current frame of the master screen server to obtain multiple current sub-images; Encoding module 402 is used to perform hash encoding on multiple current subgraphs to obtain multiple encoded current subgraphs; Subgraph comparison module 403 is used to compare the differences between each current subgraph after encoding and the corresponding historical subgraph after encoding, and to determine the current subgraph after encoding with differences as the target subgraph; multiple historical subgraphs are obtained by cropping the historical frames of the master screen server, and multiple historical subgraphs correspond one-to-one with multiple current subgraphs according to a predetermined cropping order. The sub-image sending module 404 is used to transmit the target sub-image to the sub-screen server that is synchronized with the parent screen server. The sub-screen server is used to update the image frame based on the target sub-image.

[0065] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0066] Figure 5 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the method described in the above embodiments. The number of memories 501 and processors 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and exchanging data.

[0067] In practical implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0068] Optionally, in a specific implementation, if the memory 501, processor 502 and communication interface 503 are integrated on a single chip, the memory 501, processor 502 and communication interface 503 can communicate with each other through an internal interface.

[0069] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0070] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0071] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0072] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0073] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0074] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0075] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0078] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0079] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0080] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0082] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A screen synchronization method, characterized in that, include: The current frame of the parent screen server is cropped to obtain multiple current sub-images; The multiple current subgraphs are hash-encoded to obtain multiple encoded current subgraphs; Compare the differences between each current sub-image after encoding and the corresponding historical sub-image after encoding, and determine the current sub-image after encoding with differences as the target sub-image; the multiple historical sub-images are obtained by cropping the historical frames of the master screen server, and the multiple historical sub-images correspond one-to-one with the multiple current sub-images according to a predetermined cropping order; The target sub-image is transmitted to a sub-screen server that is synchronized with the parent screen server, and the sub-screen server is used to update image frames based on the target sub-image.

2. The method according to claim 1, characterized in that, The method further includes: The cropping frequency is determined based on the channel resources between the mother screen server and the child screen server, and the display type representing the image change rate of the mother screen server. The video frames in the master screen server are cropped according to the cropping frequency.

3. The method according to claim 1, characterized in that, The step of comparing the differences between each current subgraph after encoding and its corresponding historical subgraph after encoding, and determining the current subgraph after encoding with differences as the target subgraph, includes: A first pointer is used to point to any first current subgraph in the encoded plurality of current subgraphs, and a second pointer is used to point to the first historical subgraph in the encoded plurality of historical subgraphs, wherein the first current subgraph corresponds to the first historical subgraph; If the first hash value of the encoded first current subgraph does not match the second hash value of the encoded first historical subgraph, the first current subgraph is determined to be the target subgraph. The method further includes: pointing the first pointer to the second current subgraph after the encoded first current subgraph; and pointing the second pointer to the second historical subgraph after the encoded first historical subgraph. Until the first pointer is null and the second pointer is null, the target sub-graph is sent to the sub-screen server.

4. The method according to claim 1, characterized in that, The method further includes: If the current cropping size of the current frame is different from the historical cropping size of the historical frame, the historical frame is re-cropped using the current cropping size to obtain multiple updated sub-images; The multiple update subgraphs are hash-encoded to obtain multiple encoded update subgraphs; By comparing the differences between each encoded current subgraph and its corresponding encoded updated subgraph, the encoded current subgraph with the differences is determined as the target subgraph.

5. The method according to claim 4, characterized in that, The method further includes: If the channel resources between the mother screen server and the child screen server change, the historical cropping size is updated based on the channel resources to obtain the current cropping size.

6. The method according to any one of claims 1 to 5, characterized in that, After transmitting the target sub-graph to the sub-screen server synchronized with the parent screen server, the method further includes: When there are multiple sub-screen servers, the interactive sub-graphs sent by the multiple sub-screen servers are obtained. The interactive sub-graph is the sub-graph in the image frame of the corresponding sub-screen server that contains the interactive operation part. If there are differences in the interaction subgraphs corresponding to the multiple sub-screen servers, the interaction subgraphs corresponding to the multiple sub-screen servers are merged to obtain merged data. Based on the merged data, update the image frames following the current frame in the master screen server.

7. The method according to claim 6, characterized in that, The step of merging the interactive subgraphs corresponding to the multiple sub-screen servers to obtain merged data includes: When the interaction subgraphs corresponding to the multiple sub-screen servers correspond to the same subgraph in the multiple current subgraphs, the interaction subgraphs corresponding to the multiple sub-screen servers are merged based on a predetermined merging strategy to obtain the merged data. The interaction subgraph is marked with a timestamp indicating that the interaction was performed in the corresponding sub-screen server. The predetermined merging strategy includes: using the sub-graph with the latest timestamp from the interaction sub-graphs corresponding to the multiple sub-screen servers as the merged data; or, The interactive subgraphs corresponding to the multiple sub-screen servers are overlaid as layers to obtain the merged data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images