Multi-screen cooperative control system and method based on digital twinning

Abstract

The invention discloses a multi-screen cooperative control system based on digital twinning and a method thereof, and relates to the technical field of remote cooperation. The system comprises an H5 page synchronization JS module, a synchronous communication service module and a synchronous communication management platform module. The H5 page synchronization JS module is embedded into a client page in a non-intrusive mode, captures a user operation event and generates an operation instruction; the synchronous communication service module establishes a secure communication link, transmits an operation instruction and multimedia data, and is internally provided with a control right arbitration mechanism to realize dynamic switching of rights of a master controlled party and a controlled party; and the synchronous communication management platform module monitors and analyzes the cooperation process. A virtual mapping model of a client page is constructed through a digital twinning technology, real-time synchronization and two-way control of an operation state are realized, the problems of insufficient interaction and low operation guidance efficiency in remote assistance are solved, and face-to-face multi-screen collaborative operation experience is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote collaboration, and specifically relates to remote collaborative operation and digital twin technology, in particular to a multi-screen collaborative control system and method based on digital twin, an electronic device and a storage medium, which is suitable for remote sales, online customer service, remote education and other scenarios that require real-time interface sharing and bidirectional control. BACKGROUND

[0002] Currently, in the scenarios of remote sales and customer service, sales personnel usually send exclusive operation links (generally in the form of cards, QR codes, etc.) to customers through emails, short messages or instant messaging tools (such as personal WeChat, enterprise WeChat, etc.) to guide customers to complete registration, quotation, information filling, ordering and other operations independently. However, when customers encounter problems during operation, the existing technical solutions have obvious limitations: insufficient interaction capability, traditional communication methods (such as repeated WeChat or telephone communication) cannot realize real-time sharing and synchronous guidance of the operation interface, and the efficiency is low; lack of bidirectional control mechanism.

[0003] Existing remote assistance or screen projection technologies mostly adopt a one-way mirroring mode, and customers cannot authorize sales personnel to operate directly, making it difficult to achieve a "hands-on" guidance effect; the collaborative tool has low integration, audio and video communication, interface operation and annotation tools are independent of each other, and there is a lack of unified platform support, resulting in interruption of the operation process and incoherent experience. For example, some existing technical solutions, such as patent CN116432940A (collaborative control system based on digital twin technology), although they mention using digital twin technology for state synchronization, they do not optimize for real-time operation synchronization on the web end, dynamic switching of control rights, and non-intrusive integration, especially in the specific scenario of remote sales.

[0004] Therefore, there is an urgent need for a lightweight multi-screen collaboration solution that supports bidirectional control and multimedia integration to improve the efficiency and experience of remote interaction. SUMMARY

[0005] The purpose of the present application is to provide a multi-screen collaborative control system and method based on digital twin, an electronic device and a storage medium to solve the problems of difficult interface sharing, low operation guidance efficiency and insufficient multi-tool collaboration in remote interaction.

[0006] In a first aspect, a multi-screen collaborative control system based on digital twin is provided, comprising: an H5 page synchronization JS module for being embedded in a client page of each terminal device in a non-intrusive manner, capturing user operations and interaction events, and generating standardized operation instructions; The synchronous communication service module establishes a secure communication link with the synchronous JS module of the H5 page to transmit the operation instructions, audio and video data and graffiti data. It also has a built-in control arbitration mechanism to intelligently allocate the permissions of the master and the controlled party according to the operation context and supports dynamic switching of permissions. The synchronous communication management module is used to monitor, record, trace, and analyze the entire process of collaborative operations, and generate traceable communication archives.

[0007] Furthermore, the H5 page synchronization JS module adopts a highly cohesive and loosely coupled architecture design. It achieves seamless integration with existing web projects by introducing JS plugins and supports developers to customize variables, attributes and events to adapt to different synchronization scenarios.

[0008] Furthermore, the H5 page synchronization JS module integrates control logic for the synchronization process, which can realize the switching of control roles between the master and the controlled party, as well as the application and transfer of control rights, according to the pre-set synchronization control mode.

[0009] Furthermore, the H5 page synchronization JS module also includes a third-party project integration module, which integrates third-party tools or existing projects to achieve page synchronization functionality that includes third-party projects.

[0010] Furthermore, third-party projects include: web-based audio and video communication tools, and drawing board communication tools.

[0011] Furthermore, the synchronous communication service module generates synchronous communication data and unique communication credentials during the service process, and integrates a real-time transmission service channel for graffiti, audio, and video data.

[0012] Furthermore, the synchronous communication service module supports independent deployment or rapid implementation in a SaaS model.

[0013] Furthermore, the synchronous communication management module is specifically used to realize the retrospective of the communication process, real-time monitoring, data analysis of the business communication process, and communication file management, and supports data monitoring and collection according to predetermined rules.

[0014] Furthermore, the system constructs a virtual mapping model of the client page using digital twin technology. This model includes the page's DOM structure, element attributes, and interaction logic. By transmitting operation instructions and data differences, it ensures that operation instructions are accurately synchronized and bidirectionally controlled between the two interfaces. The model construction process includes a static construction process and a dynamic construction process. The static construction process includes the construction of structural twins, and the dynamic construction process includes the construction of state and logical twins.

[0015] Secondly, a multi-screen collaborative control method based on digital twins is provided, characterized in that it employs the multi-screen collaborative control system based on digital twins described in the first aspect, the method comprising: Embed H5 page synchronous JS components in the client page to capture user operation events and generate standardized operation instructions; Establish a secure communication link between the client and the synchronous communication service to transmit the operation instructions, audio and video data, and graffiti data; Based on digital twin technology, a virtual mapping model of the client page is built on the server side to achieve real-time synchronization and two-way control of the operation status; Control rights management is performed through the synchronous communication service, and the permissions of the master and the controlled are intelligently allocated according to the operation context, supporting dynamic switching of permissions; The collaborative process is monitored, recorded, and analyzed using a synchronous communication management platform, generating traceable communication archives.

[0016] Thirdly, an electronic device is provided, including a processor and a memory, wherein a computer program is stored in the memory, characterized in that, when the computer program is executed by the processor, it implements the multi-screen collaborative control method based on digital twins as described in the second aspect.

[0017] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, it implements the multi-screen collaborative control method based on digital twins as described in the second aspect.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: Two-way interactive experience optimization: Through digital twin models and dynamic permission switching, "synchronous operation at both ends" (such as synchronous input, clicking, and swiping) is realized. Customers can authorize sales personnel to operate on their behalf at any time, which significantly reduces communication costs and improves conversion rates.

[0019] Deep multimedia collaboration: Natively integrates audio and video calling and precise drawing functions, supports uninterrupted voice communication when jumping between multiple pages, and supports multi-person screen interaction, ensuring the continuity and immersion of the interaction process.

[0020] Lightweight and highly flexible: The H5 JS components adopt a non-intrusive design, allowing for rapid integration into existing business systems without modification; the synchronous communication service supports multiple deployment modes to adapt to the needs of different enterprises.

[0021] Process traceability and optimizability: The entire process of operation, audio and video recording, and doodles can be recorded to generate visual video archives, which facilitates problem tracing, employee training, and business data analysis. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention.

[0023] Figure 2 This is a flowchart of a multi-screen collaborative control method for digital twins in one embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the device interface of the synchronous communication management platform in one embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments.

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] Please refer to Figure 1 This is a system provided in the embodiments of the present invention, demonstrating the interaction relationship between the H5 page synchronous JS module, the synchronous communication service module, and the synchronous communication management platform module, mainly including three core modules: H5 Page Synchronization JS Module: This module is a lightweight JavaScript component embedded in the client-side H5 page. It adopts a highly cohesive, loosely coupled, and non-intrusive design. Simply by importing the JS plugin, it can capture user interaction events (such as clicks, swipes, and input) on the page and generate standardized operation commands. It supports developers to define custom variables, properties, and events, and can integrate with third-party web audio / video tools and drawing tools to achieve real-time audio / video communication and synchronized drawing functionality. Crucially, it integrates synchronization process control logic, enabling dynamic switching of control between the master and the controlled party based on preset modes.

[0030] This module is embedded in the client-side H5 page as a JavaScript plugin, adopting a non-intrusive design that requires no modification to the original business code.

[0031] By listening to DOM events (such as click, input, scroll), user actions are captured, and standardized operation instructions (including operation type, target element ID, coordinate information, timestamp, etc.) are generated.

[0032] Developers can customize event types and synchronization strategies (such as whether to synchronize scroll events, input delay thresholds, etc.).

[0033] When integrating third-party tools (such as WebRTC audio and video components, Canvas drawing board), the graffiti trajectory, audio and video stream data and operation instructions are transmitted synchronously through the event delegate mechanism.

[0034] Synchronous Communication Service Module: This module serves as the system's central hub, establishing a secure and stable communication connection with the H5 page's synchronized JavaScript. It is responsible for transmitting operation commands, audio and video streams, and drawing data, and generates synchronous communication data and credentials during the service process. Its built-in control arbitration mechanism intelligently determines, manages, and distributes master and slave permissions based on the actual communication process. This module supports independent deployment or SaaS service, providing flexible deployment options.

[0035] It uses the WebSocket protocol to establish a two-way secure communication link between the client and the server, and supports TLS encrypted transmission.

[0036] Built-in control arbitration mechanism: Manages the permissions of the controlling party and the controlled party through a session state machine. For example, when the controlled party triggers a "help" event, the server automatically switches the control permission to the other party; a buffer time can be set during permission switching to prevent conflicts.

[0037] Multimedia data channel separation: operation commands are transmitted first through low-latency channels; audio and video streams are guaranteed to be smooth through adaptive bitrate technology; graffiti data is transmitted using lightweight coordinate sequence compression.

[0038] Synchronous Communication Management Module: This module is an optional monitoring and management platform. It is used to realize functions such as retrospective analysis of synchronous communication processes, real-time monitoring, business data analysis, and communication file management. It also supports monitoring or collecting data according to predefined rules, thereby providing support for operational optimization and quality assessment.

[0039] Implemented based on a microservices architecture, it provides a RESTful API for management to call.

[0040] It supports real-time monitoring of session status (such as the number of online users and synchronization latency metrics), recording of full-link operation logs (which can be traced back to specific DOM operation steps), and generating session analysis reports (such as operation time graphs).

[0041] The communication files are stored in encrypted video format, and key operation nodes are marked with a timeline to facilitate quick problem location.

[0042] Optionally, to realize the twin-screen interaction system of the present invention based on the digital twin virtual mapping model, the present invention also provides the following specific implementation schemes.

[0043] First, construct a digital twin model: that is, create a "digital twin". Construct a deep model that includes structure, state, and logic. Specific implementation methods include static construction process and dynamic construction process.

[0044] A1. The static construction process, namely the construction of structural twins, is as follows: Method: On the server side, the system loads the target H5 page through a browser (such as Puppeteer or Playwright). The browser can completely parse the page's DOM tree and CSS stylesheets, thereby constructing a virtual DOM tree that is completely consistent with the client-side page.

[0045] Content: The virtual model contains precise information about each element, such as tag name, ID, class name, position, size, style attributes, etc. It is like the "skeleton" and "skin" of a webpage.

[0046] B1. Dynamic construction process, namely the construction of state and logical twins, the specific methods and contents of which are as follows: For dynamic pages driven by JavaScript (especially frameworks like Vue and React), the system captures the page's data state and event binding logic by injecting scripts or using developer tool protocols (such as CDP).

[0047] Clicking a button will trigger a function that modifies a data variable, thereby changing the display of the page.

[0048] II. Synchronization and Control of Twin Models With a precise digital twin model, synchronization and control are no longer simple pixel-by-pixel replication, but precise operations based on structure and instructions.

[0049] A2. Operation command-based execution and twin model execution: When the controlling party (such as a salesperson) performs an operation on their digital twin model interface (such as clicking a button), the H5 page's synchronous JS module does not transmit screenshots, but instead captures the standardized command for this operation. The command format might be: { type: 'click', target: '#submit-button', timestamp: 123456789}.

[0050] This instruction is first sent to the server's digital twin model. The server then simulates the "click" operation on the twin model.

[0051] Before executing a command, the server can perform a validity check. For example, if a button in the twin model is disabled, the server can reject the operation and return an error message, thus preventing invalid or erroneous operations from occurring on the controlled party's real page.

[0052] B2. Differential Synchronization: When the controller's operations are executed on the server-side twin model, they may cause changes to the twin model's DOM structure (for example, popping up a modal or adding an item to a list).

[0053] The system uses a Diff algorithm to compare the differences between the virtual DOM of the twin model before and after the operation, and then only sends the differences (i.e. the changed data or DOM fragments) to the client of the controlled party (the client).

[0054] After receiving the discrepancy data, the defendant's H5 page synchronous JS module will intelligently update the local page instead of refreshing the entire page.

[0055] This method involves minimal data transmission, low latency, and perfectly maintains the page state (such as the content entered in a form), achieving efficient and accurate synchronization.

[0056] C2. Implementation of bidirectional control: Traditional screen mirroring is unidirectional, while the core of this patent is bidirectional control. This means that the operations of the controlled party also need to be synchronized to the controlling party.

[0057] The process is as follows: the controlled party's operations are also captured as instructions, sent to the server's digital twin model for execution, and then the calculated differences are synchronized to the controlling party.

[0058] In this way, regardless of which party operates, the interfaces of both parties can maintain a consistent state through the centralized digital twin model on the server side, achieving true "what you see is what I see, what you operate is what I operate".

[0059] III. Integration with Control Arbitration Mechanisms The digital twin model provides crucial context for control arbitration. The arbitration mechanism (within the synchronous communication service module) can intelligently allocate permissions based on the operational intentions of both parties and the current state of the twin model.

[0060] For example, in a car insurance scenario, when a customer begins typing in a form input box, the arbitration mechanism can temporarily shift control to the customer to prevent insurance sales personnel from interrupting their input. When the customer stops typing or clicks "help," the arbitration mechanism then switches control back to the insurance sales personnel. All these decisions rely on the assessment of the current state of the digital twin model.

[0061] Optionally, the control switching process of the present invention is as follows: Permission Request: The controlled party initiates a permission transfer request via an interface button or shortcut key. The H5 page synchronously sends a ControlRequest event with a session ID to the server via the JS module.

[0062] Arbitration decision: The server decides whether to approve the request based on the operation context (such as whether the current page is a sensitive page, and the historical operation frequency of both parties). If the controlling party does not perform any operation within 10 seconds, the switch is automatically approved.

[0063] State synchronization: After the permission switch, the server broadcasts the ControlSwitch event to both clients, updates the interface prompts (such as the cursor on the master client turning red), and freezes the input events of the non-master client until the switch is complete.

[0064] To further understand the present invention, the present invention also provides a specific embodiment of a multi-screen collaborative control method based on digital twins, including: Embed H5 page synchronous JS components in the client page to capture user screen operation events and generate standardized operation instructions; Establish a secure communication link between the client and the synchronous communication server to transmit the operation instructions, audio and video data, and graffiti data; Based on digital twin technology, a virtual mapping model of the client page is built on the server side to achieve real-time synchronization and two-way control of the operation status; Control rights management is performed through the synchronous communication service, which intelligently allocates permissions to the screen master and screen slave based on the operation context, and dynamically switches permissions according to the configuration. The collaborative process is monitored, recorded, and analyzed using a synchronous communication management platform, generating traceable communication archives.

[0065] In one embodiment, the present invention also provides a multi-screen collaborative control method based on digital twins, the process of which can be referred to in detail. Figure 2 The method employs the system described above and includes the following steps: Embed H5 page synchronous JS components in the client page to capture user screen operation events and generate standardized operation instructions; Establish a secure communication link between the client and the synchronous communication server to transmit the operation instructions, audio and video data, and graffiti data; Based on digital twin technology, a virtual mapping model of the client page is built on the server side to achieve real-time synchronization and two-way control of the operation status; Control rights management is performed through the synchronous communication service, which intelligently allocates permissions to the screen master and screen slave based on the operation context, and dynamically switches permissions according to the configuration. The collaborative process is monitored, recorded, and analyzed using a synchronous communication management platform, generating traceable communication archives.

[0066] To further illustrate the above embodiments in detail, the following is an example of a scenario involving pushing online car insurance quotes: S1. Initialization Phase: When a customer clicks on the car insurance quote link pushed by the salesperson via WeChat, the browser loads the H5 quote page. The embedded H5 page's synchronized JS module automatically initializes and begins to non-intrusively capture all interactive elements on the page (such as form input boxes, radio buttons, and submit buttons).

[0067] S2. Collaboration Trigger: When a customer encounters a problem while filling in vehicle information, they click the "Simultaneous Screen Sync" button on the page. The H5 page's synchronized JS module then sends a collaboration request to the synchronized communication service module. After server verification, a unique session ID is generated, and an audio / video communication connection is automatically established between the two parties.

[0068] S3. Permission Switching and Collaborative Operation: The control arbitration mechanism in the synchronous communication service module intelligently switches the "master control" permission to the salesperson's end based on the customer's operational intent (clicking for help). The salesperson sees the customer's page in real-time on their own interface through a digital twin model, and their operational instructions (such as helping the customer select a car model or fill in the VIN code) are transmitted in real-time and accurately synchronized to the customer's browser page via the synchronous communication service. During this process, both parties conduct real-time voice communication through integrated audio and video functions.

[0069] S4. Graffiti Annotation Assistance: Sales personnel use the graffiti tool integrated into the collaboration interface to circle and mark key items on the shared screen of both parties. This graffiti data is synchronized in real time through an independent channel.

[0070] S5. Process Recording and Analysis: The entire collaborative process, including all operation steps, voice dialogues, and drawing trajectories, is fully recorded by the synchronous communication management platform module. It can generate business analysis reports according to rules, and at the same time generate traceable encrypted video archives.

[0071] S6. End of Session: After the quote is completed, the session ends and control is automatically returned.

[0072] The above embodiments demonstrate the effectiveness of the present invention in improving the efficiency of remote operation guidance and reducing customer error rates. This system is not only suitable for insurance telemarketing, but can also be extended to various scenarios requiring highly reliable real-time interaction, such as online financial transactions, remote assistance at government service centers, and remote software technical support.

[0073] Please refer to Figure 3 , Figure 3 A schematic diagram of the hardware structure of a computer for implementing the above-described methods and embodiments of this disclosure is shown. Figure 3 As shown, the computer device may include a processor, a computer-readable storage medium, a bus, and a communication unit.

[0074] In its implementation, the processor executes computer-executable instructions stored in the computer-readable storage medium, enabling it to execute the multi-screen collaborative control system and method described in the above method embodiments. The processor, computer-readable storage medium, and communication unit can be connected via a bus. The specific implementation process of the processor can be found in the various method embodiments executed by the aforementioned computer device, as their implementation principles and technical effects are similar, and will not be repeated here.

[0075] Computer-readable storage media can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The memory is used to store programs or data.

[0076] In summary, this invention constructs a virtual interface mapping using digital twin technology, combining bidirectional control and multimedia synchronization to achieve cross-platform, high real-time multi-screen collaboration. Compared to traditional solutions, it demonstrates significant improvements in interaction efficiency, integration, and traceability. The above descriptions are merely preferred embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-screen collaborative control system based on digital twins, characterized in that, include: The H5 page synchronous JS module is used to non-intrusively embed into the client page of various terminal devices, capture the user's operation and interaction events on the screen, and generate standardized operation instructions. The synchronous communication service module establishes a secure communication link with the synchronous JS module of the H5 page to transmit the operation instructions, audio and video data and doodle data. It also has a built-in control arbitration mechanism to intelligently allocate permissions between the screen master and the screen slave according to the operation context, and to dynamically switch permissions according to the configuration. The synchronous communication management module is used to monitor, record, trace, and analyze the entire process of multi-screen collaborative operation, and generate traceable communication archives.

2. The system as described in claim 1, characterized in that, The H5 page synchronization JS module adopts a highly cohesive and loosely coupled architecture design. It achieves seamless integration with existing web projects by introducing JS plugins and supports developers to customize variables, properties and events to adapt to different synchronization scenarios.

3. The system as described in claim 2, characterized in that, The H5 page synchronization JS module integrates control logic for the synchronization process, and can realize the switching of control roles between the master and the controlled party, as well as the application and transfer of control rights, according to the pre-set synchronization control mode. The control logic also includes an arbitration mechanism. The arbitration mechanism intelligently allocates permissions based on the operational intentions of both parties and the current state of the twin model, and decides whether to approve the request based on the operational context. After the permission switch, the server broadcasts the control switch event to both clients, updates the interface prompts, and freezes the input events of the non-controlling party until the switch is completed.

4. The system as described in claim 2, characterized in that, The H5 page synchronization JS module also includes a third-party project integration module, which integrates third-party tools or existing projects to achieve page synchronization functionality that includes third-party projects; third-party projects include: web-based audio and video communication tools, and drawing board communication tools.

5. The system as described in claim 1, characterized in that, The synchronous communication service module generates synchronous communication data and unique communication credentials during the service process, and integrates real-time transmission service channels for graffiti, audio, and video data; the synchronous communication service module supports independent deployment or rapid implementation in SaaS mode.

6. The system as described in claim 1, characterized in that, The synchronous communication management module is specifically used to realize the retrospective of communication process, real-time monitoring, data analysis of business communication process, communication file management, and supports data monitoring and collection according to predetermined rules.

7. The system as described in claim 1, characterized in that, The system constructs a virtual mapping model of the client page using digital twin technology. This model includes the page's DOM structure, element attributes, and interaction logic. By transmitting operation instructions and data differences, it ensures that operation instructions are accurately synchronized and bidirectionally controlled between the two interfaces. The model construction process includes a static construction process and a dynamic construction process. The static construction process includes the construction of structural twins, and the dynamic construction process includes the construction of state and logical twins.

8. A multi-screen collaborative control method based on digital twins, characterized in that, The method using the system as described in any one of claims 1 to 7 comprises the following steps: Embed H5 page synchronous JS components in the client page to capture user screen operation events and generate standardized operation instructions; Establish a secure communication link between the client and the synchronous communication server to transmit the operation instructions, audio and video data, and graffiti data; Based on digital twin technology, a virtual mapping model of the client page is built on the server side to achieve real-time synchronization and two-way control of the operation status; Control rights management is performed through the synchronous communication service, which intelligently allocates permissions to the screen master and screen slave based on the operation context, and dynamically switches permissions according to the configuration. The collaborative process is monitored, recorded, and analyzed using a synchronous communication management platform, generating traceable communication archives.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-screen collaborative control method based on digital twins as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-screen collaborative control method based on digital twins as described in claim 8.

Images