Remote assistance method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of communication, and discloses a remote assistance method and device, electronic equipment and a storage medium. The method is executed by a first device, and comprises the following steps: receiving a change event sent by a second device; based on the change event, updating the current DOM mirror image tree to obtain an updated DOM mirror image tree, the DOM mirror image tree being a mirror image body of the DOM tree of the second device; an interaction event triggered by a user in the updated DOM mirror image tree is captured, a first event object is generated, and the first event object comprises an event type, an identifier of an acted DOM node and event parameters; and after serializing the first event object, sending the serialized first event object to the second equipment, so as to trigger the serialized first event object on the corresponding DOM node in the DOM tree. According to the embodiment of the invention, the invasiveness of remote assistance can be reduced, and the privacy protection intensity for the assisted party can be improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a remote assistance method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, electronic devices can be remotely controlled to resolve problems in other users' electronic devices.

[0003] In related technologies, the assisting party (i.e., the aforementioned electronic device) and the assisted party (i.e., the aforementioned electronic device of other users) first establish a remote connection. Then, the assisting party can display the simulated interface of the assisted party. Finally, the assisting party can perform at least one assistance operation in the simulated interface. After performing each of the at least one assistance operation, the assisting party sends an assistance message containing each assistance operation to the assisted party in real time, so that the assisted party can perform the operation on its own according to the received assistance message containing each assistance operation, thereby achieving the purpose of remote assistance.

[0004] However, the above methods require the recipient to install independent client software, which is highly intrusive and exposes the recipient's entire desktop environment during remote assistance, posing privacy and security risks. Summary of the Invention

[0005] The purpose of this application is to provide a remote assistance method, device, electronic device, and storage medium, which aims to reduce the intrusiveness of remote assistance and enhance the privacy protection for the assisted party.

[0006] This application provides a remote assistance method, executed by a first device, the method comprising: Receive change events sent by the second device; Based on the change event, the current DOM mirror tree is updated to obtain the updated DOM mirror tree; the DOM mirror tree is a mirror image of the DOM tree of the second device; Capture user interaction events triggered in the updated DOM mirror tree and generate a first event object; the first event object contains the event type, the identifier of the DOM node it affects, and event parameters; The first event object is serialized and sent to the second device to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

[0007] In some embodiments, the method further includes the following steps before receiving the change event sent by the second device: Get a DOM snapshot of the second device; Based on the DOM snapshot, construct the DOM mirror tree.

[0008] In some embodiments, the remote assistance method further includes: The device receives a video stream and a canvas binding message sent by the second device; the video stream is image data generated by the Canvas element detected by the second device, and the canvas binding message is used to characterize the mapping relationship between the video stream and the DOM nodes in the DOM tree. Based on the canvas binding message, the Canvas element node in the DOM mirror tree is determined, and the image data extracted from the video stream is drawn onto the Canvas element node; Capture user-triggered interaction events in the Canvas element display area and generate a second event object; the Canvas element display area is a visualization area drawn based on the video stream, and the second event object contains the event type, the identifier of the node corresponding to the Canvas element, and event coordinate parameters.

[0009] The second event object is serialized and sent to the second device to trigger the serialized second event object on the Canvas element.

[0010] In some embodiments, the remote assistance method further includes: Receive the target coordinate information sent by the second device; Based on the target coordinate information, the scrollability of the mirror nodes in the DOM mirror tree is detected to obtain the target scroll container mirror node and its corresponding scroll increment. The identifier of the target scroll container mirror node and the scroll increment are sent to the second device to determine the target scroll container node in the DOM tree and modify its scroll offset attribute.

[0011] In some embodiments, the scrollability detection of mirror nodes in the DOM mirror tree based on the target coordinate information includes: Based on the target coordinate information, locate the target mirror node in the DOM mirror tree; Perform scrollability detection on the target mirror node and its ancestor mirror nodes, and select the target scrollable container mirror node from the mirror nodes that meet the scrollability detection conditions; Calculate the scroll increment of the target scroll container mirror node.

[0012] In some embodiments, the remote assistance method further includes: Receive the tab update message sent by the second device; Based on the tab update message, update the list of available tabs displayed in the user interface; The system captures the user's selection action in the list of available tabs, sends the identifier of the target tab to the second device, and causes the second device to activate the target tab as the currently active tab.

[0013] In some embodiments, the remote assistance method further includes: In response to a guidance request initiated by the second device, establish a video call with the second device, and / or share the application interface of the first device with the second device, and display interactive events triggered by the user in the application interface of the first device.

[0014] This application also provides a remote assistance device, including: The first module is used to receive change events sent by the second device; The second module is used to update the current DOM mirror tree based on the change event to obtain the updated DOM mirror tree; the DOM mirror tree is a mirror image of the DOM tree of the second device; The third module is used to capture the interaction events triggered by the user in the updated DOM mirror tree and generate a first event object; the first event object contains the event type, the identifier of the DOM node to which it is applied, and the event parameters; The fourth module is used to serialize the first event object and send it to the second device so as to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

[0015] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described remote assistance method.

[0016] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned remote assistance method.

[0017] The beneficial effects of this application are as follows: By constructing a DOM mirror tree on the electronic device used by the assisting party and updating it based on change events sent by the assisted party, remote assistance can be carried out without installing intrusive client software on the assisted party's electronic device, lowering the user threshold and improving system compatibility. By transmitting DOM change events and interaction events, a specific web page interface of the assisted party's electronic device is simulated on the assisting party's electronic device, and only the user's operations on the simulated interface are accurately mapped to the corresponding DOM nodes on the second device, effectively avoiding privacy leakage of the assisted party and significantly improving the privacy protection and security of remote assistance. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating the application environment of the remote assistance method provided in the embodiments of this application.

[0019] Figure 2 This is a flowchart of the remote assistance method provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the structure of the remote assistance device provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. Furthermore, the information, data, and signals involved in the embodiments of this application are all authorized by relevant parties or have been fully authorized by all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0025] In traditional remote assistance technologies, the recipient needs to install independent client software to establish a remote connection, leading to increased system intrusion and exposing the recipient's entire desktop environment during the assistance process, posing privacy and security risks. The installation process involves granting system permissions and consuming additional resources; the complete transmission of the desktop environment leaves sensitive information in an uncontrolled access state, impacting the system's security isolation capabilities and user privacy protection levels. For example, in enterprise remote technical support scenarios, the recipient is an employee whose electronic device runs a desktop environment containing internal company documents. When the assisting party initiates a remote connection request, the recipient must install dedicated client software, a process requiring operating system-level permissions and consuming system storage resources. Furthermore, after the connection is established, the recipient's entire desktop interface is transmitted to the assisting party in real time, including unauthorized access to sensitive areas, potentially leading to the unnecessary exposure of documents containing trade secrets. Thus, the system's security boundary is weakened, user trust in remote assistance technology decreases, and the risk of unauthorized data access increases.

[0026] If the above issues are not addressed, privacy and security risks will persist, potentially leading to unauthorized dissemination of sensitive information. Furthermore, the invasive installation requirements will limit the applicability of remote assistance technology in privacy-sensitive environments and hinder the large-scale deployment of technical solutions.

[0027] Based on this, embodiments of this application provide a remote assistance method, apparatus, device, and medium. The remote control mechanism based on DOM mirroring avoids transmitting the entire desktop screen, thereby protecting user privacy. At the same time, it eliminates the need to install dedicated client software, reducing the intrusiveness of remote assistance.

[0028] Figure 1 This is a diagram illustrating the application environment of the remote assistance method provided in the embodiments of this application. For example... Figure 1As shown in the embodiment of this application, the remote assistance method is applied to a remote assistance system. The system includes a first device 101 and a second device 102. The first device 101 and the second device 102 are connected via network communication. The second device 102 is used to send change events of the second device 102 to the first device 101. The first device 101 is used to receive the change events sent by the second device 102, update the current DOM mirror tree based on the change events to obtain an updated DOM mirror tree, capture user interaction events triggered in the updated DOM mirror tree, generate a first event object, serialize the first event object, and send it to the second device 102 to trigger the serialized first event object on the corresponding DOM node in the DOM tree. The DOM mirror tree is a mirror image of the DOM tree of the second device 102, and the first event object includes the event type, the identifier of the affected DOM node, and event parameters. It can be understood that the first device 101 is the electronic device used by the assisting party in remote assistance, and the second device 102 is the electronic device used by the assisted party in remote assistance. The first device 101 is responsible for receiving interface update information from the second device 102, simulating the interface of the second device 102, capturing user operations on the simulated interface, and feeding these operations back to the second device 102. The second device 102 is responsible for generating and sending its interface status information to the first device 101, receiving operation instructions from the first device 101, and then executing the corresponding operations.

[0029] Figure 2 This is a flowchart of the remote assistance method provided in the first embodiment of this application. (See attached document.) Figure 2 In one embodiment, the method is performed by a first device, including but not limited to steps S201 to S204.

[0030] Step S201: Receive the change event sent by the second device.

[0031] Change events are events generated when the DOM tree on the second device undergoes structural or content changes. For example, when the content of a webpage on the second device is added, deleted, modified, styled, or its attributes are updated, corresponding change events are generated. These events are sent to the first device to synchronize and update the mirrored DOM tree.

[0032] The DOM tree, or Document Object Model tree, is a tree-like structure that a browser builds in memory after parsing web page content. Each node in the DOM tree represents an element, attribute, or text content within the web page, defining methods for accessing and manipulating web page elements.

[0033] The first device can receive interface change information from the second device in various ways. For example, the first device can periodically send requests to the second device to query for new change events. When the second device detects changes in the structure, content, or style of its DOM tree, it encapsulates these changes into change events and responds to the first device's query request by sending the change events to the first device. As an optional implementation, a persistent communication channel can be established between the first and second devices, for example, through long-connection technology. When any changes occur in the second device's DOM tree, the second device can proactively push change events to the first device.

[0034] Step S202: Based on the change event, update the current DOM mirror tree to obtain the updated DOM mirror tree.

[0035] The DOM mirror tree is a mirror image of the DOM tree on the second device. In essence, the DOM mirror tree is a mirror image built on the first device to simulate the DOM tree of the second device. This DOM mirror tree is updated in real time by receiving change events sent by the second device, thus presenting a visual effect and structure on the first device consistent with the current interface of the second device.

[0036] After receiving change events from the second device, the first device updates its locally maintained DOM mirror tree based on these events. One implementation is to include a complete snapshot of the second device's DOM tree in the change event; the first device then directly replaces the current DOM mirror tree with the new snapshot, thus achieving the update. Another implementation is to include only incremental modification instructions in the change event, such as "add a child node under a parent node," "delete a node," or "modify the attribute value of a node." Based on these incremental instructions, the first device performs corresponding local modifications on the current DOM mirror tree to keep it synchronized with the second device's DOM tree. This DOM mirror tree, as a mirror image of the second device's DOM tree, should maintain its structure and content as consistent as possible with the second device's DOM tree.

[0037] Step S203: Capture the interaction events triggered by the user in the updated DOM mirror tree and generate the first event object.

[0038] Interaction events triggered by the user in the updated DOM mirror tree refer to events triggered by the user in the simulated interface on the first device through mouse clicks, keyboard input, touch swipes, etc. These events reflect the actions the user wishes to perform on the second device during remote assistance.

[0039] The first event object contains the event type, the identifier of the DOM node on which the event occurred, and event parameters. It can be understood that the first event object is a wrapper around the captured interactive event. This first event object typically contains the event type (e.g., click, input), the identifier of the DOM node on which the event occurred (e.g., element ID or XPath path), and event-related parameters (e.g., click coordinates, input text content).

[0040] Once the DOM mirror tree on the first device is updated, the user can interact with the simulated interface provided by the first device. The first device needs to be able to capture these user actions. For example, a global event listener can be set on the root element of the DOM mirror tree to capture all bubbling mouse clicks, keyboard inputs, scrolling, and other interactive events. When an interactive event is captured, the first device will identify the event type (such as "click", "keydown"), the DOM node affected by the event (its identifier is obtained through the `target` property of the event object), and the event-related parameters (such as the coordinates of the click and the key value pressed). This information is then encapsulated into a first event object.

[0041] Step S204: Serialize the first event object and send it to the second device to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

[0042] After generating the first event object, it is serialized. For example, the first event object can be converted into a standard data format, such as a JSON string. This serialized data is then sent to the second device via a network communication protocol (such as HTTP or WebSocket). Upon receiving the serialized data, the second device deserializes it to reconstruct the original first event object. Then, based on the event type, active node identifier, and event parameters contained in the first event object, the second device finds the corresponding DOM node in its own DOM tree and simulates triggering the corresponding interactive event, thereby executing the user's actions on the first device.

[0043] The following example will provide a more detailed explanation of the above technical solution: Suppose user A is browsing a webpage using a second device. This webpage contains a form that requires users to fill out an information form. User A encounters a problem while filling out the form and needs remote assistance from user B using a first device.

[0044] First, after a remote assistance session is established, the second device continuously monitors changes to the DOM tree of its current webpage. When user A performs an action on the second device, such as switching pages, entering text, or the page content updating automatically, the DOM tree of the second device changes accordingly. These changes are captured by the second device and encapsulated into a series of change events. For example, when user A enters text in an input box on a form, the second device generates a change event containing the identifier of the input box node and the new text content. The second device then sends these change events to the first device.

[0045] Upon receiving these change events, the first device updates its locally maintained DOM mirror tree based on these events. For example, if the received change event concerns a change in the text content of an input box, the first device will find the corresponding input box node in its DOM mirror tree and update its text content, ensuring that the simulated interface displayed on the first device remains synchronized with the actual interface on the second device. In this way, the interface seen by user B on the first device is always a real-time mirror of the webpage on user A's second device.

[0046] Next, User B observes the problem encountered by User A on the first device and decides to assist. For example, User B clicks a button or enters text in an input box in the simulated form interface on the first device. The first device captures these interaction events triggered by User B in the DOM mirror tree. For example, when User B clicks the "Submit" button on the simulated interface, the first device generates a first event object, which contains the event type (e.g., "click"), the identifier of the DOM node affected (e.g., "submitButton"), and event parameters (e.g., the coordinates of the click).

[0047] Subsequently, the first device serializes the generated first event object, for example, by converting it into a JSON string, and sends it to the second device over the network. Upon receiving the serialized first event object, the second device deserializes it to reconstruct the original event information. Then, based on the DOM node identifier indicated in the first event object, the second device locates the actual "Submit" button node in its own DOM tree and simulates a click event. Thus, user B's click operation on the first device is precisely copied and executed on user A's second device, completing the form submission. Throughout this process, user B does not need to directly control user A's desktop; they only operate through a simulated web interface, and only necessary event information is transmitted, effectively protecting user A's privacy and data security.

[0048] Based on the above examples, the remote assistance method provided in this embodiment demonstrates significant technical contributions.

[0049] Compared to existing technologies that require the recipient to install independent client software, this embodiment achieves remote assistance without requiring the installation of intrusive client software on the second device by building a DOM mirror tree on the first device and updating it based on change events sent by the second device. This lowers the barrier to entry for users and improves compatibility.

[0050] Furthermore, existing technologies often expose the entire desktop environment of the recipient during remote assistance, posing privacy and security risks. The method in this embodiment, however, simulates a specific webpage interface of the second device on the first device only by transmitting DOM change events and interaction events, and precisely maps only the user's actions on the simulated interface to the corresponding DOM nodes on the second device. This means that during remote assistance, the second device only shares the DOM structure and events of a specific webpage, rather than the entire desktop screen, effectively avoiding privacy leaks and significantly improving the privacy protection and security of remote assistance.

[0051] This embodiment achieves accurate reproduction of user operations by capturing user interaction events triggered in the DOM mirror tree, generating a first event object, serializing it, and sending it to the second device. This event-driven synchronization mechanism ensures that operations on the first device can be executed accurately on the second device, guaranteeing the effectiveness and real-time nature of remote assistance. This method provides a lightweight, efficient, and secure remote assistance solution through refined event transmission and DOM synchronization.

[0052] In some embodiments, before receiving the change event sent by the second device, the method further includes: obtaining a DOM snapshot of the second device; and constructing a DOM mirror tree based on the DOM snapshot.

[0053] This application's solution ensures that the DOM mirror tree on the first device accurately and completely reflects the current DOM state of the second device at the start of the remote assistance session by first acquiring a DOM snapshot of the second device before receiving change events from the second device, and then constructing a DOM mirror tree based on this snapshot. Specifically, when the remote assistance session starts, the first device actively or passively receives a DOM snapshot sent by the second device. This snapshot contains all DOM elements of the second device's current page or application interface, along with their attributes and hierarchical relationships. Upon receiving this snapshot, the first device immediately parses the snapshot data and accurately replicates the second device's DOM structure in its local memory, forming an initial DOM mirror tree. Once this initial DOM mirror tree is established, the first device has a reliable baseline view. Based on this, the first device begins receiving subsequent real-time change events sent by the second device and incrementally updates the constructed DOM mirror tree based on these events. This mechanism of establishing a complete baseline before incremental updates avoids errors or inconsistencies that might result from directly processing change events without an initial state, ensuring the continuous accuracy of the DOM mirror tree and providing a solid foundation for subsequently capturing user interaction events and synchronizing them to the second device.

[0054] The following is a concrete example. After the first device establishes a remote assistance connection with the second device, the second device immediately executes a script. This script iterates through the `document` object of its current webpage, collecting the tag names, IDs, class names, attribute values, and text content of all HTML elements, and recording their parent-child relationships. This information is encapsulated into a JSON-formatted data packet, such as `{"tag": "html", "children": [{"tag": "head", ...}, {"tag": "body", "id": "app", "children": [...]}]}`, as a DOM snapshot. This JSON data packet is then transmitted to the first device over the network. Upon receiving this JSON-formatted DOM snapshot, the first device uses a front-end rendering library (e.g., a lightweight virtual DOM implementation) to parse the JSON data. The rendering library creates corresponding virtual DOM nodes based on the `tag` field in the JSON data, establishes parent-child relationships between nodes based on the `children` field, and assigns attributes such as `id` and `class` to the corresponding virtual DOM nodes. Through this process, the first device constructs a virtual DOM tree in its local memory that is completely identical to the current page of the second device, namely the DOM mirror tree.

[0055] Through the above technical solution, at the start of a remote assistance session, the first device can obtain a DOM snapshot of the second device and construct a DOM mirror tree based on this snapshot, thus ensuring the accurate initialization of the DOM mirror tree. This solves the problem that subsequent change events cannot effectively affect the DOM mirror tree without an initial state, avoiding errors in remote assistance operations caused by an incomplete or inaccurate DOM mirror tree. This solution provides a stable and reliable benchmark for subsequent DOM mirror tree updates based on change events, greatly improving the synchronization accuracy and user experience of remote assistance, enabling the first device to always have an accurate copy of the second device's interface, thereby achieving efficient and seamless remote guidance and operation.

[0056] In some embodiments, the remote assistance method further includes: receiving a video stream and a canvas binding message sent by a second device; determining a Canvas element node in the DOM mirror tree based on the canvas binding message, and drawing image data extracted from the video stream onto the Canvas element node; capturing interactive events triggered by the user in the Canvas element display area, and generating a second event object; serializing the second event object and sending it to the second device to trigger the serialized second event object on the Canvas element.

[0057] The video stream is image data generated by the Canvas element detected by the second device. In simpler terms, a video stream is a continuous sequence of image data generated by the Canvas element detected by the second device. This video stream can be transmitted using various encoding formats, such as H.264, VP8, or VP9, ​​to ensure transmission efficiency and compatibility. Its purpose is to transmit the dynamically rendered content within the Canvas element to the first device in real time, enabling the first device to acquire and display the visual state of the Canvas element on the second device.

[0058] Canvas binding messages are used to represent the mapping relationship between the video stream and DOM nodes in the DOM tree. In essence, canvas binding messages establish the association between the video stream and DOM nodes in the DOM tree. This message can be a JSON object containing a unique identifier for the Canvas element and its corresponding video stream identifier, or it can contain the path information of the Canvas element in the DOM tree, so that the primary device can accurately identify the Canvas element corresponding to the video stream.

[0059] The Canvas element's display area is a visual area drawn based on the video stream. When a user performs actions such as clicking, moving the mouse, or dragging within the Canvas element's display area on the first device, the first device can detect and record these interaction events. This can be achieved by registering JavaScript event listeners (such as `click`, `mousemove`, `mousedown`, `mouseup`, etc.) on the local Canvas element.

[0060] The second event object contains the event type, the identifier of the node corresponding to the Canvas element, and the event coordinate parameters.

[0061] This application's solution transmits the real-time rendered content of the Canvas element in the second device to the first device as a video stream, and establishes a mapping relationship between the video stream and the Canvas element nodes in the DOM mirror tree using canvas binding messages. This allows the first device to accurately render and display the dynamic content of the second device's Canvas element in its local DOM mirror tree. When the user on the first device interacts in the locally rendered Canvas element display area, the first device can accurately capture these interaction events and encapsulate them into a second event object containing event type, Canvas node identifier, and event coordinate parameters. Subsequently, this second event object is serialized and sent back to the second device, which then simulates triggering the event on the corresponding Canvas element. This mechanism effectively expands the scope of remote assistance, enabling remote operation and guidance of Canvas content that was previously difficult to interact with using traditional DOM mirroring methods. In this way, the solution not only maintains the ability to remotely assist with regular DOM elements but also further solves the synchronization problem of internal interactions within Canvas elements, thus providing a more comprehensive and efficient remote assistance solution.

[0062] The following is a concrete example. As a specific implementation, suppose a second device is running a Canvas-based online drawing application, and a user wants remote assistance. The first device, acting as the assistant, first receives a video stream from the second device regarding the Canvas element in the drawing application. For example, this video stream could be a live screen-sharing stream transmitted via the WebRTC protocol, containing only pixel data of the Canvas area. Simultaneously, the first device also receives a canvas binding message, which may be a JSON object containing the unique ID of the Canvas element (e.g., `canvas_drawing_area`) and the association information of the video stream. Based on this binding message, the first device locates the Canvas element node with the ID `canvas_drawing_area` in its locally maintained DOM mirror tree and renders the decoded image frames from the received video stream onto this local Canvas element node in real time. At this point, the user (assistant) on the first device can see the real-time content of the Canvas in the second device's drawing application on their screen. If the assistant wants to perform operations within the drawing area, such as clicking a tool button or drawing lines on the canvas, they can directly click or drag within the Canvas element display area rendered on the first device. The first device captures these interaction events, such as a mouse click, and generates a second event object. This second event object contains the event type (e.g., `click`), the node identifier corresponding to the Canvas element (`canvas_drawing_area`), and the precise coordinates of the click (e.g., `x: 150, y: 200`). This second event object is then serialized into a JSON string and sent to the second device via a WebSocket connection. Upon receiving the serialized event object, the second device deserializes it and simulates a click event at coordinates (150, 200) on its local `canvas_drawing_area` element. In this way, the drawing application on the second device responds to this simulated click, as if the user were interacting with it, thus enabling remote interaction with the Canvas content.

[0063] Through the above technical solution, this application effectively addresses the problem that traditional DOM mirroring technology cannot remotely assist with complex interactive content within Canvas elements. By transmitting a video stream of the Canvas element in real time and combining it with canvas binding messages, the first device can accurately render the dynamic content of the Canvas element on the second device and capture precise user interactions within these visual areas. This method extends remote assistance beyond conventional DOM element manipulation to control highly dynamic and interactive Canvas content, such as in online games, graphic editing, or data visualization applications. This significantly improves the comprehensiveness and user experience of remote assistance, enabling the assistant to provide more in-depth and intuitive guidance to the assisted user, thereby increasing the efficiency and success rate of remote assistance.

[0064] In some embodiments, the remote assistance method further includes: receiving target coordinate information sent by a second device; performing scrollability detection on the mirror nodes in the DOM mirror tree based on the target coordinate information to obtain the target scroll container mirror node and its corresponding scroll increment; and sending the identifier and scroll increment of the target scroll container mirror node to the second device to determine the target scroll container node in the DOM tree and modify its scroll offset attribute.

[0065] The target coordinate information sent by the second device refers to the coordinate data of the user's operation position captured by the second device. This coordinate information is intended to provide the first device with the precise location of the user's operation so that the first device can perform subsequent node positioning and detection in the DOM mirror tree based on this location. For example, when a user clicks or touches the screen on the second device, the second device can capture the `clientX / Y` or `pageX / Y` attributes of the click or touch event and send them to the first device as target coordinate information. As another implementation, the second device can also periodically send the current position coordinates of the mouse pointer, or send start and end coordinates when the user performs a specific operation (such as a scrolling gesture).

[0066] Scrollability testing refers to the test of identifying all container nodes in the DOM mirror tree that may respond to user scrolling. One implementation is to start from the mirror node corresponding to the target coordinates and traverse upwards through its ancestor mirror nodes, checking whether each node has CSS properties such as `overflow: scroll` and `overflow: auto`, and whether its content exceeds its visible area. Another implementation is that the first device can maintain a pre-identified list of scrollable mirror nodes and determine whether the user's operation falls within the area of ​​these nodes based on the target coordinates.

[0067] The solution in this application receives target coordinate information from a second device, enabling the first device to perform scrollability detection on mirror nodes based on the user's operation position in the DOM mirror tree. Through this detection, the first device can intelligently identify the target scroll container mirror node that the user intends to scroll and accurately calculate the corresponding scroll increment. Subsequently, the first device sends the identifier of the target scroll container mirror node and the scroll increment to the second device. Upon receiving this information, the second device can accurately locate the corresponding target scroll container node in its actual DOM tree and modify its scroll offset attribute according to the received scroll increment. This mechanism ensures that the scrolling operations performed by the user on the first device can be accurately parsed and synchronized to the second device, solving the problem of inaccurate scrolling synchronization in traditional remote assistance and greatly improving the accuracy of remote assistance and user experience.

[0068] The following example illustrates this. Suppose a user on a second device scrolls within a `div` element containing a large amount of text using their mouse wheel. The second device captures the mouse wheel event and sends the coordinates of its trigger position (e.g., `clientX=100, clientY=200`) to the first device. Upon receiving these coordinates, the first device locates the corresponding mirror node in its DOM mirror tree. Then, the first device traverses upwards from this mirror node, checking which nodes have the `overflow: auto` or `overflow:scroll` style and whose content overflows. Assume the `div` mirror node with the ID `content-area` is detected as the innermost scrollable container that meets these conditions. Simultaneously, the first device calculates a scroll increment of `50` pixels based on the `deltaY` value of the user's mouse wheel event. The first device serializes the `content-area` identifier and the `50`-pixel scroll increment and sends them to the second device. After receiving this information, the second device finds the corresponding `div` element in its DOM tree using `document.getElementById('content-area')` and increases its `scrollTop` property by `50`, thereby achieving precise scrolling of the `div` element on the second device.

[0069] Through the above technical solution, the first device can intelligently identify the target container mirror node that the user intends to scroll based on the user's operation coordinates in the DOM mirror tree, and accurately calculate the scroll increment. This solves the problem of inaccurate scroll operation synchronization in remote assistance and avoids scroll target identification errors or inconsistent scrolling behavior caused by simple event forwarding. By sending the identifier of the target scroll container and the scroll increment to the second device, the second device can accurately locate and modify the scroll offset attribute of the corresponding node in its actual DOM tree, thereby achieving accurate and smooth synchronization of scroll operations during remote assistance, significantly improving user experience and the efficiency of remote assistance.

[0070] In some embodiments, performing scrollability detection on mirror nodes in a DOM mirror tree based on target coordinate information includes: locating a target mirror node in the DOM mirror tree based on target coordinate information; performing scrollability detection on the target mirror node and its ancestor mirror nodes, and selecting a target scroll container mirror node from the mirror nodes that meet the scrollability detection conditions; and calculating the scroll increment of the target scroll container mirror node.

[0071] Locating the target mirror node in the DOM mirror tree based on target coordinate information means finding the most specific and deepest mirror node corresponding to the target coordinate position in the DOM mirror tree maintained by the first device, based on the target coordinate information sent by the second device. One implementation method is to use a depth-first or breadth-first traversal algorithm, combining the bounding box information of each mirror node, and recursively searching for the deepest node overlapping the target coordinates, starting from the root node of the DOM mirror tree. Another implementation method is to maintain a spatial index structure, such as a quadtree or R-tree, indexing all mirror nodes in the DOM mirror tree according to their position and size in the virtual rendering space. When the target coordinates are received, by querying this spatial index structure, the set of mirror nodes intersecting with the target coordinates can be quickly located, and the topmost or deepest node can be selected as the target mirror node.

[0072] Scrollability testing of a target mirror node and its ancestor mirror nodes refers to determining whether a DOM mirror node has scrollability, i.e., whether its content exceeds its visible area and whether its CSS styles allow scrolling. Ancestor mirror nodes refer to all parent mirror nodes traced upwards from the target mirror node to the root node. One implementation approach is to check the computed styles of the mirror node, particularly whether the `overflow-x` and `overflow-y` properties are set to `scroll`, `auto`, or `overlay`, and simultaneously determine whether its content height (`scrollHeight`) or width (`scrollWidth`) is greater than its visible area height (`clientHeight`) or width (`clientWidth`). Another implementation approach is to pre-calculate and mark the scrollability state of each mirror node during DOM mirror tree construction or updates. This state can be stored as node metadata and directly read during testing, thus improving testing efficiency.

[0073] Selecting the target scroll container mirror node from among the mirror nodes that meet the scrollability detection criteria means choosing the node that best matches the user's scrolling intention from all mirror nodes that meet the scrollability detection criteria, based on a specific strategy. One implementation approach is to use the "least ancestor" principle, that is, among all mirror nodes that meet the scrollability detection criteria, select the node that is closest to the target mirror node in the DOM tree hierarchy (i.e., the deepest level). Another implementation approach is to consider the context of user interaction; for example, if the clickable area indicated by the target coordinates is within the visual range of a scrollbar, then the mirror node belonging to that scrollbar is preferentially selected as the target scroll container mirror node.

[0074] Calculating the scroll increment of a target scrolling container mirror node refers to the distance that needs to be scrolled on the target scrolling container mirror node to bring the target coordinates or the target mirror node into the visible area. One implementation is to calculate the minimum vertical or horizontal scroll distance required to fully enter the visible area of ​​the target mirror node based on its current position relative to the visible area of ​​the target scrolling container mirror node. For example, if the bottom of the target mirror node extends beyond the visible area of ​​the container, the scroll increment is the difference between the bottom of the target mirror node and the bottom of the visible area of ​​the container. Another implementation is to incorporate a scrolling gesture triggered by the user on the first device (such as the direction and intensity of a mouse wheel event) and convert that gesture into a corresponding scroll increment for the target scrolling container mirror node. For example, scrolling the mouse wheel upwards by one notch calculates a preset upward scroll distance as the increment.

[0075] The solution in this application first accurately locates the 'target mirror node' operated on by the user in the DOM mirror tree by receiving the target coordinate information sent by the second device. Then, a comprehensive 'scrollability detection' is performed on the 'target mirror node' and all its 'ancestor mirror nodes' to identify all potential scrollable containers. Based on this, an intelligent selection strategy is used to select the 'target scrollable container mirror node' that best matches the user's intent from these nodes that meet the criteria, and the required 'scroll increment' is accurately calculated. This ensures that the user's scrolling intent based on coordinate input can be accurately understood, generating precise scrolling instructions even in complex nested DOM structures.

[0076] The following example illustrates this. Suppose a second device displays a webpage containing a main scroll area, within which is embedded a smaller `div` element, which also has its own scrollbar (e.g., a code editor or chat window). The first device receives target coordinate information from the second device, for example, coordinates (x=300, y=400). The first device first uses these coordinates to locate the `target mirror node` in its DOM mirror tree. If these coordinates fall inside the nested `div` element, then that `div` element is determined as the `target mirror node`. Next, the scrollability of the `div` element and all its ancestor nodes (including the parent element of the main scroll area) is checked. If both the nested `div` element and the main scroll area are scrollable, the nested `div` element is selected as the `target scroll container mirror node` according to the "nearest ancestor" principle. Subsequently, the first device calculates the `scroll increment` required to bring its content into the visible area based on the target coordinates' position within the nested `div` element. Finally, the identifier of the target scroll container mirror node and the calculated scroll increment are sent back to the second device, which performs the actual scrolling operation. If the target coordinates fall within the main scroll area but are not within a nested `div` element, the main scroll area will be selected as the target scroll container mirror node.

[0077] The above technical solution, based on target coordinate information, first precisely locates the 'target mirror node' of the user's operation in the DOM mirror tree. Then, a comprehensive 'scrollability detection' is performed on the 'target mirror node' and all its 'ancestor mirror nodes' to identify all potential scrollable containers. Based on this, an intelligent selection strategy is used to select the 'target scrollable container mirror node' that best matches the user's intent from these nodes that meet the criteria, and the required 'scroll increment' is accurately calculated. This effectively solves the problem of accurately identifying the target container that the user intends to scroll in complex webpage structures with multi-layered nested scrollable elements, avoiding inefficiency or misoperation caused by scrolling the wrong container. This solution significantly improves the accuracy of scrolling operations and user experience during remote assistance, enabling the first device to more intelligently and accurately simulate scrolling behavior on the second device.

[0078] In some embodiments, the remote assistance method further includes: receiving a tab update message sent by a second device; updating the list of available tabs displayed in the user interface based on the tab update message; capturing the user's selection operation in the list of available tabs; sending an identifier of the target tab to the second device; and causing the second device to activate the target tab as the currently active tab.

[0079] Updating the list of available tabs displayed in the user interface based on tab update messages means that the first device dynamically modifies the tab information displayed on its own user interface according to the received tab update messages. This allows remote assistance personnel to clearly see all currently available tabs and their status on the second device. This update process may include: the first device maintaining an internal data structure to store tab information, updating this data structure when an update message is received, and triggering the user interface rendering mechanism to redraw the tab list; or modifying the HTML elements displaying the tabs by directly manipulating the Document Object Model (DOM).

[0080] Capturing user selections within a list of available tabs refers to the user interface of a first device listening for user interactions with the displayed tab list, such as clicks, touches, or keyboard selections. The purpose of this is to identify which tab the user wants to switch to. This capture can be achieved using event listeners, such as detecting user clicks on list items using JavaScript's `click` event listener, or detecting user selections using the arrow keys and Enter key using a keyboard event listener.

[0081] Sending the target tab identifier to the second device means that after the first device captures the user's selection, it sends a unique identifier (such as a tab ID or index) of the user-selected tab to the second device over the network. This identifier tells the second device which tab should be activated. This sending process can be achieved by sending the identifier as a parameter via an HTTP request, or by sending a JSON message containing the identifier via a WebSocket connection.

[0082] This application's solution, based on remote assistance methods, further enhances the management capabilities of the multi-tab environment of the second device. Specifically, the first device first receives a tab update message sent by the second device, which contains real-time status information of all currently available tabs on the second device. Based on this tab update message, the first device can dynamically update and display the list of available tabs on its user interface, thereby providing remote assistance personnel with an intuitive and comprehensive multi-tab view. When a remote assistance personnel selects from the list of available tabs on the first device's user interface, the first device captures this selection operation and identifies the identifier of the target tab the user wishes to activate. Subsequently, the first device sends the identifier of the target tab to the second device. Upon receiving the identifier, the second device activates the corresponding target tab as the currently active tab according to its internal logic. Through this mechanism, remote assistance personnel can not only operate on the currently active tab of the second device through the DOM mirror tree, but also easily switch and manage multiple tabs, greatly improving the flexibility and efficiency of remote assistance. This multi-tab management capability, combined with basic DOM mirroring and event forwarding mechanisms, enables remote assistance to cover more complex application scenarios and solves the problem that remote assistance personnel cannot effectively switch and control when facing a multi-tab environment.

[0083] In one specific implementation, when a remote assistant provides remote assistance to a second device via a first device, the second device can be a computer or mobile device running a web browser. Browser plugins or embedded scripts on the second device continuously monitor the status of its open tabs, such as tab creation, closing, title changes, or activation state switching. Once any tab status change is detected, the second device generates a tab update message, which can be a JSON object containing an array of tab objects, each containing fields such as `id`, `title`, `url`, and `active`. This message is sent to the first device in real-time via a WebSocket connection. Upon receiving the message, the first device's front-end application parses the JSON data and dynamically updates a sidebar or top navigation bar on its user interface specifically for displaying tabs. This navigation bar displays the titles of all tabs in a list format and highlights the currently active tab. When the remote assistant clicks on a tab title in the navigation bar, the first device captures this click event, extracts the corresponding tab's `id`, and sends the `id` back to the second device via a WebSocket connection. After receiving the `id`, the second device will call its browser API (for example, `chrome.tabs.update(tabId, { active: true})` in the Chrome browser) to activate the specified tab and make it the currently visible tab.

[0084] Through the above technical solution, the first device can acquire and display the multi-tab status of the second device in real time, allowing remote assistants to clearly understand all available working contexts on the second device. When remote assistants need to switch operation targets, they can directly select the target tab on the user interface of the first device. The first device then converts this selection into an instruction and sends it to the second device, thereby activating the corresponding tab. This effectively solves the problem that remote assistants find it difficult to intuitively manage and switch operation targets when multiple tabs exist on the second device during remote assistance. This solution significantly improves the convenience and efficiency of remote assistance, enabling remote assistants to control the second device more flexibly and comprehensively, and is particularly suitable for complex remote assistance scenarios that require frequent switching between different applications or web interfaces.

[0085] In some embodiments, the remote assistance method further includes: responding to a guidance request initiated by a second device, establishing a video call with the second device, and / or sharing the application interface of the first device with the second device, and displaying interactive events triggered by the user in the application interface of the first device.

[0086] When a user on the second device encounters complex problems requiring more direct guidance or demonstration during remote assistance, the second device can send a guidance request to the first device. Upon receiving this request, the first device will activate advanced assistance mode. In this mode, the first device can establish a real-time video call with the second device, enabling audio and video communication between the two parties. This allows the assisting party to understand the user's problem more intuitively or provide immediate guidance through voice and visuals. Furthermore, the first device can selectively share its application interface with the second device. While sharing the application interface, any interactive events triggered by the user (i.e., the assisting party) on the shared interface, such as mouse clicks and keyboard inputs, will be captured and displayed visually on the second device. This mechanism allows the assisting party to directly demonstrate the operation steps to the user, who can clearly see every action of the assisting party, thus greatly improving the efficiency and accuracy of guidance. This proactive guidance and demonstration capability, combined with basic DOM mirroring interaction capabilities, forms a more comprehensive and efficient remote assistance solution, effectively compensating for the shortcomings of relying solely on DOM event interaction in complex scenarios.

[0087] In one specific implementation, when the user of the second device clicks a "Request Video Guidance" button in the remote assistance interface, this action triggers a guidance request message, which is sent to the first device via a WebSocket connection. Upon receiving this message, the first device initiates a WebRTC session, establishing a peer-to-peer video call with the second device. At this time, the user of the first device (the assisting party) can communicate with the user of the second device in real time via camera and microphone. Simultaneously, the assisting party can choose to share a specific tab or their entire desktop from their local browser. For example, the assisting party can share a browser tab containing tutorials for specific software operations. When the assisting party clicks or scrolls on this shared tab, these interaction events are captured by the first device and sent to the second device via a WebRTC data channel or a separate signaling channel. Upon receiving this event data, the second device will indicate the location of the assisting party's click with a brief red circle or arrow on the area displaying the shared interface, or highlight the scrolling area with a highlighted border, thus clearly showing the user the assisting party's operation trajectory and intent.

[0088] Through the aforementioned technical solution, when the user of the second device requires more in-depth guidance, the first device can respond to their guidance request and establish a video call, thereby enabling real-time audio and video communication between the assisting party and the user, greatly improving communication efficiency and the accuracy of problem understanding. Furthermore, by sharing the application interface of the first device and displaying the assisting party's interactive events, the assisting party can directly demonstrate complex operating procedures or display relevant information to the user, allowing the user to learn and understand intuitively, effectively avoiding misunderstandings that may arise from relying solely on text or verbal descriptions. This proactive, multimodal guidance approach, combined with basic remote DOM interaction capabilities, makes remote assistance services more comprehensive and efficient, significantly improving user experience and the success rate of problem resolution.

[0089] See Figure 3 This application also provides a remote assistance device that can implement the above-described remote assistance method. The device includes: The first module 301 is used to receive change events sent by the second device; The second module 302 is used to update the current DOM mirror tree based on change events to obtain the updated DOM mirror tree; the DOM mirror tree is a mirror image of the DOM tree of the second device; The third module 303 is used to capture user interaction events triggered in the updated DOM mirror tree and generate a first event object; the first event object contains the event type, the identifier of the DOM node it is applied to, and the event parameters; The fourth module 304 is used to serialize the first event object and send it to the second device so as to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

[0090] The specific implementation of the remote assistance device is basically the same as the specific implementation of the remote assistance method described above, and will not be repeated here.

[0091] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0092] The following reference Figure 4 To describe an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0093] like Figure 4As shown, the electronic device 400 is presented in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), a display unit 440, etc.

[0094] The storage unit stores program code, which can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the above-described remote assistance method section of this specification according to various exemplary embodiments of this disclosure.

[0095] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 4201 and / or cache memory 4202, and may further include a read-only memory (ROM) 4203.

[0096] Storage unit 420 may also include a program / utility 4204 having a set (at least one) program module 4205, such program module 4205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0097] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0098] Electronic device 400 can also communicate with one or more external devices 400' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. Network adapter 460 can communicate with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0099] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0100] The remote assistance method, apparatus, electronic device, and storage medium provided in this application, by constructing a DOM mirror tree on the assisting party's electronic device and updating it based on change events sent by the assisted party, enables remote assistance without requiring the installation of intrusive client software on the assisted party's electronic device. This lowers the barrier to entry for users and improves system compatibility. By transmitting DOM change events and interaction events, a specific webpage interface of the assisted party's electronic device is simulated on the assisting party's electronic device, and only the user's operations on the simulated interface are accurately mapped to the corresponding DOM nodes on the second device. This effectively avoids privacy leaks for the assisted party and significantly improves the privacy protection and security of remote assistance.

[0101] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions that cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0102] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0105] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A remote assistance method, executed by a first device, characterized in that, The method includes: Receive change events sent by the second device; Based on the change event, the current DOM mirror tree is updated to obtain the updated DOM mirror tree; the DOM mirror tree is a mirror image of the DOM tree of the second device; Capture user interaction events triggered in the updated DOM mirror tree and generate a first event object; the first event object contains the event type, the identifier of the DOM node it affects, and event parameters; The first event object is serialized and sent to the second device to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

2. The remote assistance method according to claim 1, characterized in that, The process also includes the following steps before receiving the change event sent by the second device: Get a DOM snapshot of the second device; Based on the DOM snapshot, construct the DOM mirror tree.

3. The remote assistance method according to claim 1, characterized in that, The method further includes: The device receives a video stream and a canvas binding message sent by the second device; the video stream is image data generated by the Canvas element detected by the second device, and the canvas binding message is used to characterize the mapping relationship between the video stream and the DOM nodes in the DOM tree. Based on the canvas binding message, the Canvas element node in the DOM mirror tree is determined, and the image data extracted from the video stream is drawn onto the Canvas element node; Capture user-triggered interaction events in the Canvas element display area and generate a second event object; the Canvas element display area is a visualization area drawn based on the video stream, and the second event object contains the event type, the identifier of the node corresponding to the Canvas element, and event coordinate parameters. The second event object is serialized and sent to the second device to trigger the serialized second event object on the Canvas element.

4. The remote assistance method according to claim 1, characterized in that, The method further includes: Receive the target coordinate information sent by the second device; Based on the target coordinate information, the scrollability of the mirror nodes in the DOM mirror tree is detected to obtain the target scroll container mirror node and its corresponding scroll increment. The identifier of the target scroll container mirror node and the scroll increment are sent to the second device to determine the target scroll container node in the DOM tree and modify its scroll offset attribute.

5. The remote assistance method according to claim 4, characterized in that, The scrollability detection of mirror nodes in the DOM mirror tree based on the target coordinate information includes: Based on the target coordinate information, locate the target mirror node in the DOM mirror tree; Perform scrollability detection on the target mirror node and its ancestor mirror nodes, and select the target scrollable container mirror node from the mirror nodes that meet the scrollability detection conditions; Calculate the scroll increment of the target scroll container mirror node.

6. The remote assistance method according to claim 1, characterized in that, The method further includes: Receive the tab update message sent by the second device; Based on the tab update message, update the list of available tabs displayed in the user interface; The system captures the user's selection action in the list of available tabs, sends the identifier of the target tab to the second device, and causes the second device to activate the target tab as the currently active tab.

7. The remote assistance method according to claim 1, characterized in that, The method further includes: In response to a guidance request initiated by the second device, establish a video call with the second device, and / or share the application interface of the first device with the second device, and display interactive events triggered by the user in the application interface of the first device.

8. A remote assistance device, characterized in that, include: The first module is used to receive change events sent by the second device; The second module is used to update the current DOM mirror tree based on the change event, so as to obtain the updated DOM mirror tree; The DOM mirror tree is a mirror image of the DOM tree of the second device; The third module is used to capture user interaction events triggered in the updated DOM mirror tree and generate the first event object; The first event object contains the event type, the identifier of the DOM node it affects, and event parameters; The fourth module is used to serialize the first event object and send it to the second device so as to trigger the serialized first event object on the corresponding DOM node in the DOM tree.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the remote assistance method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the remote assistance method according to any one of claims 1 to 7.

Images