Method, device and equipment for synchronizing mouse operation between browser instances and medium

Abstract

The invention provides a method, a device, equipment and a medium for synchronizing mouse operation between browser instances, and the method comprises the following steps: monitoring a window system mouse event, judging whether a mouse is in a master control browser or not, and if so, entering the next step; if not, ending the step; according to the current coordinate of the mouse, judging that the current coordinate is located in a native area, a webpage area or a third-party plug-in area of the master control browser; selecting a corresponding selector according to the corresponding area; obtaining element coordinates in the slave control browser according to the selector; and sending the mouse event to the slave control browser, and executing the mouse event according to the obtained coordinates of the coordinate elements, thereby accurately realizing synchronization between browser instances and improving user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for synchronizing mouse operations between browser instances. Background Art

[0002] Currently, in many fields, many customers have the need to manage website accounts in batches. Since a single browser instance can only log in to one account at a time for the same site, customers usually open multiple browser instances to manage multiple accounts. In order to improve the efficiency of user multi-account management, synchronizers were born. There are currently two mainstream synchronizer solutions on the market, each of which has some technical defects and limitations:

[0003] The first solution: synchronization based on absolute coordinates, that is, after the synchronizer obtains the coordinates of the mouse from the upper left corner of the master browser, it directly synchronizes the coordinates to the slave browser. This technical solution has the following disadvantages and limitations:

[0004] 1. The master browser and the slave browser are required to be completely the same size, which cannot adapt well to business scenarios where the master browser and the slave browser are of different sizes;

[0005] 2. The master browser and the slave browser must maintain a completely consistent page every time the user synchronizes an event, otherwise the wrong element will be synchronized. This usually requires the user to manually observe the slave browser and make manual corrections, thereby reducing the user's synchronization success rate and efficiency, which violates the original intention of the synchronizer to improve efficiency.

[0006] 3. It is not compatible with browsers with different kernels. Different kernel browsers have slight differences in interface, which causes events to be synchronized to the wrong elements.

[0007] 4. When the element is not in the visible area of ​​the browser, synchronization cannot be performed.

[0008] The second solution: Using pure Web technology to achieve synchronization based on element position. This solution has the following limitations:

[0009] 1. Unable to synchronize the operations of the browser native area and third-party plug-ins;

[0010] 2. It is necessary to inject code into the web page to simulate the event execution, which can be easily identified as machine behavior by the site.

[0011] Based on this, there is an urgent need to provide a synchronizer method based on element positioning to solve the above problems. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for synchronizing mouse operations between browser instances, accurately realize synchronization between browser instances and improve user experience.

[0013] In a first aspect, the present invention provides a method for synchronizing mouse operations between browser instances, comprising the following steps:

[0014] Step 1: Monitor the mouse events of the window system and determine whether the mouse is in the master browser. If so, proceed to step 2; if not, end the step;

[0015] Step 2: According to the current coordinates of the mouse, determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser; select a corresponding selector according to the corresponding area;

[0016] Step 3, obtaining the element coordinates in the slave control browser according to the selector;

[0017] Step 4: Send the mouse event to the slave browser, and execute the mouse event according to the acquired coordinate element coordinates.

[0018] In a second aspect, the present invention provides a device for synchronizing mouse operations between browser instances, comprising the following modules:

[0019] The monitoring and judging module monitors the mouse events of the window system and judges whether the mouse is in the main browser. If so, it enters the selector acquisition module; if not, it ends;

[0020] Get the selector module, and determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser according to the current coordinates of the mouse; select the corresponding selector according to the corresponding area;

[0021] A coordinate acquisition module, which acquires the coordinates of elements in the slave browser according to the selector;

[0022] The synchronization module sends the mouse event to the slave browser and executes the mouse event according to the acquired coordinate element coordinates.

[0023] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0025] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0026] The method, apparatus, device and medium provided in the embodiments of the present application support business scenarios where the master browser and the slave browser are of the same size; solve the problem of synchronization failure when there are differences in pages between the master and slave browsers; fully implement the synchronization operation of all areas of the browser based on the element position, including the web page area of ​​the browser; the native area of ​​the browser; and the third-party plug-in of the browser; the native area of ​​the browser includes: the title bar (maximize, minimize, close buttons, etc.) above the browser, the toolbar, the URL address bar, the bookmark bar, and the menu bar;

[0027] Using system-level APIs for event simulation eliminates the risk of being identified as a robot; it is well compatible with different versions of browser kernels; it solves the problem of elements not being able to be synchronized if they are not in the visible area; and it brings better performance by combining browser plug-ins and UI Automation technology.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0030] Figure 1 It is a schematic diagram of the framework of the system of the present invention;

[0031] Figure 2 is a flow chart of the method in Embodiment 1 of the present invention;

[0032] Figure 3 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0033] The embodiments of the present application provide a method, apparatus, device and medium for synchronizing mouse operations between browser instances, support business scenarios in which the master browser and the slave browser are of the same size; solve the synchronization failure problem when there are differences in pages between the master and slave browsers; and fully implement synchronization operations based on element positions in all areas of the browser, including the web page area of ​​the browser; the native area of ​​the browser; and the third-party plug-ins of the browser.

[0034] The technical solution in the embodiment of the present application has the following general idea:

[0035] like Figure 1As shown, before introducing the specific embodiments, the system framework corresponding to the method of the embodiment of the present application is first introduced:

[0036]

Main modules

[0037] 1. Synchronizer plug-in: It is only used to obtain the web element selector and web element coordinates. Since the website can identify the events simulated by the plug-in through some technical means, the plug-in does not perform any operations on the web page to prevent it from being identified as machine behavior by the website; the plug-in can be implemented based on Chrome Extensions technology.

[0038] 2.NativeBridge: It is a Windows application. The synchronizer plug-in needs to communicate with CoreEngine and return the web element selector and web element coordinates; therefore, NativeBridge needs to be introduced to complete the communication between the synchronizer plug-in and CoreEngine. NativeBridge and CoreEngine communicate through Pipe.

[0039] 3.CoreEngine: It is a Window application used for master browser event monitoring, element positioning, event queue management, and slave browser event synchronization management.

[0040]

Main Process

[0041] Mouse event monitoring: By monitoring the Window system mouse events, determine whether the mouse event coordinates are within the master browser. If so, continue with the subsequent process; the Win32API provided by the Window system performs event monitoring.

[0042] Get element selector: According to the position of the mouse, determine whether the current coordinates are in the browser native area, web page area or third-party plug-in area. Provide different selector acquisition solutions for different areas (the element structure of the web page part is usually more complex, and the performance of obtaining the web page area selector through UI Automation is low, so in the web page part, the synchronizer plug-in solution is still used to obtain the Web element selector). The bottom layer of the Window system provides a Win32 API interface, through which the position and size of the window are obtained. By comparing with the mouse position, it can be determined whether the mouse is in these areas. It includes the following steps:

[0043] 1. If the mouse coordinates are in the browser native area or third-party plug-in area, obtain the Window element selector where the mouse is currently located through the UI Automation solution provided by Windows; the third-party plug-in area refers to the synchronizer plug-in that the user downloaded and installed through the Chrome App Store or manually. Third-party plug-ins specifically refer to all other plug-ins except the synchronizer plug-in. When using a third-party plug-in, a new browser window may be opened.

[0044] 2. If the mouse coordinates are in the browser web page area:

[0045] 2.1.CoreEngine sends the relative coordinates in the web page to the synchronizer plug-in through NativeBridge;

[0046] 2.2. The synchronizer plug-in searches for the target web element based on the coordinates;

[0047] 2.3. The synchronizer plug-in generates the Web element selector, based on the Web's own CSS selector technology, to generate the shortest selector that can uniquely locate the element; the Web element selector rule: from the location of the Web element, recursively process its ancestor node, combine the ID, TagName, and Class information of the ancestor node, generate a CSS selector, until the current Web element can be uniquely located through the CSS selector, then stop recursing to the ancestor node;

[0048] 2.4. The synchronizer plug-in returns the web element selector to CoreEngine through NativeBridge;

[0049] Add the event to the queue of pending synchronization events: Generate an event to be synchronized and add the event to the queue to be executed. This event contains the element selector, event type, event unique identifier, coordinates of the event relative to the upper left corner of the window, coordinates of the event relative to the upper left corner of the screen, and coordinates of the event relative to the upper left corner of the element;

[0050] Pull the event to be synchronized from the event queue: Get an event to be synchronized from the queue, create synchronization tasks for all slave browsers (one synchronization task for each slave browser), and submit them to the thread pool for parallel execution.

[0051] To execute the synchronization task of the slave browser, follow the steps below:

[0052] 1. Get the coordinates of the slave browser element;

[0053] 1.1 If it is a Window element selector, call the UI Automation related interface to obtain the coordinate position of the element from the slave browser;

[0054] 1.2 If it is a Web element selector, CoreEngine sends a message to the synchronizer plug-in of the slave browser through NativeBridge. The synchronizer plug-in finds the target element coordinates according to the Web element selector, and returns the element coordinates to CoreEngine through NativeBridge;

[0055] 2. Get the coordinate position of the slave browser that actually needs to be triggered: add the coordinates of the slave browser element to the coordinates of the upper left corner of the element when the mouse event is triggered;

[0056] 3. If the Web element is in the invisible area, CoreEngine calls the UI Automation interface to scroll the page so that the element of the slave browser is displayed in the visible area;

[0057] 4. Send mouse events to the slave browser, and call the Windows system API to simulate system mouse events based on the acquired slave browser element coordinates.

[0058] Multiple browser instances are different from multiple browser tabs or multiple browser windows. Generally, a network platform or web application only allows one account to be logged in at the same time. This is because a single browser instance can only store a single copy of the browser environment data. Multiple tabs and multiple browser windows belong to a single browser instance. This is reflected in the following: when a new tab or new window is opened to log in to a new account, the logged-in account will automatically log out after the original tab or window is refreshed. The appearance of multiple browser instances is the same as that of multiple browser windows. Each instance corresponds to a window. In fact, the process ID (PID) corresponding to each instance (or window) is different, while the process ID (PID) of multiple browser windows of a single instance is the same. In order to build multiple environments, the most basic operation is to set the parameters of the user data path when starting the browser. For example, the parameter of Chrome is user_data_dir. In different browser instances of the same browser, log in to different accounts of the same website. Taking Chrome as an example, you can log in to multiple Google accounts through Chrome. Each time you log in to a Google account, a new browser instance will be opened. At this time, the history, bookmarks, cache, cookies and other data of these multiple Chrome browser instances are isolated by Google account. Assuming that GoogleA account corresponds to instance A, and GoogleB account corresponds to instance B, you can open the same URL https: / / www.baidu.com in instances A and B at the same time, and log in to Baidu account A and Baidu account B respectively;

[0059] Elements include: Window elements and Web elements; Window elements: UI elements that are not Web elements, collectively referred to as Window elements;

[0060] Selector: refers to a path to find a specific element. Usually the selector for the same element is the same between different browser instances; this selector is a prior art that allows developers to select elements in a web page or window according to set rules.

[0061] Embodiment 1

[0062] like Figure 2 As shown, this embodiment provides a method for synchronizing mouse operations between browser instances, including the following steps:

[0063] Step 1: Monitor the mouse events of the window system and determine whether the mouse is in the master browser. If so, proceed to step 2; if not, end the step;

[0064] Step 2: According to the current coordinates of the mouse, determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser;

[0065] If the current coordinates are in the browser native area or third-party plug-in area, get the Window element selector where the current coordinates are located through UIAutomation provided by Windows;

[0066] If the coordinates are in the browser web page area, CoreEngine sends the relative coordinates in the web page through NativeBridge; searches for the target Web element according to the relative coordinates, then generates the Web element selector, and returns the Web element selector to CoreEngine through NativeBridge;

[0067] Step 3: If the selector is a Window element selector, call the corresponding interface of UI Automation to obtain the corresponding element coordinates in the slave browser;

[0068] If the selector is a Web element selector, CoreEngine sends a message to the slave browser through NativeBridge, finds the corresponding element coordinates according to the Web element selector, and then returns the element coordinates to CoreEngine through NativeBridge;

[0069] Step 4: Send the mouse event to the slave browser. If the Web element is in the invisible area, CoreEngine calls the corresponding UI Automation interface to scroll the page so that the slave browser can be displayed in the visible area. According to the obtained element coordinates, call the Windows system API to simulate the mouse event.

[0070] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0071] Embodiment 2

[0072] like Figure 3 As shown, in this embodiment, a device for synchronizing mouse operations between browser instances is provided, including the following modules:

[0073] The monitoring and judging module monitors the mouse events of the window system and judges whether the mouse is in the main browser. If so, it enters the selector acquisition module; if not, it ends;

[0074] Get the selector module, and determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser according to the current coordinates of the mouse;

[0075] If the current coordinates are in the browser native area or third-party plug-in area, get the Window element selector where the current coordinates are located through UIAutomation provided by Windows;

[0076] If the coordinates are in the browser web page area, CoreEngine sends the relative coordinates in the web page through NativeBridge; searches for the target Web element according to the relative coordinates, then generates the Web element selector, and returns the Web element selector to CoreEngine through NativeBridge;

[0077] In the coordinate acquisition module, if the selector is the Window element selector, the corresponding interface of UI Automation is called to obtain the corresponding element coordinates in the slave browser;

[0078] If the selector is a Web element selector, CoreEngine sends a message to the slave browser through NativeBridge, finds the corresponding element coordinates according to the Web element selector, and then returns the element coordinates to CoreEngine through NativeBridge;

[0079] The synchronization module sends the mouse event to the slave browser. If the Web element is in the invisible area, CoreEngine calls the corresponding interface of UI Automation to scroll the page so that the slave browser can display it in the visible area; according to the obtained element coordinates, the Windows system API is called to simulate the mouse event.

[0080] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0081] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0082] Embodiment 3

[0083] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0084] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0085] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.

[0086] Embodiment 4

[0087] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0092] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for synchronizing mouse operations between browser instances, It is characterized in that The steps include: Step 1: Monitor the mouse events of the window system and determine whether the mouse is in the master browser. If so, proceed to step 2; if not, end the step; Step 2: According to the current coordinates of the mouse, determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser; Select the corresponding selector according to the corresponding area; Step 3, obtaining the element coordinates in the slave control browser according to the selector; Step 4: Send the mouse event to the slave browser, and execute the mouse event according to the acquired coordinate element coordinates.

2. A method for synchronizing mouse operations between browser instances according to claim 1, It is characterized in that The step 2 specifically includes: judging, according to the current coordinates of the mouse, whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser; If the current coordinates are in the browser native area or third-party plug-in area, get the Window element selector where the current coordinates are located through UIAutomation provided by Windows; If the coordinates are in the browser web page area, CoreEngine sends the relative coordinates in the web page through NativeBridge; searches for the target Web element according to the relative coordinates, then generates the Web element selector, and returns the Web element selector to CoreEngine through NativeBridge.

3. A method for synchronizing mouse operations between browser instances according to claim 2, It is characterized in that The step 3 is specifically as follows: If the selector is a Window element selector, call the corresponding interface of UI Automation to obtain the corresponding element coordinates in the slave browser; If the selector is a Web element selector, CoreEngine sends a message to the slave browser through NativeBridge, searches for the corresponding element coordinates based on the Web element selector, and then returns the element coordinates to CoreEngine through NativeBridge.

4. A method for synchronizing mouse operations between browser instances according to claim 1, It is characterized in that The step 4 is specifically as follows: The mouse event is sent to the slave browser. If the Web element is in the invisible area, CoreEngine calls the corresponding UIAutomation interface to scroll the page so that the slave browser can be displayed in the visible area. According to the obtained element coordinates, the Windows system API is called to simulate the mouse event.

5. A device for synchronizing mouse operations between browser instances, It is characterized in that Includes the following modules: The monitoring and judging module monitors the mouse events of the window system and judges whether the mouse is in the main browser. If so, it enters the selector acquisition module; if not, it ends; Get the selector module, and determine whether the current coordinates are located in the native area, web page area or third-party plug-in area of ​​the master browser according to the current coordinates of the mouse; Select the corresponding selector according to the corresponding area; A coordinate acquisition module, which acquires the coordinates of elements in the slave browser according to the selector; The synchronization module sends the mouse event to the slave browser and executes the mouse event according to the acquired coordinate element coordinates.

6. The device for synchronizing mouse operations between browser instances according to claim 5, It is characterized in that The obtaining selector module specifically comprises: judging, according to the current coordinates of the mouse, whether the current coordinates are located in the native area, the webpage area or the third-party plug-in area of ​​the master browser; If the current coordinates are in the browser native area or third-party plug-in area, get the Window element selector where the current coordinates are located through UIAutomation provided by Windows; If the coordinates are in the browser web page area, CoreEngine sends the relative coordinates in the web page through NativeBridge; searches for the target Web element according to the relative coordinates, then generates the Web element selector, and returns the Web element selector to CoreEngine through NativeBridge.

7. The device for synchronizing mouse operations between browser instances according to claim 6, It is characterized in that The coordinate acquisition module is specifically: If the selector is a Window element selector, call the corresponding interface of UI Automation to obtain the corresponding element coordinates in the slave browser; If the selector is a Web element selector, CoreEngine sends a message to the slave browser through NativeBridge, searches for the corresponding element coordinates based on the Web element selector, and then returns the element coordinates to CoreEngine through NativeBridge.

8. The device for synchronizing mouse operations between browser instances according to claim 5, It is characterized in that The synchronization module is specifically: The mouse event is sent to the slave browser. If the Web element is in the invisible area, CoreEngine calls the corresponding UIAutomation interface to scroll the page so that the slave browser can be displayed in the visible area. According to the obtained element coordinates, the Windows system API is called to simulate the mouse event.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images