A multi-client browser automated interaction method and system

Abstract

The application discloses a kind of multi-client browser automation interaction method and system, it is related to browser automation technical field, including: establishing the browser context that multiple clients share, receive browser task and form instruction queue;Execute instruction queue, capture network request response, obtain business data and save as business variable, use business variable in the instruction of needing business data;Pause instruction needing manual operation, generate breakpoint pointer pointing to next unexecuted step, save instruction queue, business variable and breakpoint pointer as task checkpoint and then transfer into manual operation;After manual operation, according to breakpoint pointer determine next unexecuted step, restore business variable from task checkpoint, and execute remaining instructions until browser task is completed.The application keeps the continuous use of business data before and after manual intervention, realizes the continuous execution of browser task from next unexecuted step, reduces the risk of parameter transmission interruption, instruction repetition and omission.

Description

Technical Field

[0001] This invention relates to the field of browser automation technology, and in particular to a method and system for automated interaction of multiple client browsers. Background Technology

[0002] With the development of enterprise-level web applications, intelligent agents, and process automation, browser-based automated interaction is increasingly being used in business scenarios such as project management, operations and maintenance, data entry, and online approval. Continuous business processes typically involve multiple browser operations, with potential data transfer relationships between these operations. During task execution, manual intervention may be required due to graphic verification codes, slider verification, or risk control prompts. Therefore, maintaining business data and resuming tasks after manual intervention have become crucial technical aspects of continuous execution.

[0003] However, existing methods still have two limitations: First, manual intervention will interrupt the continuous use of business data. If the business data that has been formed cannot be maintained during the pause and recovery process, it will be difficult for instructions that require business data to obtain the corresponding data. Second, manual operation will interrupt automatic execution. If the task execution position and business variables cannot be retained and restored together, problems such as repeated execution, missed execution, or parameters that cannot be used will easily occur when resuming execution. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multi-client browser automated interaction method to solve the problem of difficulty in continuously restoring business data and task execution location after manual intervention.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a multi-client browser automated interaction method, comprising: establishing a browser context shared by multiple clients; receiving browser tasks; assembling instructions into an instruction queue according to execution order; executing the instruction queue in the browser context; executing instructions sequentially and capturing network request responses; obtaining business data from the network request responses; saving the business data as business variables; and using the business variables in instructions that require business data; pausing the execution of instructions requiring manual operation; generating a breakpoint pointer; the breakpoint pointer pointing to the next unexecuted step in the instruction queue; saving the instruction queue, business variables, and breakpoint pointer as a task checkpoint; and transferring to manual operation after the task checkpoint is saved; resuming the execution of instructions after the manual operation is completed; determining the next unexecuted step based on the breakpoint pointer; restoring the business variables from the task checkpoint; and executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

[0008] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the steps of receiving browser tasks and forming an instruction queue according to the execution order are as follows: receiving browser tasks initiated by clients and receiving instructions sent by clients for browser tasks; recording the corresponding instructions in sequence according to the execution order of browser operations in browser tasks; forming an instruction queue and maintaining the execution order of instructions.

[0009] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the steps of executing the instruction queue in the browser context, sequentially executing instructions and capturing network request responses are as follows: reading the instruction queue of the browser task, obtaining unexecuted instructions according to the execution order; obtaining the instruction that needs to obtain business data, and reading the network address pattern, which is specified by the client for the instruction that needs to obtain business data; before executing the instruction that needs to obtain business data, calling the network request monitoring interface to start network request capture, and passing the network address pattern to the network request monitoring interface; reading the instruction command and parameters, and sending the instruction command and parameters to the browser context; the browser running engine calling the browser operation and executing the browser operation using the parameters; after the browser operation generates a network request, calling the network request waiting interface, and passing the network address pattern to the network request waiting interface; the network request waiting interface detecting network requests that match the network address pattern and obtaining the network request response; after one instruction is executed, obtaining the next unexecuted instruction according to the execution order.

[0010] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the steps of obtaining business data from the network request response, saving the business data as business variables, and using the business variables in instructions that require business data are as follows: reading the response body of the network request response and extracting business data from the response body; saving the business data as business variables; obtaining instructions according to the execution order in the instruction queue, reading the parameters in the instructions one by one and identifying variable references; finding the business variables according to the variable references and obtaining the business data; and replacing the variable references in the parameters with the business data.

[0011] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the specific steps of pausing the execution of instructions requiring manual operation are as follows: reading the page document object model to obtain the class attribute of page elements, the resource address of image elements, and page text; checking the class attribute and the resource address of image elements, and matching the risk control prompt text in the page text; recognizing graphic verification codes, slider verifications, or risk control prompt text; pausing the currently executing instructions and stopping the retrieval of new instructions from the instruction queue.

[0012] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the step of generating a breakpoint pointer, which points to the next unexecuted step in the instruction queue, specifically involves: reading the position of the currently executing instruction in the instruction queue, reading instructions sequentially from the next position to obtain the first unexecuted instruction; reading the execution position of the first unexecuted instruction in the instruction queue and creating a breakpoint pointer; and writing the execution position of the first unexecuted instruction in the instruction queue into the breakpoint pointer.

[0013] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the steps of saving the instruction queue, business variables, and breakpoint pointers as task checkpoints, and then transferring to manual operation after saving the task checkpoints, are as follows: reading the instruction queue, the already saved business variables, and the breakpoint pointers to create a task checkpoint; writing the instruction queue, business variables, and breakpoint pointers into the same task checkpoint and saving the task checkpoint; after the task checkpoint is saved, sending a pause notification to the manual operation interface; the operator directly manipulates the browser context through the manual operation interface to handle browser operations that require manual intervention; after the operator completes the browser operation, confirming the completion of the operation on the manual operation interface.

[0014] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the steps for restoring the execution of instructions after manual operation are as follows: determining the next unexecuted step based on the breakpoint pointer and restoring business variables from the task checkpoint are as follows: after the manual operation interface confirms the completion of the operation, the saved task checkpoint is loaded, and the breakpoint pointer and instruction queue are read; the execution position recorded in the breakpoint pointer is read, and the instruction is not retrieved again from the beginning position of the instruction queue, and the next unexecuted instruction is read from the instruction queue based on the execution position; the saved business variables are loaded from the task checkpoint.

[0015] As a preferred embodiment of the multi-client browser automated interaction method of the present invention, the step of executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed includes the following steps: reading the instruction commands and parameters in the next unexecuted instruction, identifying the variable references in the parameters one by one; searching for business variables according to the variable references, obtaining business data, and replacing the variable references in the parameters with the business data; sending the instruction commands and parameters to the browser context, the browser engine calling the browser operation, and executing the browser operation using the parameters; after an instruction is executed, obtaining the next unexecuted instruction according to the execution order stored in the instruction queue; obtaining and executing the remaining instructions in sequence until the browser task is completed.

[0016] Secondly, this invention provides a multi-client browser automated interaction system, comprising: a context establishment module, used to establish a browser context shared by multiple clients, receive browser tasks, and arrange instructions into an instruction queue according to execution order; a variable passing module, used to execute the instruction queue in the browser context, execute instructions sequentially and capture network request responses, obtain business data from the network request responses, save the business data as business variables, and use the business variables in instructions that require business data; a breakpoint saving module, used to pause the execution of instructions requiring manual operation, generate a breakpoint pointer, the breakpoint pointer points to the next unexecuted step in the instruction queue, save the instruction queue, business variables, and breakpoint pointer as a task checkpoint, and transfer to manual operation after the task checkpoint is saved; and an instruction recovery module, used to recover the execution of instructions after the manual operation is completed, determine the next unexecuted step based on the breakpoint pointer, recover the business variables from the task checkpoint, and start executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

[0017] The beneficial effects of this invention are as follows: by combining business variables with task checkpoints, business data can continue to be used after manual intervention, reducing the risk of parameter transmission interruption; by combining breakpoint pointers with instruction queues, tasks can continue to run from the point of no execution and maintain the execution order of instructions, reducing the risk of repeated execution and omissions, and improving the continuity and reliability of browser automation tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for automated interaction across multiple client browsers.

[0020] Figure 2 This is a schematic diagram of a multi-client browser automated interaction system.

[0021] Figure 3 A flowchart of the overall execution of the browser task.

[0022] Figure 4 A diagram illustrating the mechanism for saving and resuming task checkpoints. Detailed Implementation

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0026] Reference Figures 1-4 This is one embodiment of the present invention, which provides a multi-client browser automated interaction method, including the following steps:

[0027] S1. Establish a browser context shared by multiple clients, receive browser tasks, and arrange the instructions into an instruction queue according to the execution order.

[0028] Furthermore, multiple clients interact with the server via the Hypertext Transfer Protocol, and the server process checks whether a browser context exists; if the browser context exists, it continues to be used; if the browser context does not exist, a browser context is created.

[0029] It should be noted that the entire server process maintains only one browser context for multiple clients to use, and does not create separate browser contexts for different clients. In this embodiment, the multiple clients are those that are allowed to share the same browser session.

[0030] It should also be noted that the browser context is the execution environment within a browser instance, used by multiple clients to perform browser operations. The browser context includes multiple page instances, which are tab instances within the browser context. Furthermore, when multiple browser tasks are executed in parallel, the server executes the browser operations of different browser tasks in different page instances. Page navigation, page input, and page operations of different browser tasks operate on different page instances.

[0031] When multiple clients sharing the same browser session are allowed to use the browser context, the browser context saves browser session information and locally stored data, and maintains the browser session information and locally stored data during browser operations initiated by multiple clients.

[0032] It should be noted that browser session information consists of login credentials and session identifiers saved by the browser, used to maintain the login state during browser operations. Local storage data refers to persistent data stored in the browser by the web application, used to maintain the state of the page application that needs to be used during browser operations.

[0033] Taking login as an example, after a client performs a login operation, the browser context saves the browser session information formed during the login process; when other clients initiate browser operations using the same browser context, the browser context continues to retain the browser session information formed during the login operation.

[0034] Furthermore, it receives browser tasks initiated by the client and receives instructions sent by the client for the browser tasks.

[0035] It should be noted that a browser task is a browser operation task initiated by the client, used to organize the browser operations that need to be performed. Instructions are the operation content required to execute the browser task, used to perform browser operations within the browser context.

[0036] Taking the specific organization process of a browser task as an example, a browser task organizes multiple consecutive browser operations according to the business process, and the client sends corresponding instructions for different browser operations in the browser task; for example, multiple consecutive browser operations can be login, create a project and create a requirement in sequence.

[0037] Furthermore, after receiving the instructions sent by the client for the browser task, the instructions are recorded sequentially according to the execution order to form an instruction queue.

[0038] Once the instruction queue is formed, unexecuted instructions remain in the queue, maintaining the original execution order.

[0039] It should be noted that the instruction queue is a queue that stores instructions in the order of execution, and is used to provide instructions sequentially.

[0040] S2. Execute the instruction queue in the browser context, execute the instructions in sequence and capture the network request response, obtain business data from the network request response, save the business data as business variables, and use the business variables in instructions that require business data.

[0041] It should be noted that the browser runtime engine is the underlying engine that actually executes browser operations, used to execute browser operations contained in instructions within the browser context.

[0042] Furthermore, after the browser task begins execution, the server reads the browser task's instruction queue, retrieves unexecuted instructions according to the execution order stored in the queue, reads the instruction commands and parameters from the instructions, and sends the instructions and parameters to the browser context. The browser's runtime engine then invokes browser operations according to the instructions and executes the browser operations using the parameters. After one instruction is executed, the server continues to retrieve the next unexecuted instruction according to the execution order in the instruction queue. It should also be noted that instructions include instruction commands and parameters. For instructions that require retrieving business data, the parameters include the network address pattern.

[0043] Taking the execution process of browser operations as an example, the instructions can correspond to clicks, fills in, key presses, or page navigation operations. The browser engine calls the browser operation corresponding to the instructions and uses the parameters in the instructions to complete the corresponding operation.

[0044] The client specifies a network address pattern for the command that requires business data and sends the network address pattern as a command parameter to the server. The server retrieves commands according to the execution order in the command queue, reads the network address pattern, and calls the network request monitoring interface to start network request capture before executing the command, while passing the network address pattern to the network request monitoring interface. After the network request monitoring interface is started, it continuously captures network requests and network request responses that match the network address pattern and saves the captured network requests and network request responses.

[0045] Furthermore, the browser's runtime engine executes browser operations according to the instructions. After a browser operation generates a network request, the server calls the network request waiting interface, passing the network address pattern to the interface. The network request waiting interface reads the network requests captured after the network request monitoring interface starts, matches the network request address with the network address pattern; if a matching network request is detected, the network request response is obtained. If the network request waiting interface times out and no matching network request is detected, the server determines that the current instruction cannot continue execution and suspends the current instruction.

[0046] It should be noted that the network address pattern is an expression used to match network request addresses, represented using regular expressions. The network request monitoring interface is the interface for initiating network request capture, used to capture network requests according to the network address pattern. The network request waiting interface is the interface for obtaining network request responses, used to receive the network address pattern, detect network requests whose network addresses match the network address pattern, and obtain the network request responses.

[0047] After the server receives the network request response, it reads the response body from the network request response, parses the response body and reads the fields containing the business data that need to be obtained, obtains the field values, and saves the field values ​​as business data as business variables.

[0048] It should also be noted that the response body in a network request response contains the data returned by the interface and is used to carry business data. Business variables are variables that store business data and are used to provide business data before the instruction is executed.

[0049] Taking the network request response generated by the project creation operation as an example, the response body uses JavaScript object notation and is returned as a string. The server parses the response body according to JavaScript object notation, reads the project identifier field, obtains the project identifier, and saves the project identifier as a business variable. After the business variable saves the project identifier, the server can retrieve the project identifier through the variable reference.

[0050] It should also be noted that the project identifier is a unique identifier returned in the network request response generated by the project creation operation, used to indicate the project involved in the creation requirement operation.

[0051] The server retrieves instructions in the instruction queue in execution order, reads the parameters in each instruction one by one, and identifies variable references in the parameters. After identifying the variable references, it searches for the business variables according to the variable references, obtains the business data, and replaces the variable references in the parameters with the business data.

[0052] After the variable reference replacement is completed, the server reads the command and parameters and sends them to the browser context. The browser's runtime engine reads the command, calls the corresponding browser operation, reads the parameters, and executes the browser operation according to the parameters.

[0053] It should be noted that variable references are reference markers in the instruction parameters, containing the names of business variables, and are used to locate business variables and retrieve business data. For example, when the name of the business variable is projectUUID, the variable reference can be represented as "$projectUUID".

[0054] The server retrieves the instructions for creating a requirement according to the execution order in the instruction queue, reads the request body from the instruction parameters, and reads the project identifier field from the request body. The server identifies the variable references in the project identifier field, locates the business variable based on the variable references, obtains the project identifier, and replaces the variable references with the project identifier. After completing the variable reference replacement, the server sends the instructions and parameters to the browser context. The browser's runtime engine reads the instructions and parameters and executes the requirement creation operation using the parameters after variable reference replacement.

[0055] S3. Pause the execution of instructions requiring manual operation, generate a breakpoint pointer, the breakpoint pointer points to the next unexecuted step in the instruction queue, save the instruction queue, business variables and breakpoint pointer as a task checkpoint, and switch to manual operation after the task checkpoint is saved.

[0056] Furthermore, during instruction execution, the server repeatedly reads the current page's document object model to obtain the class attributes of page elements and reads the current page's text. The server examines the content in the class attributes to identify CAPTCHAs or slider verifications; the server uses regular expressions to match risk control prompts in the page text. After the server identifies a CAPTCHA, slider verification, or risk control prompt, it pauses the currently executing instruction.

[0057] It should be noted that the class attribute is the category attribute of a page element in Hypertext Markup Language.

[0058] For example, when the class attribute contains "captcha", "verify-code", or "seccode", it can recognize image verification codes; when the class attribute contains "slider" or "nc_wrapper", it can recognize slider verification; when the page text contains text such as "Please complete verification", "Security verification", "Access too frequently" or "Account has an anomaly", it can recognize risk control prompts.

[0059] Furthermore, after pausing a currently executing instruction, the server stops retrieving new instructions from the instruction queue. While network request capture has been initiated for the paused instruction, the network request monitoring interface continues to monitor network requests. The server maintains the execution order of unexecuted instructions in the instruction queue; the server reads the position of the currently executing instruction in the instruction queue and sequentially reads instructions starting from the next position to obtain the first unexecuted instruction; the server reads the execution position of the first unexecuted instruction in the instruction queue, creates a breakpoint pointer, and writes the execution position of the first unexecuted instruction in the instruction queue to the breakpoint pointer.

[0060] It should be noted that breakpoint pointers are data that record the execution position, used to indicate the next unexecuted step. The execution position corresponds to the next unexecuted instruction in the instruction queue.

[0061] After the server generates a breakpoint pointer, it reads the browser task's task identifier, instruction queue, saved business variables, and breakpoint pointer. The server then creates a task checkpoint, writes the task identifier, instruction queue, business variables, and breakpoint pointer into the same checkpoint, and saves the checkpoint. Until the checkpoint is saved, the server maintains the execution order of unexecuted instructions in the instruction queue and does not fetch new instructions.

[0062] It should be noted that task checkpoints are data records that store task identifiers, instruction queues, business variables, and breakpoint pointers; task identifiers are used to distinguish task checkpoints corresponding to different browser tasks.

[0063] For example, when the server saves task checkpoints, it can use serialization. It should be noted that serialization involves organizing the task identifier, instruction queue, business variables, and breakpoint pointers in the task checkpoint according to JavaScript object representation, and then converting the organized data into text data. The checkpoint file is the file that stores the serialized task checkpoints. In practice, the server organizes the task identifier, instruction queue, business variables, and breakpoint pointers in the task checkpoint according to JavaScript object representation, serializes the task checkpoint, and writes the serialized task checkpoint to the checkpoint file. For example, the checkpoint file corresponding to different browser tasks can be distinguished based on the task identifier.

[0064] After the task checkpoint is saved, the server stops automatically retrieving instructions and sends a pause notification to the manual operation interface. The operator uses the manual operation interface to manage the page instance used when the instruction paused, handling browser operations requiring manual intervention and completing any unfinished browser operations from the paused instruction. When a paused instruction needs to retrieve business data, the server retrieves the network request response captured by the network request monitoring interface during the manual operation, reads the response body, extracts the business data from the response body, and saves the business data as a business variable. During the manual operation, the server stops sending automatically executed instructions to the page instance used when the instruction paused. The operator confirms completion of the operation after completing the browser operation.

[0065] It should be noted that the manual operation interface is an interactive interface for manually operating the browser context, used to perform manual operations during the instruction pause. The manual confirmation of completion indicates that the browser operation that was not yet completed under the paused instruction has been finished.

[0066] It should also be noted that after the manual confirmation operation is completed, the server does not re-execute the suspended instructions, but resumes execution from the first unexecuted instruction based on the breakpoint pointer.

[0067] S4. Resume instruction execution after manual operation is completed, determine the next unexecuted step based on the breakpoint pointer, restore business variables from the task checkpoint, and start executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

[0068] Furthermore, after the user confirms the completion of the operation on the manual operation interface, the server loads the corresponding task checkpoint based on the task identifier of the browser task that needs to be restored, and reads the breakpoint pointer, instruction queue and business variables from the task checkpoint.

[0069] The server reads the execution position recorded in the breakpoint pointer, does not retrieve instructions from the beginning of the instruction queue, and reads the first unexecuted instruction at the execution position from the instruction queue.

[0070] Furthermore, the server loads the saved business variables from the task checkpoints and retains the business variables saved during manual operations. It then reads the instructions and parameters from the next unexecuted instruction and identifies variable references within the parameters. Once a variable reference is identified, the server searches for the corresponding business variable. After finding the business variable, the server retrieves the business data and replaces the variable references in the parameters with the business data. If no business variable corresponding to the variable reference is found, the current instruction cannot complete parameter assembly. It should be noted that when the next unexecuted instruction needs to retrieve business data, the server initiates network request capture before executing the instruction.

[0071] For example, after a manual operation is completed, the server reads the text data in the checkpoint file, parses the text data according to the JavaScript object notation, obtains and loads the business variables; the server reads the parameters in the next unexecuted instruction, identifies the variable references in the parameters, finds the already loaded business variables according to the variable references, obtains the business data, and then uses the business data to replace the variable references in the parameters.

[0072] After the server completes the parameter reading, it sends the instructions and parameters to the browser context used by the user. The browser's runtime engine then invokes the browser operation according to the instructions and executes the operation using the parameters. After the browser operation generates a network request, the server obtains the network request response, reads the response body, extracts the business data from the response body, and saves the business data as business variables.

[0073] After an instruction is executed, the server retrieves the next unexecuted instruction according to the execution order stored in the instruction queue, reads the instruction command and parameters, and identifies variable references in the parameters. Once a variable reference is identified, the server searches for the business variable based on the reference, obtains the business data, and replaces the variable reference in the parameters with the business data. Instructions that require retrieving business data initiate network request capture before execution.

[0074] The server sends instructions and parameters to the browser context. The browser's runtime engine invokes browser operations according to the instructions and executes them using the parameters. After the browser operation generates a network request, the server obtains the network request response, retrieves business data from the response, and saves the business data as business variables. The server continues to retrieve and execute the remaining instructions according to the execution order stored in the instruction queue until the browser task is completed.

[0075] This embodiment also provides a multi-client browser automated interaction system, including:

[0076] The context establishment module is used to establish a browser context shared by multiple clients, receive browser tasks, and arrange instructions into an instruction queue according to the execution order.

[0077] The variable passing module is used to execute the instruction queue in the browser context, execute instructions sequentially, capture network request responses, obtain business data from the network request responses, save the business data as business variables, and use the business variables in instructions that require business data.

[0078] The breakpoint saving module is used to pause the execution of instructions that require manual operation, generate a breakpoint pointer, and point to the next unexecuted step in the instruction queue. The instruction queue, business variables, and breakpoint pointer are saved as task checkpoints. After the task checkpoints are saved, manual operation is initiated.

[0079] The instruction recovery module is used to resume instruction execution after manual operation is completed. It determines the next unexecuted step based on the breakpoint pointer, restores business variables from the task checkpoint, and starts executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

[0080] In summary, this invention: by combining business variables with task checkpoints, business data can continue to be used after manual intervention, reducing the risk of parameter transmission interruption; by combining breakpoint pointers with instruction queues, tasks can continue running from the point of no execution and maintain the execution order of instructions, reducing the risk of repeated execution and missed execution, and improving the continuity and reliability of browser automation tasks.

[0081] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for automated interaction across multiple client browsers, characterized in that, include: Establish a shared browser context for multiple clients, receive browser tasks, and arrange instructions into an instruction queue according to the execution order; The browser context executes the instruction queue, executes the instructions sequentially and captures network request responses, retrieves business data from the network request responses, saves the business data as business variables, and uses the business variables in instructions that require business data. Pause the execution of instructions that require manual operation, generate a breakpoint pointer, the breakpoint pointer points to the next unexecuted step in the instruction queue, save the instruction queue, business variables and breakpoint pointer as a task checkpoint, and switch to manual operation after the task checkpoint is saved. Resume instruction execution after manual operation is completed, determine the next unexecuted step based on the breakpoint pointer, restore business variables from the task checkpoint, and start executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

2. The multi-client browser automated interaction method as described in claim 1, characterized in that, The steps for receiving browser tasks and arranging instructions into an instruction queue according to execution order are as follows: Receive browser tasks initiated by the client and receive instructions sent by the client for the browser tasks; Record the corresponding instructions sequentially according to the execution order of browser operations in the browser task; Form an instruction queue and maintain the execution order of the instructions.

3. The multi-client browser automated interaction method as described in claim 2, characterized in that, The execution of the instruction queue in the browser context, executing instructions sequentially and capturing network request responses, involves the following steps: Read the browser task's instruction queue and retrieve unexecuted instructions in the execution order; Obtain the instruction that requires the business data, and read the network address pattern, which is specified by the client for the instruction that requires the business data; Before executing instructions that require obtaining business data, call the network request monitoring interface to start network request capture and pass the network address pattern to the network request monitoring interface. Read the commands and parameters, and send the commands and parameters to the browser context; The browser engine invokes browser operations and executes them using parameters. After a browser operation generates a network request, it calls the network request waiting interface and passes the network address pattern to the network request waiting interface. The network request waits for the interface to detect network requests that match the network address pattern and then obtains the network request response. After one instruction is executed, the next unexecuted instruction is fetched according to the execution order.

4. The multi-client browser automated interaction method as described in claim 3, characterized in that, The steps for obtaining business data from the network request response, saving the business data as business variables, and using the business variables in instructions that require business data are as follows: Read the response body from the network request response and extract business data from the response body; Save business data as business variables; Retrieve instructions according to the execution order in the instruction queue, read the parameters in each instruction one by one, and identify variable references; Find business variables by variable reference and retrieve business data; Replace variable references in parameters with business data.

5. The multi-client browser automated interaction method as described in claim 4, characterized in that, The specific steps for pausing the execution of commands requiring manual operation are as follows: Read the page document object model to obtain the class attribute of page elements, the resource address of image elements, and the page text; Check the class attribute and resource address of image elements, and match them with risk control prompts in the page text; Recognize graphic verification codes, slider verifications, or risk control prompts; Pause the currently executing instruction and stop fetching new instructions from the instruction queue.

6. The multi-client browser automated interaction method as described in claim 5, characterized in that, The process of generating a breakpoint pointer, which points to the next unexecuted step in the instruction queue, is as follows: Read the position of the currently executing instruction in the instruction queue, and start reading instructions sequentially from the next position to obtain the first unexecuted instruction; Read the execution position of the first unexecuted instruction in the instruction queue and create a breakpoint pointer; Write the execution position of the first unexecuted instruction in the instruction queue to the breakpoint pointer.

7. The multi-client browser automated interaction method as described in claim 6, characterized in that, The steps for saving the instruction queue, business variables, and breakpoint pointers as task checkpoints, and then transferring to manual operation after saving the task checkpoints, are as follows: Read the instruction queue, saved business variables, and breakpoint pointers, and create task checkpoints; Write the instruction queue, business variables, and breakpoint pointers to the same task checkpoint and save the task checkpoint. After the task checkpoints are saved, a pause notification is sent to the manual operation interface; Manual intervention is required to directly manipulate the browser context through a user interface to handle browser operations that require human intervention. After manually completing the browser operation, confirm the completion of the operation on the manual operation interface.

8. The multi-client browser automated interaction method as described in claim 6 or 7, characterized in that, The process of restoring instruction execution after manual operation is completed involves determining the next unexecuted step based on the breakpoint pointer and restoring business variables from the task checkpoint. The specific steps are as follows: After the user confirms the operation in the manual operation interface, the saved task checkpoint is loaded, and the breakpoint pointer and instruction queue are read. Read the execution position recorded in the breakpoint pointer, do not retrieve the instruction from the beginning of the instruction queue, and read the next unexecuted instruction from the instruction queue according to the execution position; Load the saved business variables from the task checkpoint.

9. The multi-client browser automated interaction method as described in claim 8, characterized in that, The process of executing the remaining instructions in the instruction queue starting from the next unexecuted step until the browser task is completed includes the following steps: Read the instructions and parameters from the next unexecuted instruction, and identify the variable references in the parameters one by one; Find the business variable by variable reference, obtain the business data, and replace the variable reference in the parameter with the business data; The command and parameters are sent to the browser context, the browser engine invokes browser operations, and executes the browser operations using the parameters. After an instruction is executed, the next unexecuted instruction is retrieved according to the execution order stored in the instruction queue; Retrieve and execute the remaining instructions sequentially until the browser task is complete.

10. A multi-client browser automated interaction system, based on the multi-client browser automated interaction method according to any one of claims 1 to 9, characterized in that, include: The context establishment module is used to establish a browser context shared by multiple clients, receive browser tasks, and arrange instructions into an instruction queue according to the execution order. The variable passing module is used to execute the instruction queue in the browser context, execute instructions sequentially and capture network request responses, obtain business data from the network request responses, save the business data as business variables, and use the business variables in instructions that require business data. The breakpoint saving module is used to pause the execution of instructions that require manual operation, generate a breakpoint pointer, which points to the next unexecuted step in the instruction queue, and save the instruction queue, business variables and breakpoint pointer as a task checkpoint. After the task checkpoint is saved, manual operation is initiated. The instruction recovery module is used to resume instruction execution after manual operation is completed. It determines the next unexecuted step based on the breakpoint pointer, restores business variables from the task checkpoint, and starts executing the remaining instructions in the instruction queue from the next unexecuted step until the browser task is completed.

Images