Browser control method and device, electronic equipment and medium

Abstract

The invention relates to a browser control method and device, electronic equipment and a medium, and belongs to the technical field of computers. The method is applied to a first electronic device, the first electronic device is connected with a second electronic device, the second electronic device is used for installing a browser, and the method comprises the steps that when an instruction input by a user and related to the browser is received, the browser in the second electronic device is controlled to conduct starting operation, and the browser is controlled to start screen recording; page screenshots obtained by screen recording of the browser are displayed on the first electronic device in real time, and the page screenshots comprise first page screenshots obtained after starting operation of the browser; marking operable elements in the first page screenshot to obtain a first picture; and performing first operation on the browser according to the first picture and the first step in the instruction. According to the method and the device, the marked browser page screenshot is combined with the user instruction to guide the operation on the browser, so that the task can be simplified to improve the working efficiency.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a browser control method, device, electronic device and computer-readable storage medium. Background Art

[0002] Remote browser technology is a technology that transfers the browser's operating environment from the local device to a remote server or virtual environment. The current remote browser technology is mainly used to achieve remote cross-device access and remote browser isolation (RBI). Among them, remote cross-device access means that users can access remote browser services through different clients to perform operations such as web browsing; remote browser isolation creates an isolation layer between the user's device and Internet content to prevent malicious code such as cross-site scripting (XSS) attacks from directly affecting the user's device and data. Remote browser technology can enhance security and prevent data loss, but it also has the following problems: low work efficiency and poor user experience. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a browser control method, device, electronic device and computer-readable storage medium to improve the problems of low working efficiency and poor user experience existing in the current remote browser technology.

[0004] The embodiment of the present application is implemented as follows: In a first aspect, an embodiment of the present application provides a browser control method, which is applied to a first electronic device, wherein the first electronic device is connected to a second electronic device, and the second electronic device is used to install a browser. The method includes: upon receiving an instruction related to the browser input by a user, controlling the browser in the second electronic device to start an operation, and controlling the browser to start screen recording; displaying a page screenshot obtained by screen recording of the browser in real time on the first electronic device, wherein the page screenshot includes a first page screenshot after the browser starts the operation; marking the operable elements in the first page screenshot to obtain a first picture; and performing a first operation on the browser according to the first picture and the first step in the instruction.

[0005] In the above embodiment, by annotating the screenshot of the browser page of the second electronic device, the first electronic device can accurately identify the operable elements in the browser page, thereby assisting the first electronic device to more accurately understand the user input instructions and convert them into specific browser operations, reducing execution failures due to ambiguous or erroneous instructions, improving the accuracy of operations, and thus improving the user experience; and combining the annotated image (first image) with the user input instructions to operate the browser can efficiently complete complex tasks, and can also reduce the occupancy of computing resources and network bandwidth, thereby improving the response speed of operations and improving work efficiency.

[0006] In combination with a possible implementation manner of the first aspect, the marking of the operable elements in the first page screenshot to obtain the first image includes: obtaining the coordinates of the operable elements in the first page screenshot; and marking the operable elements in the first page screenshot according to the coordinates of the operable elements to obtain the first image.

[0007] In the above embodiment, by obtaining the coordinates of the operable elements in the first page screenshot, and then using the coordinates of the operable elements to mark the operable elements in the first page screenshot, the first electronic device can combine the coordinates of the operable elements with the user input instructions to guide the first electronic device to operate the browser, thereby enhancing the first electronic device's ability to parse instructions, providing the first electronic device with clear operating instructions, reducing exception handling caused by ambiguous or erroneous instructions, improving the accuracy of the operation, and thereby improving the user experience.

[0008] In combination with a possible implementation manner of the first aspect, before performing a first operation on the browser according to the first picture and the first step in the instruction, the method also includes: generating a prompt word according to the instruction; decomposing the instruction into multiple steps according to the prompt word, wherein the multiple steps include the first step.

[0009] In the above embodiment, by generating prompt words and decomposing the user input instructions according to the prompt words, a complex task can be broken down into multiple simple steps, and then each step is executed step by step, thereby reducing the resource requirements for executing each step, reducing the complexity of resource calculation, and dynamically adjusting the execution strategy of each step, thereby improving work efficiency; and the process of decomposing the user input instructions can more accurately identify the user's true intentions, enhance the ability to parse instructions, thereby improving the accuracy of the operation, so that the user input instructions are correctly executed, thereby improving the user experience.

[0010] In combination with a possible implementation manner of the first aspect embodiment, the first operation is performed on the browser according to the first image and the first step in the instruction, including: when the first image indicates that the browser has been started, converting the first step into a target instruction based on a preset data structure; when the instruction type of the target instruction is a page element operation type, controlling the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and controlling the browser to perform a first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first image.

[0011] In the above embodiment, the user input instruction is converted into a target instruction based on a preset data structure. This can convert the user's vague, non-standard or colloquial expression into an instruction that can be understood by the electronic device. It can not only express the user's intention more accurately and reduce misunderstandings or ambiguities, but also increase the probability that the user input instruction is correctly processed, thereby improving the user experience; and when the instruction type of the target instruction is a page element operation type, the page element that needs to be operated can be accurately located through the coordinates of the operable elements in the target instruction, which not only improves the accuracy of the operation, so that the user input instruction is correctly executed, thereby improving the user experience; it also improves the response speed of the operation, thereby improving work efficiency.

[0012] In combination with a possible implementation manner of the embodiment of the first aspect, the method further includes: when the instruction type of the target instruction is a tool type, calling a tool corresponding to the target instruction to perform a first operation on the browser.

[0013] In the above embodiment, when the instruction type of the target instruction is a tool type, by calling the tool corresponding to the target instruction to perform the first operation on the browser, the operation steps can be reduced, time can be saved, and work efficiency can be improved; and by calling the tool corresponding to the target instruction to perform the first operation on the browser, the operation process can be smoother and the target instruction can be completed easily, thereby improving the user experience.

[0014] In combination with a possible implementation manner of the embodiment of the first aspect, after performing a first operation on the browser according to the first image and the first step in the instruction, the method also includes: obtaining a second page screenshot after the first operation is performed on the browser; marking the operable elements in the second page screenshot to obtain a second image; performing a second operation on the browser according to the second image and the second step in the instruction until all steps in the instruction are completed.

[0015] In the above embodiment, a screenshot is taken and annotated for the page after each operation of the browser, so as to determine whether each operation is completed and discover potential abnormal situations in time to guide the execution of subsequent operations and ensure that the user input instructions are correctly executed, thereby improving the user experience; and the annotated picture after each operation is combined with the user input instructions to operate the browser, so as to efficiently complete complex tasks, thereby improving the response speed of the operation and improving work efficiency.

[0016] In combination with a possible implementation manner of the first aspect, the method further includes: monitoring browser events of the page screenshots obtained by the user performing screen recording of the browser, and controlling the browser to operate according to the browser events.

[0017] In the above embodiment, by monitoring the browser events of the page screenshots obtained by the user through screen recording of the browser, the user's operations can be captured in real time, and the browser can be controlled to operate according to the browser events, which can reduce the time the user waits for page loading or response, thereby improving work efficiency; and by monitoring the user's browser events and responding in real time, the user can be provided with a smooth experience close to that of the local browser, providing smooth interaction, thereby improving the user's experience.

[0018] In a second aspect, an embodiment of the present application provides a browser control device, which is included in a first electronic device, wherein the first electronic device is connected to a second electronic device, and the second electronic device is used to install a browser, and the device includes: a functional module, which is used to control the browser in the second electronic device to start an operation and control the browser to start screen recording when receiving an instruction related to the browser input by a user; a display module, which is used to display a page screenshot obtained by screen recording of the browser in real time on the first electronic device, wherein the page screenshot includes a first page screenshot after the browser starts the operation; a labeling module, which is used to label the operable elements in the first page screenshot to obtain a first picture; and a functional module, which is also used to perform a first operation on the browser according to the first picture and the first step in the instruction.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the processor being connected to the memory; the memory being used to store programs; the processor being used to call the programs stored in the memory to execute the method provided in the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run by a processor, it is executed to perform the method provided by the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment.

[0021] Other features and advantages of the present application will be described in the following description. The purpose and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings. The above and other purposes, features and advantages of the present application will be more clearly shown in the drawings.

[0023] Figure 1 A schematic diagram of a connection structure between a first electronic device and a second electronic device provided in an embodiment of the present application is shown.

[0024] Figure 2 A flowchart of a browser control method provided in an embodiment of the present application is shown.

[0025] Figure 3 A schematic diagram of marking operable elements in a first page screenshot according to the coordinates of the operable elements provided in an embodiment of the present application is shown.

[0026] Figure 4 A module schematic diagram of a browser control device provided in an embodiment of the present application is shown.

[0027] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following embodiments can be used as examples to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the scope of protection of the present application. It will be appreciated by those skilled in the art that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other.

[0029] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0030] Furthermore, the term "and / or" in this application is merely a term used to describe the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0031] In the description of the embodiments of the present application, unless otherwise clearly specified or limited, the technical term "connection" may refer to a direct connection or an indirect connection through an intermediate medium.

[0032] An embodiment of the present application provides a browser control method, which combines a marked screenshot of the browser page of a second electronic device with a user input instruction to guide the operation of the browser, so that the first electronic device can understand the user input instruction more accurately, simplify complex tasks, improve the accuracy and response speed of the operation, thereby improving work efficiency and enhancing the user experience.

[0033] The connection structure between the first electronic device and the second electronic device involved in the browser control method provided in the embodiment of the present application is as follows Figure 1As shown. Among them, the connection method between the first electronic device and the second electronic device includes but is not limited to a network connection (using the TCP / IP protocol for communication), a cloud service transit connection, a secure shell tunnel (Secure Shell Tunnel, SSH tunnel) connection, etc. The first electronic device controls the browser in the second electronic device to operate by executing the browser control method provided in the embodiment of the present application. In one possible implementation, the first electronic device may include an intelligent body, and the intelligent body executes the browser control method provided in the embodiment of the present application to control the browser in the second electronic device to operate. The second electronic device is used to install a browser for use by the first electronic device. The browser in the second electronic device includes but is not limited to Google Chrome, Mozilla Firefox, Brave, Tor Browser, etc. The above-mentioned first electronic device and second electronic device can be computers, mobile phones, servers and other devices.

[0034] The browser control method provided in the embodiment of the present application can be applied to the above-mentioned first electronic device. Figure 2 The flowchart shown illustrates the browser control method from the perspective of the first electronic device. The browser control method may include the following steps: Step S10: upon receiving a browser-related instruction input by a user, controlling the browser in the second electronic device to start an operation, and controlling the browser to start screen recording.

[0035] In the above steps, when the first electronic device receives a browser-related instruction input by the user (in the embodiment of the present application, the user input instruction and the instruction both refer to the browser-related instruction input by the user), it can control the browser in the second electronic device to start the operation through a command line or an application programming interface (Application Programming Interface, API interface), and at the same time open the remote debugging protocol port of the browser in the second electronic device. The first electronic device uses the remote debugging protocol to control the browser to start screen recording through the remote debugging protocol port.

[0036] In one possible implementation, the first electronic device controls the browser in the second electronic device to start the operation through a command line, which may be to send instructions to the second electronic device through a command line to start the executable file of the browser in the second electronic device and make the browser run in headless mode. The headless mode means that the browser runs without a graphical user interface (GUI).

[0037] Among them, when the first electronic device controls the browser in the second electronic device to start the operation through the API interface, it can be controlled by the API interface provided by tools such as Playwright and Puppeteer to start the browser in the second electronic device. Tools such as Playwright and Puppeteer are installed on the second electronic device. For example, the first electronic device sends an HTTP request to the second electronic device. After receiving the HTTP request, the second electronic device triggers the running of the Playwright script and starts the browser installed in the second electronic device through the Playwright script. The content, format, structure, etc. of the HTTP request must comply with the specifications of the remote debugging protocol supported by the browser installed on the second electronic device.

[0038] The purpose of opening the remote debugging protocol port of the browser in the second electronic device is to enable the first electronic device to control the browser in the second electronic device through the remote debugging protocol port using the remote debugging protocol. For example, when the browser in the second electronic device is a Chrome browser, by opening the Chrome browser's remote debugging protocol (Chrome DevTools Protocol, CDP) port, the first electronic device can control the Chrome browser through the CDP protocol.

[0039] The specific process of the first electronic device controlling the browser to start screen recording through the remote debugging protocol port using the remote debugging protocol may include: the first electronic device sends a screen recording instruction to the browser through the remote debugging protocol, the browser starts screen recording when receiving the screen recording instruction, and returns the page screenshot obtained by the screen recording to the first electronic device asynchronously in real time through the remote debugging protocol.

[0040] In step S10, the browser-related instructions input by the user may be instructions based on natural language, which refers to commands expressed by the user in the form of daily language. For example, the user inputs "search for recent news through Baidu" on the first electronic device, and "search for recent news through Baidu" is an instruction based on natural language.

[0041] Step S20: Displaying the page screenshots obtained by screen recording of the browser in real time on the first electronic device, wherein the page screenshots include the first page screenshots after the browser is started.

[0042] In step S20, when the first electronic device receives the page screenshot obtained by screen recording of the browser, it can render the page screenshot in real time to the page canvas of the first electronic device through the canvas application program interface (Canvas API), so that the first electronic device can display the page screenshot obtained by screen recording of the browser in real time, so that the user can intuitively understand the content of the browser in the second electronic device, provide the user with a usage experience similar to that of a local browser, and improve the user's experience. In addition, when the user uses the browser in the second electronic device, the first electronic device will continue to display the page screenshot obtained by screen recording of the browser in real time until the user ends the use of the browser in the second electronic device.

[0043] Among them, Canvas API is a new tag in HTML5. Its core is a <canvas>The Canvas API provides a bitmap that can be manipulated with JavaScript. The Canvas API has functions such as drawing graphics, real-time rendering, image processing, and animation.

[0044] Step S30: marking the operable elements in the first page screenshot to obtain a first picture.

[0045] In one possible implementation, the first electronic device annotates the operable elements in the first page screenshot, and the specific process of obtaining the first image may include: the first electronic device sends an instruction to the browser through the remote debugging protocol, and the browser obtains the coordinates of the operable elements in the first page screenshot according to the instruction, and returns the coordinates of the operable elements in the first page screenshot to the first electronic device; after the first electronic device obtains the coordinates of the operable elements in the first page screenshot, it annotates the operable elements in the first page screenshot according to the coordinates of the operable elements to obtain the first image. Among them, the process of annotating the operable elements in the first page screenshot according to the coordinates of the operable elements can be as follows: Figure 3 As shown. Among them, Figure 3 The process of marking the operable elements 1, 2, 3 and 4 in the first page screenshot according to the coordinates of the operable elements is shown. 1, 2, 3 and 4 are the numbers of the operable elements, indicating the order of the operable elements.

[0046] In one possible implementation, when the first electronic device includes a large language model, the first electronic device annotates the operable elements in the first page screenshot, and the specific process of obtaining the first image may include: the first electronic device sends an instruction to the browser through a remote debugging protocol, and the browser obtains the coordinates of the operable elements in the first page screenshot according to the instruction, and returns the coordinates of the operable elements in the first page screenshot to the first electronic device; after obtaining the coordinates of the operable elements in the first page screenshot, the first electronic device numbers the coordinates of the operable elements to obtain the numbered coordinates; and then annotates the operable elements in the first page screenshot according to the numbered coordinates to obtain the first image. Alternatively, the first electronic device sends an instruction to the browser through a remote debugging protocol, and the browser obtains the coordinates of the operable elements in the first page screenshot according to the instruction, and returns the coordinates of the operable elements in the first page screenshot to the first electronic device. After obtaining the coordinates of the operable elements in the first page screenshot, the first electronic device marks the operable elements in the first page screenshot according to the coordinates of the operable elements, and numbers the marked operable elements to obtain a first picture.

[0047] Wherein, in the case where the first electronic device includes a large language model, numbering the coordinates of the operable elements or numbering the annotated operable elements is to enable the large language model to locate and identify the operable elements in the first image more quickly; it can also help the large language model to extract text information from the first image. For example, through numbering, the large language model can identify the text content of each element and associate the text content of each element with its number, so as to better understand the page content, so that when the steps in the user input instruction are converted into instructions based on a preset data structure, the numbering can help the large language model to more clearly understand the page elements corresponding to each step, so as to more accurately execute the user input instruction. Moreover, in an embodiment of the present application, the large language model can be used to decompose the user input instruction and convert the decomposed instruction into an instruction based on a preset data structure. The large language model can be a multimodal model such as LLaMA, Qwen, VisCPM, etc.

[0048] In the above embodiment, the operable elements include but are not limited to buttons, input boxes, selection boxes, links, etc. The operable element coordinates are square boxes, and the operable element coordinates include but are not limited to information such as the length, height, and left highest point of the box. In addition, the first electronic device can also annotate the operable elements in the first page screenshot through an image visual library such as OpenCV.

[0049] By obtaining the coordinates of the operable elements in the screenshot of the first page, and then using the coordinates of the operable elements to mark the operable elements in the screenshot of the first page, the first electronic device can combine the coordinates of the operable elements with the user input instructions to guide the first electronic device to operate the browser, thereby enhancing the first electronic device's ability to parse instructions, providing the first electronic device with clear operating instructions, reducing exception handling caused by ambiguous or erroneous instructions, improving the accuracy of the operation, and thereby improving the user experience.

[0050] Step S40: Perform a first operation on the browser according to the first picture and the first step in the instruction.

[0051] In a possible implementation, according to the first image and the first step in the instruction, the specific process of performing the first operation on the browser may include: when the first image indicates that the browser has been started, converting the first step into a target instruction based on a preset data structure; and judging the instruction type of the target instruction, when the instruction type of the target instruction is a page element operation type, controlling the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and controlling the browser to perform the first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first image. That is to say, when the first step is to operate on the page elements, when the first step is converted into a target instruction based on a preset data structure, the target instruction will contain the coordinates of the page element corresponding to the first step, and the coordinates are obtained through the first image.

[0052] In one possible implementation, the first electronic device may include an intelligent agent. Then, according to the first image and the first step in the instruction, the specific process of performing the first operation on the browser may include: when the first image indicates that the browser has been started, the intelligent agent converts the first step into a target instruction based on a preset data structure; and determines the instruction type of the target instruction. When the instruction type of the target instruction is a page element operation type, the intelligent agent controls the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and controls the browser to perform the first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first image.

[0053] In one possible implementation, the first electronic device may include not only an intelligent agent but also a large language model; then, according to the first picture and the first step in the instruction, the specific process of performing a first operation on the browser may include: when the first picture indicates that the browser has been started, the large language model converts the first step into a target instruction based on a preset data structure, and sends the target instruction to the intelligent agent; when the intelligent agent receives the target instruction, it determines the instruction type of the target instruction, and when the instruction type of the target instruction is a page element operation type, it controls the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and controls the browser to perform a first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first picture.

[0054] The large language model may be included in the intelligent agent, or may be a component of the first electronic device parallel to the intelligent agent.

[0055] In the above embodiment, the first step in the instruction refers to the first step of a plurality of steps having an execution order obtained after decomposing the instruction.

[0056] Among them, the specific process of converting the first step into a target instruction based on a preset data structure may include: converting the first step in the natural language-based user input instruction into a JSON object (target instruction based on a preset data structure) that conforms to the JSON_SCHEMA definition through a preset BrowserAction data model. In some embodiments, when the first electronic device includes an intelligent agent, the preset BrowserAction data model may be included in the intelligent agent; in some embodiments, when the first electronic device includes a large language model, the preset BrowserAction data model may be included in the large language model. The data dictionary of the above-mentioned BrowserAction data model is shown in Table 1: Table 1

[0057] In the BrowseAction data model, the Element_Handler field represents the page element operation field, where the operations on the page elements include but are not limited to single click, double click, fill, hover, key, etc.; the field description table of the Element_Handler field can be shown in Table 2: Table 2

[0058] The new_page field indicates opening a new page, and will return the page that needs to be opened by the browser in the second electronic device. The task_complete field indicates that the task is completed, that is, the user input instruction is completed. unable indicates that the operation cannot be performed, and generally the operation requires manual intervention, such as entering a user password. The tool field indicates calling an automated tool, and the field description table of the tool field can be shown in Table 3: Table 3

[0059] The instruction type of the target instruction being the page element operation type means that the instruction type of the target instruction is Element_Handler.

[0060] In the above embodiment, when the instruction type of the target instruction is a page element operation type, the page element that needs to be operated can be accurately located through the coordinates of the operable elements in the target instruction, which not only improves the accuracy of the operation, so that the user input instruction is correctly executed, thereby improving the user experience; it also improves the response speed of the operation, thereby improving work efficiency.

[0061] In a possible implementation manner, when the instruction type of the target instruction is a tool type, a tool corresponding to the target instruction is called to perform a first operation on the browser.

[0062] In the above embodiment, the target instruction's instruction type is a tool type, which means that the target instruction's instruction type is tool. When the target instruction's instruction type is tool, the first electronic device will call the tool corresponding to the target instruction to perform a first operation on the browser. For example, when the browser page in the second electronic device contains a verification code, the target instruction will choose to call the verification code bypass tool to operate the browser, and the verification code bypass tool can identify the verification code on the browser page and return the instruction operation of automatically filling in the verification code.

[0063] Under this implementation mode, by calling the tool corresponding to the target instruction to perform the first operation on the browser, the operation steps can be reduced, time can be saved, and work efficiency can be improved; and by calling the tool corresponding to the target instruction to perform the first operation on the browser, the operation process can be smoother and the target instruction can be completed easily, thereby improving the user experience.

[0064] In one possible implementation, before performing a first operation on the browser according to the first picture and the first step in the instruction, the browser control method provided in the embodiment of the present application also includes: generating a prompt word according to the instruction; decomposing the instruction into multiple steps according to the prompt word, wherein the multiple steps include the first step.

[0065] In this embodiment, the first electronic device may include an agent, which may generate prompt words according to the instruction and decompose the instruction into multiple steps according to the prompt words.

[0066] In this implementation, the first electronic device may include not only an agent but also a large language model. The agent may generate prompt words according to the instruction and send the prompt words to the large language model, and the large language model may decompose the instruction into multiple steps according to the prompt words.

[0067] Among them, an agent refers to an entity that can perceive the environment and make decisions based on the perceived information to achieve a specific goal. An agent can be a software program, a robot, or a system. The functions of an agent include but are not limited to the following: perception function, decision-making function, action function, learning function, etc. The prompt words in the above embodiment include but are not limited to the following: system prompt words: "You are a browser robot that supports using browser commands to control the browser. You can choose your corresponding operations through screenshots and instructions provided by the user. Note that you only need to follow the steps. (You are a browser robot that supports using browser commands to control the browser. You can choose your corresponding operations through screenshots and instructions provided by the user. Note that you only need to follow the steps)", Represents the user's natural language instructions, Indicates the execution action description of each stage, Indicates that according to the definition of JSON_SCHEMA, a JSON object of type "BrowserAction" is returned. Represents the automated tool being invoked.

[0068] The following is an example to illustrate the generated prompt words. If the first step in the user input instruction requires calling the verification code bypass tool, the generated prompt words are: You can choose the tools to use according to your needs. The list of supported tools is as follows: (You can choose the appropriate tool according to your needs. The following is a list of supported tools: ).

[0069] In a possible implementation, after performing a first operation on the browser according to the first picture and the first step in the instruction, the browser control method provided in the embodiment of the present application also includes: obtaining a screenshot of the second page after the first operation is performed on the browser; annotating the operable elements in the screenshot of the second page to obtain a second picture; performing a second operation on the browser according to the second picture and the second step in the instruction until all the steps in the instruction are completed. That is to say, after the instruction is decomposed into a plurality of steps with an execution order, each time the browser is operated according to the steps, it is necessary to take a screenshot of the browser page after the operation and annotate the screenshot until all the steps in the instruction are completed. For example, the instruction input by the user is "Search for recent news in Baidu", which can be decomposed into three steps: the first step is to open the Baidu page, the second step is to enter the search for recent news in the search box of the Baidu page, and the third step is to click the search button of the Baidu page; after the browser is operated according to the first step, the second step, and the third step, it is necessary to take a screenshot of the page after the operation and annotate the screenshot.

[0070] The second step in the above embodiment refers to the second step among a plurality of steps having an execution order obtained after decomposing the instruction.

[0071] In this implementation, the specific process of performing a second operation on the browser according to the second image and the second step in the instruction may include: when the second image indicates that the first step has been completed, converting the second step into a first target instruction based on a preset data structure, and determining the instruction type of the first target instruction; when the instruction type of the first target instruction is a page element operation type, controlling the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the first target instruction, and controlling the browser to perform a second operation on the operable elements corresponding to the first target instruction, wherein the coordinates of the operable elements in the first target instruction are obtained through the second image.

[0072] If the second image indicates that the first step is not completed or executed incorrectly, the first electronic device determines whether the first step cannot be executed based on the information of the second image; if the information of the second image indicates that the first step cannot be executed, the first electronic device sends a message to the user that the instruction cannot be executed, and saves relevant information related to the failure to execute the first step for subsequent analysis and improvement; if the information of the second image indicates that the first step can be executed, the first electronic device again controls the browser in the second electronic device to perform the first operation.

[0073] In one possible implementation, before performing the second operation on the browser according to the second picture and the second step in the instruction, the browser control method provided in the embodiment of the present application further includes: adding a description of the execution of the first step to the prompt word generated according to the instruction. The description of the execution of the first step refers to the step content of the first step. For example, the description of the first step: opening the Baidu page is Currently executed to: step 1. {{Open Baidu page}}. In one possible implementation, the browser control method provided in the embodiment of the present application also includes: monitoring browser events generated by a user taking screenshots of a page obtained by screen recording the browser, and controlling the browser to operate according to the browser events.

[0074] In this implementation, the first electronic device can monitor browser events caused by the page screenshots obtained by the user through screen recording of the browser through the Canvas canvas, and when the browser events are monitored, send instructions related to the browser events to the browser through the remote debugging protocol. When the browser receives the instructions related to the browser event, the browser performs the same operation on the browser page as the browser event.

[0075] Among them, browser events include but are not limited to mouse events, keyboard events, window and document events, form events, etc. Mouse events may include events such as mouse click, mouse double click, mouse press, etc. Keyboard events may include events such as keyboard press, keyboard release, input box content change, etc. Window and document events may include events such as window size change, window scrolling, document loading completion, etc. Form events may include events such as form submission, form reset, text selection, etc.

[0076] The browser control method provided by the embodiment of the present application will be described below through an example. Take the first electronic device including an intelligent agent and a large language model as an example. If the user input command is "Baidu search for recent news", when the intelligent agent receives the user input command, it controls the browser in the second electronic device to start the operation, and controls the browser to start screen recording; the page screenshot obtained by the browser screen recording is displayed in real time on the first electronic device, wherein the page screenshot includes the first page screenshot after the browser starts the operation; the operable elements in the first page screenshot are marked to obtain the first picture.

[0077] The agent generates the first prompt word according to the user input command "Baidu search for recent news", which is: You are a browser robot that supports using browser commands to control the browser. You can choose your corresponding operations through screenshots and instructions provided by the user. Note that you only need to follow the steps. The user goals for the current task are: Search Baidu for recent news. You need to return the results to me in JSON objects JSON objects oftype "BrowserAction" according to the following JSON Schema definitions:

[0078] (You need to return the result to me as a JSON object of type "BrowserAction" according to the following JSON Schema definition: ) Return JSON object with 2 spaces of indentation. The intelligent agent sends the above-mentioned first prompt word to the large language model. When receiving the above-mentioned first prompt word, the large language model decomposes the user input instruction "Baidu search recent news" into three steps according to the above-mentioned first prompt word: the first step is to open the Baidu page, the second step is to enter the search for recent news in the search box of the Baidu page, and the third step is to click the search button of the Baidu page.

[0079] In the case where the first picture represents that the browser has been started, the large language model converts the first step into a first instruction based on a preset data structure, the instruction type of the first instruction is new page, and the operation value is https: / / www.baidu.com , that is, the first instruction is to open a new page, the page URL is https: / / www.baidu.com The large language model sends the first instruction to the agent, and the agent then sends the first instruction to the browser. When the browser receives the first instruction, it opens the Baidu page. After the browser performs the first operation according to the first instruction, the agent obtains a screenshot of the second page after the first operation, and annotates the operable elements in the screenshot of the second page to obtain a second picture.

[0080] The agent adds a description of the execution of the first step to the generated first prompt word, and obtains the second prompt word, which is: You are a browser robot that supports using browser commands to control the browser. You can choose your corresponding operations through screenshots and instructions provided by the user. Note that you only need to follow the steps. The user goals for the current task are: Search Baidu for recent news. Currently executed to: step 1. {{ Open Baidu page}}. You need to return the results to me in JSON objects JSON objects oftype "BrowserAction" according to the following JSON Schema definitions:

[0081] (You need to return the result to me as a JSON object of type "BrowserAction" according to the following JSON Schema definition: ) Return JSON object with 2 spaces of indentation. The intelligent agent sends the second prompt word and the second picture to the large language model. The large language model can judge whether the first step is completed based on the second prompt word and the second picture. If the second picture indicates that the first step has been completed, the second step is converted into a second instruction based on a preset data structure, and the second instruction is sent to the intelligent agent. According to the second instruction, the intelligent agent performs a second operation on the browser, and takes a screenshot of the browser page after the second operation to obtain a third page screenshot; the operable elements in the third page screenshot are annotated to obtain a third picture. The intelligent agent then adds a description of the execution of the second step to the second prompt word to obtain a third prompt word; the third prompt word and the third picture are then sent to the large language model. The large language model judges whether the second step is completed based on the third prompt word and the third picture. If the third picture indicates that the second step has been completed, the third step is converted into a third instruction based on a preset data structure, and the third instruction is sent to the intelligent agent. The agent performs a third operation on the browser according to the third instruction, and takes a screenshot of the browser page after the third operation to obtain a fourth page screenshot; the operable elements in the fourth page screenshot are annotated to obtain a fourth picture; the agent then adds a description of the execution of the third step to the third prompt word to obtain a fourth prompt word, and then sends the fourth prompt word and the fourth picture to the large language model. The large language model can determine whether the third step is completed based on the fourth prompt word and the fourth picture. If the third step is completed, it indicates that the user input instruction has been completed.

[0082] like Figure 4 As shown, Figure 4 The structure diagram of a browser control device 10 provided in an embodiment of the present application is shown. The browser control device 10 is included in a first electronic device, the first electronic device is connected to a second electronic device, and the second electronic device is used to install a browser. The browser control device 10 includes a function module 11, a display module 12, and a marking module 13.

[0083] The function module 11 is used to control the browser in the second electronic device to start the operation and control the browser to start screen recording when receiving a browser-related instruction input by the user.

[0084] The display module 12 is used to display in real time on the first electronic device the page screenshots obtained by screen recording of the browser, wherein the page screenshots include the first page screenshots after the browser is started.

[0085] The marking module 13 is used to mark the operable elements in the first page screenshot to obtain a first picture.

[0086] Functional module 11 is also used to perform a first operation on the browser according to the first picture and the first step in the instruction.

[0087] In a possible implementation manner, the marking module 13 is specifically used to obtain the coordinates of the operable elements in the first page screenshot; and mark the operable elements in the first page screenshot according to the coordinates of the operable elements to obtain a first image.

[0088] In a possible implementation manner, the functional module 11 is further configured to generate a prompt word according to the instruction; and decompose the instruction into a plurality of steps according to the prompt word, wherein the plurality of steps include the first step.

[0089] In one possible implementation, the functional module 11 is specifically used to convert the first step into a target instruction based on a preset data structure when the first image represents that the browser has been started; when the instruction type of the target instruction is a page element operation type, control the browser to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and control the browser to perform a first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first image.

[0090] In a possible implementation manner, the function module 11 is further configured to, when the instruction type of the target instruction is a tool type, call a tool corresponding to the target instruction to perform a first operation on the browser.

[0091] In a possible implementation manner, the marking module 13 is further used to obtain a second page screenshot after the first operation is performed on the browser; and mark the operable elements in the second page screenshot to obtain a second image.

[0092] In a possible implementation manner, the functional module 11 is further used to perform a second operation on the browser according to the second picture and the second step in the instruction until all the steps in the instruction are completed.

[0093] In a possible implementation manner, the functional module 11 is further used to monitor browser events generated by a user recording a page screenshot obtained by screen recording the browser, and control the browser to operate according to the browser events.

[0094] The browser control device 10 provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment. like Figure 5 As shown, Figure 5 The structure block diagram of an electronic device 20 provided in an embodiment of the present application is shown. The electronic device 20 is the first electronic device in the above embodiment. The electronic device 20 includes: a processor 21 and a memory 22.

[0095] Figure 5 The components and structures of the electronic device 20 shown are merely exemplary and non-limiting. The electronic device 20 may also have other components and structures as required.

[0096] The processor 21, the memory 22, and other components that may appear in the electronic device 20 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, the processor 21, the memory 22, and other components that may appear may be electrically connected to each other via one or more communication buses or signal lines.

[0097] The memory 22 is used to store programs, for example, the program corresponding to the browser control method mentioned above is stored.

[0098] The processor 21 is used to call the program stored in the memory to execute the browser control method mentioned above.

[0099] Among them, the memory 22 can be, but is not limited to, a random access memory (Random Access Memory, RAM), a read only memory (Read Only Memory, ROM), a programmable read-only memory (Programmable Read-Only Memory, PROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0100] The processor 21 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 21 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a graphics processor (GPU), an accelerated processor (Accelerated Processing Unit), a multimedia application processor (MAP), a microprocessor, etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. Or the processor 21 may also be any conventional processor, etc.

[0101] The embodiment of the present application also provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium), on which a computer program is stored. When the computer program is run by the processor 21 as described above, the method disclosed in any of the embodiments shown above is executed.

[0102] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0103] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0105] The aforementioned computer-readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0106] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / canvas>

Claims

1. A browser control method, characterized in that: Applied to a first electronic device, the first electronic device is connected to a second electronic device, the second electronic device is used to install a browser, and the method includes: When receiving a command related to the browser input by a user, controlling the browser in the second electronic device to start an operation, and controlling the browser to start screen recording; Displaying in real time on the first electronic device a screenshot of a page obtained by screen recording of the browser, wherein the screenshot of the page includes a first screenshot of the page after the browser performs a startup operation; Annotate the operable elements in the first page screenshot to obtain a first picture; According to the first picture and the first step in the instruction, a first operation is performed on the browser.

2. The method according to claim 1, characterized in that The step of marking the operable elements in the first page screenshot to obtain a first picture includes: Obtaining coordinates of operable elements in the first page screenshot; The operable elements in the first page screenshot are marked according to the operable element coordinates to obtain a first picture.

3. The method according to claim 1, characterized in that Before performing a first operation on the browser according to the first picture and the first step in the instruction, the method further includes: Generate a prompt word according to the instruction; The instruction is decomposed into a plurality of steps according to the prompt word, wherein the plurality of steps include the first step.

4. The method according to claim 2, characterized in that: The performing a first operation on the browser according to the first picture and the first step in the instruction includes: In the case where the first picture indicates that the browser has been started, converting the first step into a target instruction based on a preset data structure; When the instruction type of the target instruction is a page element operation type, the browser is controlled to traverse the operable elements in the browser page according to the coordinates of the operable elements in the target instruction, and the browser is controlled to perform a first operation on the operable elements corresponding to the target instruction, wherein the coordinates of the operable elements in the target instruction are obtained through the first image.

5. The method according to claim 4, characterized in that The method further comprises: When the instruction type of the target instruction is a tool type, a tool corresponding to the target instruction is called to perform a first operation on the browser.

6. The method according to claim 1, characterized in that After performing a first operation on the browser according to the first picture and the first step in the instruction, the method further includes: Obtaining a screenshot of a second page after performing a first operation on the browser; Annotate the operable elements in the second page screenshot to obtain a second picture; According to the second picture and the second step in the instruction, a second operation is performed on the browser until all the steps in the instruction are completed.

7. The method according to claim 1, characterized in that The method further comprises: The browser events generated by the user performing screen recording of the browser and the screenshot of the page are monitored, and the browser is controlled to operate according to the browser events.

8. A browser control device, characterized in that: The device is included in a first electronic device, the first electronic device is connected to a second electronic device, and the second electronic device is used to install a browser. The device includes: A functional module, configured to control the browser in the second electronic device to start an operation and control the browser to start screen recording when receiving an instruction related to the browser input by a user; A display module, configured to display in real time on the first electronic device a page screenshot obtained by screen recording by the browser, wherein the page screenshot includes a first page screenshot after the browser performs a startup operation; A marking module, used for marking the operable elements in the first page screenshot to obtain a first picture; The functional module is further used to perform a first operation on the browser according to the first picture and the first step in the instruction.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is used to call the program stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images