Webpage automation task detection method and device, and electronic device

Abstract

This application provides a method, apparatus, and electronic device for detecting automated web page tasks, relating to the field of web page processing technology. The method for detecting automated web page tasks includes: acquiring execution information of a web page operation task; the execution information includes: a first operation instruction for instructing an operation on a web page and a web page operation interface; detecting whether the task execution result of the web page operation task is correct based on the first operation instruction and the web page operation interface; and, if the task execution result is incorrect, determining the error type of the task execution result; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the web page. This application can accurately analyze the error type of the execution result when an error is detected in the execution result of an automated web page task, thereby optimizing the execution process of the automated web page task.

Description

Technical Field

[0001] This application relates to the field of web page processing technology, and in particular to a method, apparatus and electronic device for detecting automated web page tasks. Background Technology

[0002] With the rapid development of artificial intelligence and web automation technologies, automating web operation tasks through large-scale models has become a trend. Taking the VLA (Visual-Language-Action) model as an example, the VLA model can understand natural language commands and interact with the web environment. When a user inputs a command via voice, the VLA model recognizes the command and executes the corresponding action on the webpage, thereby promoting advancements in automated operations and maintenance, automated testing, and data collection technologies.

[0003] In web page automation processes, accurately determining the success or failure of an operation and precisely attributing the error type directly impacts the closed-loop, self-healing, and human-machine collaboration efficiency of web page automation tasks. This places higher demands on the model's judgment and error correction capabilities. Currently, related technologies can only simply determine the success or failure of web page operations, making it difficult to conduct detailed analysis of the reasons or types of failures, resulting in low efficiency in subsequent repair of web page automation tasks. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for detecting web page automation tasks, in order to alleviate or solve one or more technical problems existing in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for detecting web page automation tasks, including:

[0006] Obtain execution information for webpage operation tasks; the execution information includes: a first operation instruction for instructing the operation of the webpage and a webpage operation interface;

[0007] Based on the first operation instruction and the webpage operation interface, detect whether the task execution result of the webpage operation task is correct;

[0008] In the event of an error in the task execution result, the error type of the task execution result is determined; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an error in the operation performed on the webpage.

[0009] Secondly, embodiments of this application provide a detection device for web page automation tasks, comprising:

[0010] The acquisition module is used to acquire the execution information of the webpage operation task; the execution information includes: a first operation instruction for instructing the operation of the webpage and the webpage operation interface;

[0011] The detection module is used to detect whether the task execution result of the web operation task is correct based on the first operation instruction and the web operation interface;

[0012] The determination module is used to determine the error type of the task execution result when the task execution result is incorrect; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0016] According to the technical solution of this application embodiment, execution information of a webpage operation task is obtained. The execution information includes a first operation instruction for instructing the operation on the webpage and a webpage operation interface. Based on the first operation instruction and the webpage operation interface, the correctness of the task execution result of the webpage operation task is detected. If the task execution result is incorrect, the error type of the task execution result is determined. The error type includes a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage. Therefore, this technical solution can clearly distinguish whether the error type of the task execution result is an error in the first operation instruction or an incorrect operation performed on the webpage when an error is detected in the task execution result of a webpage operation task, thereby optimizing the execution process of the automated webpage task in a targeted manner. Furthermore, determining the error type of the task execution result through the first operation instruction and the webpage operation interface ensures that the error type discrimination does not rely on human subjective experience but is based on objective visual information and operation instructions, thus ensuring the accuracy of the error type.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0019] Figure 1 A flowchart illustrating the webpage automation task detection method provided in an embodiment of this application is shown;

[0020] Figure 2 The diagram shows a web interface of the webpage automation task detection method provided in an embodiment of this application.

[0021] Figure 3 Another webpage operation interface diagram of the webpage automation task detection method provided in the embodiments of this application is shown;

[0022] Figure 4 A flowchart of a method for detecting web page automation tasks according to another embodiment of this application is shown;

[0023] Figure 5 A block diagram of a web page automation task detection device provided in an embodiment of this application is shown;

[0024] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0026] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0027] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flowchart of the web page automation task detection method provided in an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101, S102 and S103.

[0029] Step S101: Obtain the execution information of the webpage operation task. The execution information includes: a first operation instruction for instructing the operation of the webpage and the webpage operation interface.

[0030] Optionally, the execution information also includes historical operation records of actions performed on the webpage. Historical operation records refer to records of operations performed on the webpage based on the first operation instruction after it has been received. The content of the historical operation records is not limited and may include at least one of the following: the type of operation performed on the webpage, the operation location, the operation content, etc. For example, the operation type may be a click operation, a selection operation, an input operation, etc.; the operation location is the interaction location with the webpage; and the operation content includes the input content on the webpage.

[0031] Before executing step S101, a webpage operation task is first executed. The execution of the webpage operation task includes the following steps: receiving a first operation instruction for the webpage operation task; and performing corresponding operations on the webpage based on the first operation instruction. During or after the execution of the webpage operation task, execution information of the webpage operation task can be stored.

[0032] A webpage user interface includes at least one of the following: a first interface before performing an operation on the webpage, a second interface after performing an operation on the webpage, and a third interface including operation information for the operation on the webpage. The operation information includes operation location information and / or operation content.

[0033] The webpage interface can be obtained by taking a screenshot. Optionally, in response to receiving a first operation instruction, a screenshot of the webpage is taken to obtain a first interface. Then, it is detected whether the content of the webpage interface has changed. If a change is detected, a screenshot of the changed webpage is taken to obtain a second and a third interface.

[0034] If the operation information for interacting with a webpage includes the location of the operation, then when taking a screenshot of the webpage, the operation location can be marked on the screenshot. The marking method is not limited; for example, a red dot or a line box can be used to mark the operation location. This way, the obtained webpage interface will have the relevant markings of the operation location, making it easier to check whether the task execution result is correct.

[0035] This application does not limit the timing of executing step S101. Step S101 can be executed immediately after the webpage operation task is completed; it can also be executed after a preset time after the webpage operation task is completed; or it can be executed according to a preset cycle. Before executing step S101, the execution information of the webpage operation task can be stored in advance.

[0036] Optionally, the user inputs voice information, and the user-input voice information is subjected to speech recognition, thereby generating a first operation command based on the speech recognition result. For example, after the user inputs the voice information "Select 'XX' site", the voice information "Select 'XX' site" is subjected to speech recognition, thereby generating the first operation command "Select 'XX' site".

[0037] Step S102: Based on the first operation instruction and the web operation interface, check whether the task execution result of the web operation task is correct.

[0038] Optionally, step S102 can be performed as follows:

[0039] First, based on the initial operation instruction and historical operation records, the expected execution result of the webpage operation task is determined. Optionally, the expected execution result can also be determined in conjunction with the webpage operation interface. The expected execution result is the correct task execution result of the webpage operation task, and may include at least one of the following information: expected operation location, expected operation effect, and expected operation content.

[0040] Secondly, check whether the task execution result matches the expected execution result. If the task execution result does not match the expected execution result, the task execution result is determined to be incorrect; if the task execution result matches the expected execution result, the task execution result is determined to be correct.

[0041] The mismatch between the task execution result and the expected execution result may include at least one of the following: the operation location on the webpage is different from the expected operation location; the operation content on the webpage (such as the content entered in the input box) is different from the expected operation content; or the operation effect on the webpage is different from the expected operation effect.

[0042] For example, based on the operation location markers on the webpage's interface, the operation location on the webpage can be determined. It can then be checked whether this operation location matches the expected operation location; if they do not match, the operation location on the webpage is determined to be different from the expected operation location. As another example, the first operation instruction might instruct the user to enter search terms in the webpage's input box. The webpage's interface includes a screenshot of the entered search terms. Based on this screenshot, the input content on the webpage can be determined. It can then be checked whether the input content matches the expected input content (i.e., the expected operation content); if they do not match, the operation content on the webpage is determined to be different from the expected operation content. As yet another example, based on the webpage's interface diagram, the interface content after performing an operation on the webpage can be determined, i.e., the operation effect. It can then be checked whether this operation effect matches the expected operation effect; if they do not match, the operation effect on the webpage is determined to be different from the expected operation effect.

[0043] Step S103: If the task execution result is incorrect, determine the error type of the task execution result; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an error in the operation performed on the webpage.

[0044] The first type indicates an error in the first operation instruction, meaning an incorrect first operation instruction was generated. For example... Figure 2 As shown, the first operation instruction is "click the button 'Search'", but the webpage display interface 20 does not contain the button "Search", but instead contains the button 21 "Search". This causes the first operation instruction to fail to execute correctly, indicating that an incorrect first operation instruction was generated.

[0045] The second type indicates that a correct first operation instruction was generated, but an incorrect operation was performed on the webpage based on that instruction. For example... Figure 3 As shown, the first operation instruction is "search for 'GUI virtual environment' on the webpage homepage". However, the task execution result is that "artificial intelligence model" was entered into the input box 31 on the webpage homepage 30 to perform the search, indicating that the wrong operation was performed on the webpage.

[0046] In this embodiment, the detection method for web page automation tasks and the web page automation tasks can be executed by the same pre-configured artificial intelligence model, or by different artificial intelligence models respectively. For example, the web page automation tasks are executed by a pre-configured VLA model, and the detection method for the web page automation tasks can be executed by the same VLA model, or by other pre-configured models.

[0047] According to the technical solution of this application embodiment, execution information of a webpage operation task is obtained. The execution information includes a first operation instruction for instructing the operation on the webpage and a webpage operation interface. Based on the first operation instruction and the webpage operation interface, the correctness of the task execution result of the webpage operation task is detected. If the task execution result is incorrect, the error type of the task execution result is determined. The error type includes a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage. Therefore, this technical solution can clearly distinguish whether the error type of the task execution result is an error in the first operation instruction or an incorrect operation performed on the webpage when an error is detected in the task execution result of a webpage operation task, thereby optimizing the execution process of the automated webpage task in a targeted manner. Furthermore, determining the error type of the task execution result through the first operation instruction and the webpage operation interface ensures that the error type discrimination does not rely on human subjective experience but is based on objective visual information and operation instructions, thus ensuring the accuracy of the error type.

[0048] In some embodiments, determining the error type of the task execution result can be performed by steps A1 and A2:

[0049] Step A1: If the first condition is met, determine the error type as a first type used to indicate an error in the first operation instruction; the first condition includes: the first operation instruction does not match the interface content of the first interface.

[0050] The mismatch between the first operation instruction and the content of the first interface can include the following: the content of the first interface does not contain the operation element corresponding to the first operation instruction; or the content of the first interface hides the operation element corresponding to the first operation instruction. The operation element corresponding to the first operation instruction refers to the operation object on the webpage indicated by the first operation instruction.

[0051] For example, the operation element corresponding to the first operation instruction is "select category", but the first interface does not contain the operation element "select category". That is, the interface content of the first interface does not contain the operation element corresponding to the first operation instruction. Therefore, it can be determined that the first operation instruction and the interface content of the first interface do not match.

[0052] For example, the operation element corresponding to the first operation instruction is "March 2022", but the earliest date option displayed on the first interface can only be selected as "April 2022", and more dates can be viewed through the date scroll bar. This indicates that the operation element "March 2022" corresponding to the first operation instruction is hidden. That is, the operation element corresponding to the first operation instruction is hidden in the interface content of the first interface. Therefore, it can be determined that the first operation instruction and the interface content of the first interface do not match.

[0053] Step A2: If the second condition is met, determine the error type as a second type used to indicate an erroneous operation performed on the webpage; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, or the operation information on the third interface does not match the first operation instruction.

[0054] The mismatch between the content of the second interface and the expected execution result can include: the content of the second interface does not contain the webpage content corresponding to the expected execution result. For example, if the first operation instruction is "click the 'Expand Categories' option", the expected execution result is the webpage content of the expanded categories. However, the content of the second interface does not contain the expanded category content, therefore it can be determined that the content of the second interface does not match the expected execution result.

[0055] Operation information includes operation location information and / or operation content. Operation location information can be represented by operation location markers on the interface. A mismatch between the operation information on the third interface and the first operation instruction may include: the location indicated by the operation location markers on the third interface differs from the operation location indicated by the first operation instruction; or the operation content on the third interface differs from the operation content indicated by the first operation instruction.

[0056] For example, the first operation instruction is "Enter 'XX' in the first input box", but the third interface displays "XX" as being entered in the second input box. That is, the operation position is marked in the second input box. This indicates that the operation position is incorrect when operating on the webpage. Therefore, it can be determined that the operation information on the third interface does not match the first operation instruction.

[0057] For example, the first operation instruction is "Enter 'XX' in the first input box", but the third interface displays "YY" as entered in the first input box. This indicates that the operation position is correct, but the operation content is incorrect. Therefore, it can be determined that the operation information on the third interface does not match the first operation instruction.

[0058] In this embodiment, by analyzing screenshots of web page operations, the first operation instruction, and historical operation records, the error type of the task execution result can be objectively determined based on the actual visual interface and operation records, without relying on human subjective judgment, thus ensuring the accuracy of the error type.

[0059] In some embodiments, if the error type of the task execution result is determined to be of the first type, the operation instructions are regenerated to instruct the webpage to perform an operation that matches the expected execution result.

[0060] The regeneration of the operation instructions can be performed by following steps B1, B2, and B3:

[0061] Step B1: Detect whether the first interface element corresponding to the first operation command is hidden on the first interface.

[0062] Optionally, when performing step B1, the configuration file of the webpage can be queried to determine whether the configuration file contains the first interface element corresponding to the first operation instruction; if it does, it is determined that the first interface element corresponding to the first operation instruction is hidden on the first interface.

[0063] Step B2: If the first interface elements are hidden on the first interface, a second operation instruction is generated; the second operation instruction is used to instruct the display of the first interface elements and to perform operations on the first interface elements.

[0064] Assuming the first operation command is "Select date option 'March 2022'", the corresponding first interface element is the date "March 2022". After determining that the first operation command is incorrect, by querying the webpage's configuration file, it was found that the first interface element is hidden and configured to be displayed by scrolling. Therefore, the following example of a second operation command can be generated: "Scroll down to view more dates and select date option 'March 2022'".

[0065] Step B3: If the first interface elements are not hidden on the first interface, then determine the second interface elements on the first interface that are related to the first operation instruction, and generate a third operation instruction based on the second interface elements; the third operation instruction is used to instruct the operation on the second interface elements.

[0066] If the first interface element is not hidden on the first interface, it means that the first interface element corresponding to the first operation instruction is an incorrect element or does not belong to the current webpage. In this case, the second interface element related to the first operation instruction on the first interface can be identified, and then the third operation instruction can be generated.

[0067] Assuming the first operation instruction is "Click the 'Search' button", the first interface element corresponding to the first operation instruction is the "Search" button. After determining that the first operation instruction is incorrect, by querying the configuration file, it is determined that the "Search" button is not hidden on the first interface, and that the first interface contains a second interface element related to the first operation instruction, namely the "Search" button. Therefore, the third operation instruction, as shown in the example below, can be generated: "Click the 'Search' button".

[0068] In this embodiment, when the error type of the task execution result is determined to be a first operation instruction error, the correct operation instruction can be regenerated based on the first operation instruction, so that the model can reoperate the webpage based on the correct operation instruction, thereby obtaining the correct task execution result. This achieves the effect of automatic repair after the webpage automation task execution error and optimizes the execution process of the webpage automation task.

[0069] In some embodiments, after performing step S103, error information of the task execution result is output. The error information may include at least one of the following: whether an error occurred, the cause of the error, the type of error, and the regenerated operation instructions. By outputting error information, the output of the detection results of the webpage operation task is standardized, thereby optimizing the process of the webpage operation task.

[0070] The output format of the detection results can be predefined. For example, for a Type I error, the output format is predefined as follows: {"result":false, "reason":"Error reason description (no more than 20 characters)", "plan":"New control instruction (single-step instruction, no more than 15 characters)", "type":"plan"}. For a Type II error, the output format is predefined as follows: {"result":false, "reason":"Error reason description (no more than 20 characters)", "type":"action"}. For a correct result, the output format is predefined as follows: {"result":true}.

[0071] For example, if the first instruction is "Select 'March 2022'", but the webpage does not display the option for 'March 2022', the first instruction cannot be executed correctly. Therefore, the error type can be determined as Type 1, and the reason is "There is no March 2022 option on the interface; the earliest you can select is April 2022". Therefore, the following error message can be output:

[0072] {"result":false, "reason":"The interface does not have a March 2022 option; the earliest selectable month is April 2022." "plan":"Select the earliest available month", "type":"plan"}. Here, "result" represents the detection result, "false" indicates an error, and "reason" indicates the reason for the error. In "plan":"Select the earliest available month", "plan" represents the regenerated operation instruction, i.e., "Select the earliest available month". In "type":"plan", "type" represents the error type, and "plan" indicates that the error type is an instruction error type, i.e., the first type.

[0073] Furthermore, if the task execution result is detected as correct, a message indicating successful task execution can be output, such as the following format: {"result":true}. "true" indicates that the result is correct.

[0074] It should be noted that the output formats listed above are merely examples and do not limit the output of detection results (including error messages or correct messages) in this application.

[0075] The following uses an autonomous driving system with a GUI (Graphical User Interface) virtual environment as an example to illustrate the specific implementation steps of the web page automation task detection method provided in this application. In the autonomous driving system with a GUI virtual environment, the page is the virtual interface.

[0076] The virtual environment autonomous driving system is pre-configured with a VLA model and a detection model. A VLA model is a multimodal artificial intelligence model that combines visual perception, language understanding, and action control, designed to enable machines (such as robots and autonomous vehicles) to understand their environment and perform physical actions through natural language commands. In the autonomous driving system, the VLA model receives user-input voice information, recognizes the voice information, generates a first operation command, and executes web page operation tasks based on the first operation command. The VLA model includes a planning sub-model and an action sub-model. The planning sub-model is mainly used to plan the execution commands for the web page operation tasks, including generating the first operation command. The action sub-model is used to perform corresponding operations on the virtual interface according to the first operation command generated by the planning sub-model, obtaining the task execution result of the web page operation task. The detection model is used to detect the task execution result output by the VLA model to check whether the task execution result is correct, and to determine the error type when the task execution result is incorrect.

[0077] Figure 4A flowchart of the web page automation task detection method provided in an embodiment of this application is shown, such as... Figure 4 As shown, the method may include steps S401 to S409.

[0078] Step S401: The planning sub-model performs speech recognition on the user-input speech information and generates the first operation command based on the recognition result.

[0079] Step S402: The execution sub-model performs operations on the webpage based on the first operation instruction to obtain the task execution result of the webpage operation task; during the webpage operation, a screenshot of the webpage is taken to obtain the webpage operation interface; and the webpage operation process is recorded to obtain the historical operation record.

[0080] The webpage operation interface includes: a first interface before performing any operation on the webpage, a second interface after performing any operation on the webpage, and a third interface containing operation information. The operation information includes operation location information and / or operation content.

[0081] The content of historical operation records is not limited and may include at least one of the following: the type of operation performed on the webpage, the location of the operation, the content of the operation, etc. For example, the operation type is a click operation, a selection operation, an input operation, etc.; the operation location is the interaction location with the webpage; the operation content includes the input content on the webpage.

[0082] Step S403: The detection model obtains the execution information of the web page operation task. The execution information includes the first operation instruction, the web page operation interface, and the historical operation record.

[0083] The execution subject of steps S404 to S409 below is the detection model, so the execution subject will not be described again.

[0084] Step S404: Determine the expected execution result of the webpage operation task based on the first operation instruction and historical operation records.

[0085] The expected execution result is the correct execution result of the webpage operation task.

[0086] Step S405: If the task execution result does not match the expected execution result, determine that the task execution result is incorrect.

[0087] Step S406: Determine the error type of the task execution result.

[0088] If the error type is a first type indicating an error in the first operation instruction, then step S407 is executed; if the error type is a second type indicating an error in the operation performed on the webpage, then step S409 is executed.

[0089] Step S407: Regenerate the operation instructions based on the interface content on the webpage and the first operation instructions.

[0090] Step S408: Output the error information of the task execution result. The error information includes the error type, error reason and regenerated operation instructions.

[0091] Optionally, if the error message includes regenerated operation instructions, the error message can be sent to the execution sub-model, which can then operate the webpage again according to the regenerated operation instructions to obtain the correct task execution result.

[0092] Step S409: Output the error information of the task execution result. The error information includes the error type and error reason.

[0093] For example, a user inputs the voice message "Drive to Cafe X". The planning sub-model analyzes the user's voice input and generates the following first operation instruction: Drive along route A on the virtual interface to point B. The execution sub-model executes this first operation, performing corresponding operations on the virtual interface. In the context of an autonomous driving scenario in a virtual environment, this manifests as the execution sub-model controlling the autonomous vehicle to drive along route A, i.e., controlling the change of the autonomous vehicle's position on the virtual interface. During the driving process, the execution sub-model can take screenshots of the virtual interface, optionally taking screenshots at predetermined intervals to obtain screenshots of the virtual interface at multiple time points during the driving process. After the driving task is completed, the detection model obtains the execution information, which includes the first operation instruction "Drive to Cafe X", the virtual interface screenshots, and the driving record (i.e., the historical operation record). The driving record represents the autonomous vehicle's position information on the virtual interface at different times and / or the autonomous vehicle's driving trajectory on the virtual interface. Based on the execution information, the detection model can determine the expected execution result of the first operation instruction: the autonomous vehicle drives along route A to point B.

[0094] The detection model compares the actual task execution result with the expected result. If an inconsistency exists, the task execution result is determined to be incorrect. Then, if the detection model detects that the virtual interface does not contain route A, the error type of the task execution result is determined to be the first type, indicating an error in the first operation instruction. In this case, the detection model can regenerate the operation instruction. Assuming the route from the current location to point B on the virtual interface is C, the detection model can generate the following new operation instruction: Drive along route C on the virtual interface to point B. If the detection model detects that the virtual interface contains route A, but the autonomous vehicle is traveling on route D, or the destination of the autonomous vehicle is not point B, the error type of the task execution result is determined to be the second type, indicating an incorrect operation performed on the webpage.

[0095] According to the technical solution of this application embodiment, execution information of a webpage operation task is obtained. This execution information includes a first operation instruction and a webpage operation interface for instructing the operation on the webpage. Based on the first operation instruction and the webpage operation interface, the correctness of the task execution result of the webpage operation task is detected. If the task execution result is incorrect, the error type of the task execution result is determined. The error type includes a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage. Therefore, this technical solution can clearly distinguish whether the error type of the task execution result is an error in the first operation instruction or an incorrect operation performed on the webpage when an error is detected in the task execution result of a webpage operation task, thereby optimizing the execution process of the automated webpage task in a targeted manner. Furthermore, determining the error type of the task execution result through the first operation instruction and the webpage operation interface ensures that the error type discrimination does not rely on human subjective experience but is based on objective visual information and operation instructions, thus ensuring the accuracy of the error type. Furthermore, when the error type of the task execution result is determined to be a first operation instruction error, the correct operation instruction can be regenerated, enabling the model to reoperate the webpage based on the correct operation instruction, thereby obtaining the correct task execution result. This achieves the effect of automatic repair after the execution error of the webpage automation task and optimizes the execution process of the webpage automation task.

[0096] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a detection device for web page automation tasks.

[0097] Figure 5 A block diagram of a web page automation task detection device provided in an embodiment of this application is shown, such as... Figure 5 As shown, the detection device for automated web page tasks includes:

[0098] The acquisition module 51 is used to acquire the execution information of the webpage operation task; the execution information includes: a first operation instruction for indicating the operation of the webpage and the webpage operation interface;

[0099] The detection module 52 is used to detect whether the task execution result of the web operation task is correct based on the first operation instruction and the web operation interface;

[0100] The determination module 53 is used to determine the error type of the task execution result when the task execution result is incorrect; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage.

[0101] In some embodiments, the execution information further includes: a historical record of operations performed on the webpage;

[0102] When the detection module 52 detects whether the task execution result of the web operation task is correct according to the first operation instruction and the web operation interface, it performs the following steps:

[0103] Based on the first operation instruction and the historical operation record, determine the expected execution result of the webpage operation task;

[0104] Detect whether the task execution result matches the expected execution result;

[0105] If the task execution result does not match the expected execution result, the task execution result is determined to be incorrect.

[0106] In some embodiments, the webpage operation interface includes at least one of the following: a first interface before operating the webpage, a second interface after operating the webpage, and a third interface including operation information for operating the webpage; the operation information includes operation location information and / or operation content.

[0107] In some embodiments, when determining the error type of the task execution result, the determining module 53 performs the following steps:

[0108] If the first condition is met, the error type is determined to be the first type; the first condition includes: the first operation instruction does not match the interface content of the first interface;

[0109] If the second condition is met, the error type is determined to be the second type; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, and the operation information on the third interface does not match the first operation instruction.

[0110] In some embodiments, the apparatus further includes:

[0111] The generation module is configured to, after determining the error type of the task execution result, regenerate the operation instructions if the error type is determined to be the first type, to instruct the webpage to perform an operation that matches the expected execution result.

[0112] In some embodiments, the generation module performs the following steps when regenerating the operation instructions:

[0113] Detect whether the first interface element corresponding to the first operation instruction is hidden on the first interface;

[0114] If the first interface element is hidden on the first interface, a second operation instruction is generated; the second operation instruction is used to instruct the first interface element to be displayed and to perform operations on the first interface element.

[0115] If the first interface element is not hidden on the first interface, then the second interface element related to the first operation instruction on the first interface is determined, and a third operation instruction is generated based on the second interface element; the third operation instruction is used to instruct to perform an operation on the second interface element.

[0116] In some embodiments, the apparatus further includes:

[0117] The screenshot module is used to take a screenshot of the webpage in response to receiving the first operation instruction before obtaining the execution information of the webpage operation task, so as to obtain the first interface;

[0118] The second detection module is used to detect whether the interface content of the webpage has changed;

[0119] The screenshot module is also used to take a screenshot of the changed webpage when the interface content is detected to have changed, so as to obtain the second interface and the third interface.

[0120] The apparatus according to an embodiment of this application acquires execution information of a webpage operation task. The execution information includes a first operation instruction for instructing an operation on the webpage and a webpage operation interface. Based on the first operation instruction and the webpage operation interface, it detects whether the task execution result of the webpage operation task is correct. If the task execution result is incorrect, it determines the error type of the task execution result. The error type includes a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage. Therefore, this apparatus can clearly distinguish whether the error type of the task execution result is an error in the first operation instruction or an incorrect operation performed on the webpage when an error is detected in the task execution result of a webpage operation task, thereby optimizing the execution flow of the automated webpage task in a targeted manner. Furthermore, by determining the error type of the task execution result through the first operation instruction and the webpage operation interface, the error type discrimination does not rely on human subjective experience but is based on objective visual information and operation instructions, ensuring the accuracy of the error type.

[0121] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0122] Figure 6 This is a block diagram for implementing the electronic device provided in the embodiments of this application. Figure 6As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0123] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0124] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0125] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0126] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0127] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0128] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0129] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0130] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0134] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0135] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0136] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0138] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting automated web page tasks, characterized in that, include: Obtain execution information for webpage operation tasks; The execution information includes: a first operation instruction for instructing the operation of the webpage and a webpage operation interface; Based on the first operation instruction and the webpage operation interface, detect whether the task execution result of the webpage operation task is correct; In the event of an error in the task execution result, the error type of the task execution result is determined; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an error in the operation performed on the webpage; If the error type is determined to be the first type, the operation instruction is regenerated to instruct the webpage to perform an operation that matches the expected execution result of the webpage operation task; the expected execution result is determined based on the first operation instruction and the historical operation record of the webpage operation, and the expected execution result includes at least one of the following: expected operation location, expected operation effect, and expected operation content; Determining the error type of the task execution result includes: determining the error type as the first type when a first condition is met; the first condition includes: the first operation instruction does not match the interface content of the first interface before the operation on the webpage; The regeneration of the operation instruction includes: detecting whether a first interface element corresponding to the first operation instruction is hidden on the first interface; if the first interface element is hidden on the first interface, generating a second operation instruction; the second operation instruction is used to instruct the display of the first interface element and to perform an operation on the first interface element; if the first interface element is not hidden on the first interface, determining a second interface element on the first interface related to the first operation instruction, and generating a third operation instruction based on the second interface element; the third operation instruction is used to instruct the operation on the second interface element. The operation performed on the webpage that matches the expected execution result of the webpage operation task includes at least one of the following: the operation position on the webpage is the same as the expected operation position, the operation content on the webpage is the same as the expected operation content, and the operation effect on the webpage is the same as the expected operation effect.

2. The method according to claim 1, characterized in that, The execution information also includes: historical operation records of operations performed on the webpage; The step of detecting whether the task execution result of the web operation task is correct based on the first operation instruction and the web operation interface includes: Based on the first operation instruction and the historical operation record, determine the expected execution result of the webpage operation task; Detect whether the task execution result matches the expected execution result; If the task execution result does not match the expected execution result, the task execution result is determined to be incorrect.

3. The method according to claim 2, characterized in that, The webpage operation interface includes at least one of the following: a first interface before operating the webpage, a second interface after operating the webpage, and a third interface including operation information for operating the webpage; the operation information includes operation location information and / or operation content.

4. The method according to claim 3, characterized in that, The method for determining the error type of the task execution result includes: If the second condition is met, the error type is determined to be the second type; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, and the operation information on the third interface does not match the first operation instruction.

5. The method according to claim 3, characterized in that, Before obtaining the execution information of the webpage operation task, the method further includes: In response to receiving the first operation instruction, a screenshot of the webpage is taken to obtain the first interface; Detect whether the content of the webpage has changed; If a change in the interface content is detected, a screenshot of the changed webpage is taken to obtain the second interface and the third interface.

6. A detection device for automated web page tasks, characterized in that, include: The acquisition module is used to obtain execution information of web page operation tasks; The execution information includes: a first operation instruction for instructing the operation of the webpage and a webpage operation interface; The detection module is used to detect whether the task execution result of the web operation task is correct based on the first operation instruction and the web operation interface; The determination module is used to determine the error type of the task execution result when the task execution result is incorrect; the error type includes: a first type indicating an error in the first operation instruction, or a second type indicating an incorrect operation performed on the webpage; If the error type is determined to be the first type, the operation instruction is regenerated to instruct the webpage to perform an operation that matches the expected execution result of the webpage operation task; the expected execution result is determined based on the first operation instruction and the historical operation record of the webpage operation, and the expected execution result includes at least one of the following: expected operation location, expected operation effect, and expected operation content; The determining module is configured to determine the error type as the first type when a first condition is met; the first condition includes: the first operation instruction does not match the interface content of the first interface before the operation on the webpage; The regeneration of the operation instruction includes: detecting whether a first interface element corresponding to the first operation instruction is hidden on the first interface; if the first interface element is hidden on the first interface, generating a second operation instruction; the second operation instruction is used to instruct the display of the first interface element and to perform an operation on the first interface element; if the first interface element is not hidden on the first interface, determining a second interface element on the first interface related to the first operation instruction, and generating a third operation instruction based on the second interface element; the third operation instruction is used to instruct the operation on the second interface element. The operation performed on the webpage that matches the expected execution result of the webpage operation task includes at least one of the following: the operation position on the webpage is the same as the expected operation position, the operation content on the webpage is the same as the expected operation content, and the operation effect on the webpage is the same as the expected operation effect.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images