Webpage automation task detection method and device and electronic equipment

Abstract

The embodiment of the invention provides a webpage automation task detection method and device and electronic equipment, and relates to the technical field of webpage processing, and the webpage automation task detection method comprises the following steps: obtaining execution information of a webpage operation task; the execution information comprises a first operation instruction and a webpage operation interface, wherein the first operation instruction is used for indicating to operate a webpage; detecting whether a task execution result of the webpage operation task is correct or not according to the first operation instruction and the webpage operation interface; under the condition that the task execution result is wrong, determining the error type of the task execution result; the error type comprises a first type used for representing that the first operation instruction is wrong or a second type used for representing that wrong operation is executed on the webpage. According to the method, when the execution result of the webpage automation task is detected to be wrong, the error type of the execution result can be accurately analyzed, so that the execution process of the webpage automation task is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of webpage processing, and in particular to a webpage automation task detection method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of artificial intelligence and webpage automation technology, it has become a trend to automatically execute webpage operation tasks through large models. Taking a VLA (Visual-Language-Action) model as an example, the VLA model can understand natural language operation instructions and interact with the webpage environment. After the user inputs an operation instruction by voice, the VLA model identifies the operation instruction and executes an action corresponding to the operation instruction on the webpage, thereby promoting the progress of technologies such as automated operation and maintenance, automated testing, and data collection.

[0003] In the webpage automation process, accurately determining whether the operation is successful or not and accurately attributing the error type directly affects the closed loop, self-healing, and human-machine collaboration efficiency of the webpage automation task, which puts higher requirements on the discrimination and error correction capabilities of the model. In related technologies, only the success or failure of webpage operation can be simply determined, and detailed analysis cannot be made for the failure reason or failure type, thereby resulting in low repair efficiency of the webpage automation task. SUMMARY

[0004] Embodiments of the present application provide a webpage automation task detection method, device and electronic equipment to alleviate or solve one or more technical problems in the prior art.

[0005] In a first aspect, embodiments of the present application provide a webpage automation task detection method, comprising: obtaining execution information of a webpage operation task; the execution information comprising a first operation instruction for indicating an operation on a webpage and a webpage operation interface; detecting whether a task execution result of the webpage operation task is correct according to the first operation instruction and the webpage operation interface; in the case where the task execution result is incorrect, determining an error type of the task execution result; the error type comprising a first type for indicating that the first operation instruction is incorrect, or a second type for indicating that an incorrect operation is performed on the webpage.

[0006] In a second aspect, embodiments of the present application provide a webpage automation task detection device, comprising: an obtaining module configured to obtain execution information of a webpage operation task; the execution information comprising a first operation instruction for indicating an operation on a webpage and a webpage operation interface; detecting whether a task execution result of the web operation task is correct according to the first operation instruction and the web operation interface; determining a type of error of the task execution result in a case where the task execution result is incorrect; the type of error comprises a first type indicating that the first operation instruction is incorrect, or a second type indicating that an incorrect operation is performed on the web page.

[0007] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor implements the method of any of the embodiments of the present application when executing the computer program.

[0008] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0009] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0010] According to the technical solution of the embodiments of the present application, the execution information of the web operation task is obtained, and the execution information includes the first operation instruction for indicating the operation on the web page and the web operation interface. Whether the task execution result of the web operation task is correct is detected according to the first operation instruction and the web operation interface. In a case where the task execution result is incorrect, the type of error of the task execution result is determined. The type of error comprises the first type indicating that the first operation instruction is incorrect, or the second type indicating that the incorrect operation is performed on the web page. It can be seen that when the task execution result of the web operation task is detected to be incorrect, the type of error of the task execution result is explicitly distinguished as the first operation instruction error or the incorrect operation performed on the web page, so that the execution process of the web automation task is optimized in a targeted manner. Moreover, the type of error of the task execution result is determined by the first operation instruction and the web operation interface, so that the determination of the type of error does not depend on the subjective experience of a person, but is based on the objective visual information and the operation instruction, thereby ensuring the accuracy of the type of error.

[0011] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the embodiments can be implemented in accordance with the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0012] In the drawings, like reference numerals refer to same or similar components throughout the several views. The drawings are not necessarily to scale. It should be understood that the drawings are merely illustrative of certain embodiments of the application and that they are not to be used to limit the scope of the application.

[0013] Figure 1 A flow chart of a method for detecting a web automation task is shown; Figure 2 A web operation interface diagram of the method for detecting a web automation task is shown; Figure 3 Another web operation interface diagram of the method for detecting a web automation task is shown; Figure 4 A flow chart of a method for detecting a web automation task is shown; Figure 5 A block diagram of a device for detecting a web automation task is shown; Figure 6 A block diagram of an electronic device is shown. DETAILED DESCRIPTION

[0014] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0015] To facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way, and all of them belong to the protection scope of the embodiments of the present application.

[0016] The technical solutions of the present application and how the technical solutions of the present application solve the foregoing technical problems will be described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments of the present application will be described in detail below with reference to the drawings.

[0017] Figure 1 A flow chart of a method for detecting a web automation task is shown; Figure 1 As shown in the flow chart, the method can include steps S101, S102 and S103.

[0018] In step S101, execution information of the webpage operation task is acquired, and the execution information includes: a first operation instruction for indicating operation on the webpage, and a webpage operation interface.

[0019] Optionally, the execution information further includes a historical operation record of operation on the webpage. The historical operation record refers to a record of operation on the webpage based on the first operation instruction after the first operation instruction is received. The record content of the historical operation record is not limited, and can include at least one of the following information: an operation type of the operation performed on the webpage, an operation position, operation content, and the like. For example, the operation type is a click operation, a selection operation, an input operation, and the like; the operation position is an interaction position with the webpage; and the operation content includes input content on the webpage.

[0020] Before step S101 is performed, the webpage operation task is first performed. The execution manner of the webpage operation task includes the following steps: receiving a first operation instruction of the webpage operation task; and performing corresponding operation on the webpage based on the first operation instruction. During the execution of the webpage operation task, or after the execution of the webpage operation task, the execution information of the webpage operation task can be stored.

[0021] The webpage operation interface includes at least one of the following: a first interface before operation on the webpage, a second interface after operation on the webpage, and a third interface including operation information of operation on the webpage. The operation information includes operation position information and / or operation content.

[0022] The webpage operation interface can be obtained through screenshot. Optionally, in response to receiving the first operation instruction, the webpage is screenshot to obtain the first interface. Then, it is detected whether the interface content of the webpage changes, and in the case that the interface content changes, the changed webpage is screenshot to obtain the second interface and the third interface.

[0023] If the operation information of operation on the webpage includes an operation position, when the webpage is screenshot, the operation position can be marked on the screenshot, and the marking form is not limited, for example, the operation position can be marked with a red dot, or the operation position can be marked with a line frame; and the like. In this way, the webpage operation interface obtained has the related mark of the operation position, which facilitates detection of whether the task execution result is correct.

[0024] The execution time of step S101 is not limited in the present application, and step S101 can be performed immediately after the execution of the webpage operation task ends, or step S101 can be performed after a preset time period after the execution of the webpage operation task ends, or step S101 can be performed according to a preset period. Before step S101 is performed, the execution information of the webpage operation task can be stored in advance.

[0025] Optionally, the user inputs voice information, and voice recognition is performed on the voice information input by the user, and then a first operation instruction is generated according to a voice recognition result. For example, after the user inputs voice information “select ‘XX’ site”, voice recognition is performed on the voice information “select ‘XX’ site”, and then the first operation instruction “select ‘XX’ site” is generated.

[0026] In step S102, whether a task execution result of the web operation task is correct is detected according to the first operation instruction and the web operation interface.

[0027] Optionally, step S102 can be implemented as the following steps: First, an expected execution result of the web operation task is determined according to the first operation instruction and the historical operation record. Optionally, the expected execution result can also be determined in combination with the web operation interface. The expected execution result is a correct task execution result of the web operation task, and can include at least one of the following information: an expected operation position, an expected operation effect, and an expected operation content.

[0028] Second, whether the task execution result and the expected execution result match is detected. In a case where the task execution result and the expected execution result do not match, it is determined that the task execution result is incorrect; and in a case where the task execution result and the expected execution result match, it is determined that the task execution result is correct.

[0029] The case where the task execution result and the expected execution result do not match can include at least one of the following: an operation position on the web page and the expected operation position are different, an operation content on the web page (such as input content in an input box) and the expected operation content are different, and an operation effect on the web page and the expected operation effect are different.

[0030] For example, according to an operation position marker on the web operation interface, the operation position on the web page can be determined, and whether the operation position on the web page and the expected operation position are consistent is detected. If not, it is determined that the operation position on the web page and the expected operation position are different. For another example, the first operation instruction indicates that search content is input in an input box on the web page, and the web operation interface includes a screenshot of inputting the search content. According to the screenshot, the input content on the web page can be determined, and whether the input content and the expected input content (i.e., the expected operation content) are consistent is detected. If not, it is determined that the operation content on the web page and the expected operation content are different. For another example, according to the web operation interface, the interface content after the operation on the web page, i.e., the operation effect, can be determined, and whether the operation effect and the expected operation effect are consistent is detected. If not, it is determined that the operation effect on the web page and the expected operation effect are different.

[0031] Step S103 : if the task execution result is an error, determine the error type of the task execution result; the error type includes: a first type for indicating an error in the first operation instruction, or a second type for indicating an error in the operation performed on the webpage.

[0032] The first type indicates that the first operation instruction is wrong, that is, an erroneous first operation instruction is generated. Figure 2 As shown, the first operation instruction is "click the button 'Query'", but the display interface 20 of the web page does not include the button "Query", but includes the button 21 "Search", which causes the first operation instruction to be unable to be executed correctly, indicating that an erroneous first operation instruction is generated.

[0033] The second type indicates that a correct first operation instruction is generated, but an incorrect operation is performed on the web page based on the first operation instruction. Figure 3 As shown, the first operation instruction is "search for 'GUI virtual environment' on the homepage of the webpage", however, the task execution result is to enter "artificial intelligence model" in the input box 31 of the homepage 30 of the webpage for search, indicating that an incorrect operation was performed on the webpage.

[0034] In this embodiment, the web automation task detection method and the web automation task can be executed by the same pre-configured artificial intelligence model, or they can be executed separately by different artificial intelligence models. For example, if the web automation task is executed by a pre-configured VLA model, the web automation task detection method can be executed by the same VLA model, or it can be executed by another pre-configured model.

[0035] According to the technical solution of the embodiment of the present application, the execution information of the web page operation task is obtained, and the execution information includes a first operation instruction and a web page operation interface for indicating the operation on the web page; based on the first operation instruction and the web page operation interface, it is detected whether the task execution result of the web page operation task is correct. In the case of an error in the task execution result, the error type of the task execution result is determined, and the error type includes: a first type for indicating an error in the first operation instruction, or a second type for indicating an erroneous operation performed on the web page. It can be seen that this technical solution can clearly distinguish the error type of the task execution result as a first operation instruction error or an erroneous operation performed on the web page when detecting an error in the task execution result of the web page operation task, thereby optimizing the execution process of the web page automation task in a targeted manner. In addition, the error type of the task execution result is determined by the first operation instruction and the web page operation interface, so that the judgment of the error type does not rely on people's subjective experience, but is based on objective visual information and operation instructions, thereby ensuring the accuracy of the error type.

[0036] In some embodiments, determining the error type of the task execution result may be performed by performing the following steps A1 and A2: Step A1, in the case of meeting a first condition, determining the error type as a first type for indicating a first operation instruction error; the first condition includes that the first operation instruction does not match the interface content of the first interface.

[0037] For example, the first operation instruction corresponds to the operation element of "select category", but the first interface does not contain the operation element of "select category", i.e., the first interface content does not contain the operation element corresponding to the first operation instruction, so it can be determined that the first operation instruction does not match the interface content of the first interface.

[0038] For example, the first operation instruction corresponds to the operation element of "select category", but the first interface does not contain the operation element of "select category", i.e., the first interface content does not contain the operation element corresponding to the first operation instruction, so it can be determined that the first operation instruction does not match the interface content of the first interface.

[0039] For example, the first operation instruction corresponds to the operation element of "select category", but the first interface does not contain the operation element of "select category", i.e., the first interface content does not contain the operation element corresponding to the first operation instruction, so it can be determined that the first operation instruction does not match the interface content of the first interface.

[0040] Step A2, in the case of meeting a second condition, determining the error type as a second type for indicating an error operation on the webpage; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, and the operation information on the third interface does not match the first operation instruction.

[0041] For example, the first operation instruction corresponds to the operation element of "select category", but the first interface does not contain the operation element of "select category", i.e., the first interface content does not contain the operation element corresponding to the first operation instruction, so it can be determined that the first operation instruction does not match the interface content of the first interface.

[0042] The operation information includes operation position information and / or operation content. The operation position information can be indicated by an operation position marker on the interface. The operation information on the third interface not matching the first operation instruction can include that a position indicated by the operation position marker on the third interface is different from an operation position indicated by the first operation instruction, or the operation content on the third interface is different from operation content indicated by the first operation instruction.

[0043] For example, the first operation instruction is “input ‘XX’ in the first input box”, but the third interface shows that “XX” is input in the second input box, i.e., the operation position marker is at the second input box, which indicates that the operation position is incorrect when the webpage is operated, and thus it can be determined that the operation information on the third interface does not match the first operation instruction.

[0044] For another example, the first operation instruction is “input ‘XX’ in the first input box”, but the third interface shows that “YY” is input in the first input box, which indicates that the operation position is correct when the webpage is operated, but the operation content is incorrect, and thus it can be determined that the operation information on the third interface does not match the first operation instruction.

[0045] In this embodiment, by analyzing the webpage operation screenshot, the first operation instruction and the historical operation record, the error type of the task execution result can be objectively determined based on the actual visual interface and the operation record, and does not depend on the subjective judgment of people, and thus the accuracy of the error type can be ensured.

[0046] In some embodiments, in a case where it is determined that the error type of the task execution result is the first type, the operation instruction is regenerated to indicate performing an operation on the webpage that matches the expected execution result.

[0047] The operation instruction can be regenerated by performing the following steps B1, B2 and B3: Step B1, detecting whether the first interface element corresponding to the first operation instruction is hidden on the first interface.

[0048] Optionally, when step B1 is performed, a configuration file of the webpage can be queried to determine whether the first interface element corresponding to the first operation instruction is included in the configuration file; if yes, it is determined that the first interface element corresponding to the first operation instruction is hidden on the first interface.

[0049] Step B2, if the first interface element is hidden on the first interface, a second operation instruction is generated; the second operation instruction is used to indicate displaying the first interface element and performing an operation on the first interface element.

[0050] Suppose the first operation instruction is "select the date option 'March 2022'", the first interface element corresponding to the first operation instruction is the date "March 2022". After determining that the first operation instruction is incorrect, by querying the configuration file of the webpage, it is determined that the first interface element is in a hidden state, and the first interface element is configured to be displayed by operating the scroll bar. Then, the following example of the second operation instruction can be generated: "operate the scroll bar to view more dates, and select the date option 'March 2022'".

[0051] In step B3, if the first interface element is not hidden on the first interface, a second interface element related to the first operation instruction on the first interface is determined, and a third operation instruction is generated according to the second interface element; the third operation instruction is used to instruct the operation of the second interface element.

[0052] In step B3, if the first interface element is not hidden on the first interface, a second interface element related to the first operation instruction on the first interface is determined, and a third operation instruction is generated according to the second interface element; the third operation instruction is used to instruct the operation of the second interface element.

[0053] Suppose the first operation instruction is "click the button 'Search'", the first interface element corresponding to the first operation instruction is the button "Search". After determining that the first operation instruction is incorrect, by querying the configuration file, it is determined that the button "Search" is not hidden on the first interface, and it is determined that the first interface contains a second interface element related to the first operation instruction, that is, the button "Search", then the following example of the third operation instruction can be generated: "click the button 'Search'".

[0054] In this embodiment, when the error type of the task execution result is determined to be the first operation instruction error, the correct operation instruction can be regenerated based on the first operation instruction, so that the model can re-operate the webpage based on the correct operation instruction, thereby obtaining the correct task execution result, and the automatic repair effect after the webpage automation task execution error is realized, and the execution process of the webpage automation task is optimized.

[0055] In some embodiments, after step S103 is performed, the error information of the task execution result is output. The error information can include at least one of the following: whether it is incorrect, the error reason, the error type, and the regenerated operation instruction. By outputting the error information, the detection result of the webpage operation task is standardized, so that the process of the webpage operation task is optimized.

[0056] The output form of the detection result can be predefined, for example, for the case that the task execution result is a first type of error, the output form is predefined as follows: {"result": false, "reason": "error reason description (not more than 20 words)", "plan": "new control instruction (single-step instruction, not more than 15 words)", "type": "plan"}. For the case that the task execution result is a second type of error, the output form is predefined as follows: {"result": false, "reason": "error reason description (not more than 20 words)", "type": "action"}. For the case that the task execution result is correct, the output form is predefined as follows: {"result": true}.

[0057] For example, the first operation instruction is "select 'March 2022'", and since the display interface of the webpage does not contain the option 'March 2022', the first operation instruction cannot be executed correctly, so it can be determined that the error type is the first type, and the error reason is "the interface does not have the March 2022 option, the earliest can only select to April 2022". Therefore, the following error information can be output: {"result": false, "reason": "the interface does not have the March 2022 option, the earliest can only select to April 2022". "plan": "select the earliest available month", "type": "plan"}. Among them, "result" represents the detection result, "false" represents that the detection result is an error, "reason" represents the error reason. In "plan": "select the earliest available month", "plan" represents the regenerated operation instruction, that is, "select the earliest available month". In "type": "plan", "type" represents the error type, and "plan" represents that the error type is the instruction error type, that is, the first type.

[0058] In addition, if it is detected that the task execution result is correct, the task execution correct information can be output, for example, the following form of information can be output: {"result": true}. "true" represents that the detection result is correct.

[0059] It should be noted that the output forms listed above are only examples and do not limit the output of the detection results (including error information or correct information) in the present application.

[0060] The following is an example of an automatic driving system in a GUI (Graphical User Interface) virtual environment to illustrate the specific implementation steps of the webpage automation task detection method provided by the present application. In the GUI virtual environment automatic driving system, the page is a virtual interface.

[0061] The VLA model and the detection model are pre-configured in the virtual environment automatic driving system. The VLA model is a multi-modal artificial intelligence model that combines visual perception, language understanding, and action control, aiming to enable machines (such as robots, autonomous vehicles) to understand the environment through natural language instructions and perform physical actions. In the automatic driving system, the VLA model receives user input voice information, recognizes the voice information, generates a first operation instruction, and executes a webpage operation task based on the first operation instruction. The VLA model includes a planning (plan) sub-model and an execution (action) sub-model. The planning sub-model is mainly used to plan the execution instructions of the webpage operation task, including generating the first operation instruction. The execution sub-model is used to execute the corresponding operation on the virtual interface according to the first operation instruction generated by the planning sub-model, to obtain the task execution result of the webpage operation task. The detection model is used to detect the task execution result output by the VLA model to detect whether the task execution result is correct, and to determine the error type of the task execution result when the task execution result is incorrect.

[0062] Figure 4 The flowchart of the webpage automation task detection method provided by the embodiments of the present application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the method can include steps S401 to S409.

[0063] In step S401, the planning sub-model performs voice recognition on the user input voice information and generates a first operation instruction based on the recognition result.

[0064] In step S402, the execution sub-model performs operations on the webpage based on the first operation instruction to obtain the task execution result of the webpage operation task; takes screenshots of the webpage during the webpage operation process to obtain the webpage operation interface; and records the webpage operation process to obtain the historical operation record.

[0065] The webpage operation interface includes a first interface before the webpage operation, a second interface after the webpage operation, and a third interface including operation information of the webpage operation. The operation information includes operation position information and / or operation content.

[0066] The record content of the historical operation record is not limited, and can include at least one of the following information: operation type of the operation performed on the webpage, operation position, operation content, and the like. For example, the operation type is a click operation, a selection operation, an input operation, and the like; the operation position is the interaction position of the webpage; and the operation content includes the input content on the webpage.

[0067] In step S403, the detection model obtains execution information of the webpage operation task, and the execution information includes the first operation instruction, the webpage operation interface, and the historical operation record.

[0068] The execution subjects of steps S404 to S409 below are all the detection model, and thus the execution subject will not be repeated.

[0069] In step S404, the expected execution result of the webpage operation task is determined according to the first operation instruction and the historical operation record.

[0070] The expected execution result is the correct task execution result of the webpage operation task.

[0071] In step S405, the task execution result is determined to be incorrect in a case where the task execution result and the expected execution result do not match.

[0072] In step S406, the error type of the task execution result is determined.

[0073] If the error type is a first type for indicating that the first operation instruction is incorrect, step S407 is performed; or if the error type is a second type for indicating that an incorrect operation is performed on the webpage, step S409 is performed.

[0074] In step S407, the operation instruction is regenerated according to the interface content on the webpage and the first operation instruction.

[0075] In step S408, error information of the task execution result is output, and the error information includes the error type, the error reason, and the regenerated operation instruction.

[0076] Optionally, in a case where the error information includes the regenerated operation instruction, the error information can be sent to the execution sub-model, and the execution sub-model performs operation on the webpage again according to the regenerated operation instruction, so as to obtain the correct task execution result.

[0077] In step S409, error information of the task execution result is output, and the error information includes the error type and the error reason.

[0078] For example, the user inputs voice information "drive to X cafe", the planning sub-model analyzes the user input voice information, and generates the following first operation instruction: drive along route A on the virtual interface to point B. The execution sub-model performs the first operation based on the first operation instruction, and performs corresponding operations on the virtual interface. In the automatic driving scene of the virtual environment, it is specifically manifested as: the execution sub-model controls the automatic driving vehicle to drive along route A, that is, controls the position of the automatic driving vehicle on the virtual interface to change. During the driving process, the execution sub-model can take screenshots of the virtual interface, and optionally, take a screenshot every predetermined time interval, thereby obtaining virtual interface screenshots at multiple time nodes during the driving process. After the driving task is executed this time, the detection model obtains the execution information, which includes the first operation instruction "drive to X cafe", the virtual interface screenshots, and the driving record (i.e. the historical operation record). The driving record represents the position information of the automatic driving vehicle at different times on the virtual interface, and / or the driving trajectory of the automatic driving vehicle on the virtual interface. The detection model can determine that the expected execution result of the first operation instruction is that the automatic driving vehicle drives along route A to point B according to the execution information.

[0079] The detection model compares the actual task execution result with the expected execution result. If there is inconsistency, it can be determined that the task execution result is incorrect. Then, if the detection model detects that the virtual interface does not contain route A, it can be determined that the error type of the task execution result is the first type for indicating that the first operation instruction is incorrect. In this case, the detection model can regenerate the operation instruction. Assuming that the route from the current position to point B on the virtual interface is route C, the detection model can generate the following new operation instruction: drive along route C on the virtual interface to point B. If the detection model detects that the virtual interface contains route A, but the automatic driving vehicle drives route D, or the end position of the automatic driving vehicle is not point B, it can be determined that the error type of the task execution result is the second type for indicating that the operation on the webpage is incorrect.

[0080] According to the technical solution of the embodiment of the present application, the execution information of the webpage operation task is acquired, and the execution information includes: the first operation instruction for indicating the operation on the webpage and the webpage operation interface; whether the task execution result of the webpage operation task is correct is detected according to the first operation instruction and the webpage operation interface. In the case that the task execution result is incorrect, the error type of the task execution result is determined, and the error type includes: the first type for indicating the error of the first operation instruction or the second type for indicating the error operation on the webpage. It can be seen that when the task execution result of the webpage operation task is detected to be incorrect, the error type of the task execution result is explicitly distinguished as the error of the first operation instruction or the error operation on the webpage, so that the execution process of the webpage automation task is targetedly optimized. Moreover, the error type of the task execution result is determined through the first operation instruction and the webpage operation interface, so that the judgment of the error type does not depend on the subjective experience of people, but is based on the objective visual information and the operation instruction, ensuring the accuracy of the error type. In addition, when the error type of the task execution result is determined to be the error of the first operation instruction, the correct operation instruction can be regenerated, so that the model can re-operate the webpage based on the correct operation instruction, thereby obtaining the correct task execution result, realizing the automatic repair effect after the execution error of the webpage automation task, and optimizing the execution process of the webpage automation task.

[0081] Corresponding to the application scenario of the method and the method provided in the embodiments of the present application, the embodiments of the present application further provide a webpage automation task detection device.

[0082] Figure 5 A block diagram of the webpage automation task detection device provided in the embodiments of the present application is shown as Figure 5 The webpage automation task detection device includes: The acquisition module 51 is configured to acquire execution information of a webpage operation task, and the execution information includes: a first operation instruction for indicating an operation on a webpage and a webpage operation interface. The detection module 52 is configured to detect whether a task execution result of the webpage operation task is correct according to the first operation instruction and the webpage operation interface. The determination module 53 is configured to determine an error type of the task execution result in the case that the task execution result is incorrect, and the error type includes: a first type for indicating the error of the first operation instruction or a second type for indicating the error operation on the webpage.

[0083] In some embodiments, the execution information further includes: a historical operation record of the operation on the webpage. The detection module 52 performs the following steps when detecting whether the task execution result of the web operation task is correct according to the first operation instruction and the web operation interface: According to the first operation instruction and the historical operation record, determining an expected execution result of the web operation task; Detecting whether the task execution result and the expected execution result match; In the case where the task execution result and the expected execution result do not match, determining that the task execution result is incorrect.

[0084] In some embodiments, the web operation interface includes at least one of the following: a first interface before the web page is operated, a second interface after the web page is operated, and a third interface including operation information of the operation on the web page; the operation information includes operation position information and / or operation content.

[0085] In some embodiments, the determination module 53 performs the following steps when determining the error type of the task execution result: In the case where a first condition is met, determining that the error type is the first type; the first condition includes that the first operation instruction does not match the interface content of the first interface; In the case where a second condition is met, determining that the error type is the second type; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, and the operation information on the third interface does not match the first operation instruction.

[0086] In some embodiments, the apparatus further includes: The generation module is configured to, after determining the error type of the task execution result, in the case where the error type is the first type, regenerate an operation instruction to indicate performing an operation on the web page that matches the expected execution result.

[0087] In some embodiments, the generation module performs the following steps when regenerating the operation instruction: Detecting whether a first interface element corresponding to the first operation instruction is hidden on the first interface; If the first interface element is hidden on the first interface, generating a second operation instruction; the second operation instruction is used to indicate displaying the first interface element and operating the first interface element; If the first interface element is not hidden on the first interface, a second interface element related to the first operation instruction on the first interface is determined, and a third operation instruction is generated according to the second interface element; the third operation instruction is used to instruct to operate the second interface element.

[0088] In some embodiments, the apparatus further includes: The screenshot module is configured to, before the execution information of the web page operation task is acquired, in response to receiving the first operation instruction, take a screenshot of the web page to obtain the first interface. The second detection module is configured to detect whether the interface content of the web page changes. The screenshot module is further configured to, in a case where it is detected that the interface content changes, take a screenshot of the changed web page to obtain the second interface and the third interface.

[0089] According to the apparatus of the embodiments of the present application, the execution information of the web page operation task is acquired, and the execution information includes a first operation instruction used to instruct to operate the web page and a web page operation interface. The task execution result of the web page operation task is detected according to the first operation instruction and the web page operation interface. In a case where the task execution result is incorrect, the error type of the task execution result is determined, and the error type includes a first type used to represent that the first operation instruction is incorrect or a second type used to represent that an incorrect operation is performed on the web page. It can be seen that, when the task execution result of the web page operation task is detected to be incorrect, the apparatus can clearly distinguish whether the error type of the task execution result is that the first operation instruction is incorrect or that an incorrect operation is performed on the web page, so as to optimize the execution process of the web page automation task in a targeted manner. Moreover, the error type of the task execution result is determined by using the first operation instruction and the web page operation interface, so that the error type is determined based on objective visual information and operation instructions, rather than depending on subjective experience of a person, and the accuracy of the error type is ensured.

[0090] The functions of each module in the apparatuses of the embodiments of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0091] Figure 6 A block diagram of an electronic device for implementing the embodiments of the present application is provided. As shown in the figure, the electronic device includes a memory 601 and a processor 602, and the memory 601 stores a computer program that can run on the processor 602. The processor 602 implements the method in the above embodiments when executing the computer program. The number of the memory 601 and the processor 602 can be one or more. In specific implementation, the electronic device can further include a communication interface 603 for communicating with external devices and transmitting data. Figure 6 ​

[0092] In a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are independently implemented, the memory 601, the processor 602 and the communication interface 603 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0093] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0094] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0095] The embodiment of the present application provides a computer program product, which includes a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0096] The embodiment of the present application further provides a chip, which includes a processor, is used for calling and running instructions stored in a memory from the memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0097] The embodiment of the present application further provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0098] It is to be understood that the above-described processor can be a Central Processing Unit (CPU), but can also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or the like. The general purpose processor can be a microprocessor or any conventional processor, or the like. It is to be appreciated that the processor can be an Advanced RISC Machines (ARM) architecture processor.

[0099] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory, for example. The volatile memory can include a Random Access Memory (RAM), which is used as an external cache. Many forms of RAM are available by way of example but not limitation. For example, a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Sync Link DRAM (SLDRAM), and a Direct Rambus RAM (DR RAM) can be used.

[0100] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0101] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0102] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0103] Any process or method described in the flowchart or otherwise described herein can be understood as a representation of code including one or more executable instructions for implementing specific logical functions or processes. Also, the scope of the preferred embodiments of the present disclosure includes additional implementations, in which the functions can be performed in an order other than that shown or discussed, including in a substantially simultaneous manner or in reverse order, according to the functions involved.

[0104] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer-readable medium for instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can take instructions from an instruction execution system, device or equipment and execute instructions) or in conjunction with these instructions execution system, device or equipment.

[0105] It should be understood that each part of the present application can be realized by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium and includes one or a combination of the steps of the embodiment methods when executed.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0107] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting web page automation tasks, characterized in that: include: Get the execution information of web page operation tasks; The execution information includes: a first operation instruction for instructing to operate the webpage and a webpage operation interface; detecting, based on the first operation instruction and the webpage operation interface, whether a task execution result of the webpage operation task is correct; In the case that the task execution result is erroneous, an error type of the task execution result is determined; the error type includes: a first type for indicating an error in the first operation instruction, or a second type for indicating an erroneous operation performed on the webpage.

2. The method according to claim 1, characterized in that The execution information also includes: historical operation records of operations on the web page; The detecting, based on the first operation instruction and the webpage operation interface, whether the task execution result of the webpage operation task is correct includes: determining an expected execution result of the webpage operation task according to the first operation instruction and the historical operation record; Detecting whether the task execution result matches the expected execution result; In a case where the task execution result does not match the expected execution result, it is determined that the task execution result is erroneous.

3. The method according to claim 2, characterized in that The web page operation interface includes at least one of the following: a first interface before operating the web page, a second interface after operating the web page, and a third interface including operation information of operating the web page; the operation information includes operation location information and / or operation content.

4. The method according to claim 3, characterized in that Determining the error type of the task execution result includes: If a first condition is met, determining that the error type is the first type; the first condition includes: the first operation instruction does not match the interface content of the first interface; When the second condition is met, the error type is determined to be the second type; the second condition includes at least one of the following: the interface content of the second interface does not match the expected execution result, and the operation information on the third interface does not match the first operation instruction.

5. The method according to claim 3, characterized in that After determining the error type of the task execution result, the method further includes: When it is determined that the error type is the first type, an operation instruction is regenerated to instruct execution of an operation matching the expected execution result on the webpage.

6. The method according to claim 5, characterized in that The regeneration operation instruction includes: Detecting whether a first interface element corresponding to the first operation instruction is hidden on the first interface; If the first interface element is hidden on the first interface, a second operation instruction is generated; the second operation instruction is used to instruct to display the first interface element and operate the first interface element; If the first interface element is not hidden on the first interface, a second interface element related to the first operation instruction on the first interface is determined, and a third operation instruction is generated based on the second interface element; the third operation instruction is used to instruct an operation on the second interface element.

7. The method according to claim 3, characterized in that Before obtaining the execution information of the webpage operation task, the method further includes: In response to receiving the first operation instruction, taking a screenshot of the webpage to obtain the first interface; Detecting whether the interface content of the webpage has changed; When a change in the interface content is detected, a screenshot is taken of the changed web page to obtain the second interface and the third interface.

8. A detection device for web page automation tasks, characterized in that: include: The acquisition module is used to obtain the execution information of the web page operation task; The execution information includes: a first operation instruction for instructing to operate the webpage and a webpage operation interface; a detection module, configured to detect whether a task execution result of the webpage operation task is correct based on the first operation instruction and the webpage operation interface; The determination module is used to determine the error type of the task execution result when the task execution result is erroneous; the error type includes: a first type for indicating an error in the first operation instruction, or a second type for indicating an erroneous operation performed on the webpage.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images