Automatic operation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an automatic operation method and device, equipment and a storage medium. The method comprises the following steps: in response to an instruction input operation, obtaining at least one natural language description instruction for a target application, determining element information of at least one interface element in the target application, and based on the element information of the at least one interface element and the at least one natural language description instruction, determining at least one natural language description instruction for the target application. A target automation instruction corresponding to each natural language description instruction is generated by utilizing a first artificial intelligence large model in combination with a knowledge base, and the knowledge base comprises element information of at least one interface element and interaction information; and sequentially executing at least one target automation instruction corresponding to the at least one natural language description instruction by utilizing the automation framework so as to realize automatic operation on the target application. According to the scheme, the target automation instruction is generated through the first artificial intelligence large model, the automation instruction development efficiency can be improved, and then the automation operation efficiency for the interface is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an automated operation method, device, equipment and storage medium. Background Art

[0002] RPA (Robotic Process Automation) technology can simulate manual execution of automated operations. Among them, RPA can be implemented by using an automation framework such as the Appium framework to execute fixed scripts. For example, the automation framework can be applied to the target application (APP, Application), and the automation framework can execute scripts to simulate manual interactive operations such as clicking and sliding on the interface of the target application. However, when the position, attributes or structure of the interface elements change, the fixed scripts often cannot adapt to these changes and lack flexibility, resulting in process interruption or execution failure. If developers are required to write automation instructions in real time to generate scripts to adapt to changes in the interface, they need to have a certain programming technology background, which has a high threshold, requires a lot of human resources, and has low efficiency. Summary of the invention

[0003] The embodiments of the present invention provide an automated operation method, device, equipment and storage medium to solve the problem of low efficiency of automated operation on an interface in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides an automated operation method, comprising:

[0005] In response to the instruction input operation, obtaining at least one natural language description instruction for the target application;

[0006] Determining element information of at least one interface element in the target application;

[0007] Based on the element information of the at least one interface element and the at least one natural language description instruction, a target automation instruction corresponding to each natural language description instruction is generated by using the first artificial intelligence big model in combination with a knowledge base; the knowledge base includes the element information and interaction information of the at least one interface element in the target application;

[0008] The automation framework is used to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement an automated operation on the target application.

[0009] In a second aspect, an embodiment of the present invention provides an automated operation device, the device comprising:

[0010] An acquisition module, configured to acquire at least one natural language description instruction for a target application in response to an instruction input operation;

[0011] A determination module, used to determine element information of at least one interface element in the target application;

[0012] A generation module, configured to generate a target automation instruction corresponding to each natural language description instruction by using the first artificial intelligence big model in combination with a knowledge base based on the element information of the at least one interface element and the at least one natural language description instruction; the knowledge base includes the element information and interaction information of the at least one interface element in the target application;

[0013] The execution module is used to use the automation framework to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement the automation operation on the target application.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the automated operation method in the first aspect. The electronic device may also include a communication interface for communicating with other devices or communication systems.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the automated operation method as described in the first aspect above.

[0016] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor can implement the automated operation method as described in the first aspect above.

[0017] In an embodiment of the present invention, in response to an instruction input operation, at least one natural language description instruction for a target application is obtained, element information of at least one interface element in the target application is determined, and based on the element information of at least one interface element and at least one natural language description instruction, a first artificial intelligence large model is combined with a knowledge base to generate a target automation instruction corresponding to each natural language description instruction, wherein the knowledge base includes the element information and interaction information of at least one interface element in the target application, and then, the automation framework is used to execute at least one target automation instruction corresponding to the at least one natural language description instruction in sequence to achieve automated operation on the target application.

[0018] By using the first artificial intelligence big model to generate the target automation instructions, it is possible to avoid manual writing by developers, reduce human resource consumption, improve the efficiency of automation instruction development, and thus improve the efficiency of automated operations on the interface. Among them, the first artificial intelligence big model can be combined with a knowledge base including element information of at least one interface element and interaction information, which helps the first artificial intelligence big model to better understand the interface information and improve the accuracy of the generated target automation instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of an embodiment of an automated operation method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the structure of an automated operating device provided by an embodiment of the present invention;

[0022] Figure 3 For Figure 2 A schematic diagram of the structure of an electronic device corresponding to the automated operation device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The technical solution of the embodiment of the present invention can be applied to the scenario of performing automated operations on the interface of a target application.

[0025] As described in the background technology, the automation framework can execute scripts to simulate manual interactive operations such as clicking and sliding on the interface of the target application. However, when the position, attributes or structure of the interface elements change, the fixed scripts often cannot adapt to these changes in real time, lack flexibility, and cause process interruption or execution failure. If developers are required to write automation instructions to generate scripts in real time to adapt to changes in the interface, they need to have a certain programming technology background, which has a high threshold, consumes a lot of human resources, and has low efficiency.

[0026] In order to solve the technical problem of low efficiency of automated operation on the interface, the inventors thought that the automated instructions can be generated in combination with the artificial intelligence big model to avoid manual writing by developers, improve the efficiency of automated instruction development, and further improve the efficiency of automated operation on the interface. Therefore, after a series of studies, the inventors proposed the technical solution of the present invention. In an embodiment of the present invention, in response to the instruction input operation, at least one natural language description instruction for the target application is obtained, the element information of at least one interface element in the target application is determined, and based on the element information of at least one interface element and at least one natural language description instruction, the first artificial intelligence big model is combined with the knowledge base to generate a target automated instruction corresponding to each natural language description instruction, wherein the knowledge base includes the element information and interaction information of at least one interface element in the target application, and then, the automation framework is used to sequentially execute at least one target automated instruction corresponding to at least one natural language description instruction to realize automated operation on the target application.

[0027] By using the first artificial intelligence big model to generate target automation instructions, developers can avoid manual writing, reduce human resource consumption, improve the efficiency of automation instruction development, and thus improve the efficiency of automated operations on the interface. Among them, the first artificial intelligence big model can be combined with a knowledge base including element information of at least one interface element and interaction information, which helps the first artificial intelligence big model to better understand the interface information and improve the accuracy of the generated target automation instructions. In addition, different natural language description instructions can indicate different interface interaction operations. The embodiment of the present invention can generate corresponding target automation instructions according to different natural language description instructions. Using the automation framework to execute different target automation instructions can achieve different interface interaction operations. Compared with the traditional automation framework that executes fixed scripts for automation operations, the flexibility of automation operations can be improved.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features and steps in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0030] The automated operation method provided in the embodiment of the present invention can be executed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, or a server. The server can be a physical server including an independent host, or a virtual server, or a server or server cluster in the cloud.

[0031] Figure 1 A flowchart of an embodiment of an automated operation method provided by an embodiment of the present invention, the method may include the following steps:

[0032] 101: In response to an instruction input operation, obtain at least one natural language description instruction for a target application.

[0033] The at least one natural language description instruction may be provided by a developer, such as by providing an instruction input interface, and in response to an instruction input operation by the developer on the instruction input interface, obtaining at least one natural language description instruction for the target application.

[0034] The natural language description instructions may refer to instructions for performing interactive operations described in a natural language.

[0035] The developer may provide a natural language description instruction text such as "open the APP, click the login button, search for the product 'mobile phone', and add the first search result to the shopping cart". The present invention may split a set of natural language description instructions contained in the natural language description instruction text into multiple independent natural language description instructions, namely "open the APP", "click the login button", "search for the product 'mobile phone'", and "add the first search result to the shopping cart", according to separators such as commas.

[0036] 102: Determine element information of at least one interface element in the target application.

[0037] The element information of the interface element may include the attribute information and hierarchical information of the interface element, etc. The attribute information of the interface element may include, for example, ID (Identification), type, size, and position, etc. The hierarchical information of the interface element may include, for example, the nesting relationship of the interface element, etc.

[0038] Optionally, determining element information of at least one interface element in the target application may include: using an automation framework to obtain element information of at least one interface element in the target application in an extensible markup language format; removing element information corresponding to interface elements that cannot interact; and parsing the element information into element information of a key-value pair structure.

[0039] The automation framework may connect to the target application to obtain element information of at least one interface element in the target application in an extensible markup language format. The extensible markup language may refer to XML or the like.

[0040] Non-interactive interface elements may include invisible interface elements and redundant elements. Invisible interface elements refer to hidden elements that cannot be seen by users on the interface. Redundant elements may refer to elements that have no actual function or meaning, such as meaningless container nodes.

[0041] Parsing the element information into the element information of the key-value pair structure may refer to parsing the element information into the element information in JSON format.

[0042] 103: Based on the element information of at least one interface element and at least one natural language description instruction, the first artificial intelligence big model is combined with the knowledge base to generate a target automation instruction corresponding to each natural language description instruction.

[0043] The first AI big model can be LLM (Large Language Model), which is a natural language processing (NLP) model based on deep learning, designed to understand and generate human language. LLM can learn the patterns and structures of language by training a large amount of text data, so as to perform a variety of tasks such as text generation, machine translation, question-answering system, etc.

[0044] Among them, a first prompt instruction can be generated based on the element information of at least one interface element, at least one natural language description instruction and an instruction generation template, and the first prompt instruction can be input into the first artificial intelligence big model to utilize the first artificial intelligence big model in combination with the knowledge base to generate a target automation instruction corresponding to each natural language description instruction. The specific implementation method can be described in detail below.

[0045] The knowledge base may include element information and interaction information of at least one interface element in the target application. The interaction information of the interface element may include information describing how the interface element responds to user interaction, such as the jump order and state transition rules of the interface element.

[0046] The knowledge base can be integrated into the first artificial intelligence model as additional reference information when the first artificial intelligence model generates answers. The first artificial intelligence model generates target automation instructions in combination with the knowledge base, which can be the first artificial intelligence model querying the knowledge base according to the first prompt instruction, and generating target automation instructions according to the relevant content retrieved from the knowledge base, which can improve the accuracy and completeness of the target automation instructions.

[0047] 104: Utilize the automation framework to sequentially execute at least one target automation instruction corresponding to at least one natural language description instruction to implement automated operations on the target application.

[0048] The automation framework may be Appium, Selenium, or Cypress, etc., which can be used to perform automation operations. The target automation instruction may be an RPA instruction, etc.

[0049] In the embodiment of the present invention, by using the first artificial intelligence big model to generate the target automation instructions, it is possible to avoid manual writing by developers, reduce human resource consumption, improve the efficiency of automation instruction development, and further improve the efficiency of automated operations on the interface. Among them, the first artificial intelligence big model can generate target automation instructions in combination with a knowledge base including element information of at least one interface element and interaction information, which helps the first artificial intelligence big model to better understand the interface interaction operations and improve the accuracy of the generated target automation instructions. In addition, different natural language description instructions can indicate different interface interaction operations. The embodiment of the present invention can generate corresponding target automation instructions according to different natural language description instructions. Using the automation framework to execute different target automation instructions can achieve different interface interaction operations. Compared with the traditional automation framework that executes fixed scripts for automation operations, the flexibility of automation operations can be improved.

[0050] In some embodiments, the method may further include: determining an execution order of at least one natural language description instruction; and determining, according to the execution order, a natural language description instruction currently to be executed and a natural language description instruction previous to the natural language description instruction currently to be executed.

[0051] As in the above example, the execution order of "opening the APP", "clicking the login button", "searching for the product 'mobile phone'", and "adding the first search result to the shopping cart" can be determined.

[0052] Based on the element information of at least one interface element and at least one natural language description instruction, using the first artificial intelligence big model in combination with the knowledge base, generating a target automation instruction corresponding to each natural language description instruction may include: obtaining the element information of at least one interface element in the current interface corresponding to the natural language description instruction currently to be executed; based on the element information of at least one interface element, the natural language description instruction currently to be executed and the previous natural language description instruction, using the first artificial intelligence big model in combination with the knowledge base, generating a target automation instruction corresponding to the natural language description instruction currently to be executed.

[0053] The current interface corresponding to the natural language description instruction to be executed can refer to the interface updated after the previous natural language description instruction to be executed. For example, if the natural language description instruction to be executed is "click the login button", the corresponding current interface is the interface updated after executing "open the APP", such as the login interface.

[0054] Using an automation framework to sequentially execute at least one target automation instruction corresponding to at least one natural language description instruction to implement automated operations on a target application may include: using the automation framework to execute the target automation instruction to perform the corresponding automated operations in the current interface of the target application to obtain an updated interface; returning to the execution order to determine the current natural language description instruction to be executed and the steps of the previous natural language description instruction to continue execution.

[0055] Among them, based on the element information of at least one interface element, the natural language description instruction currently to be executed and the previous natural language description instruction, using the first artificial intelligence big model in combination with the knowledge base to generate the target automation instruction corresponding to the natural language description instruction currently to be executed can include: based on the element information of at least one interface element, the natural language description instruction currently to be executed, the previous natural language description instruction and the instruction generation template to generate a first prompt instruction, input the first prompt instruction into the first artificial intelligence big model, so as to use the first artificial intelligence big model in combination with the knowledge base to generate the target automation instruction corresponding to the natural language description instruction currently to be executed.

[0056] For example, the natural language description instruction to be executed is "click the login button", and the previous natural language description instruction is "open the APP". The current interface corresponding to the natural language description instruction to be executed can be the login interface, that is, the interface updated after executing the target automation instruction corresponding to the previous natural language description instruction such as "open the APP". If the natural language description instruction to be executed is "open the APP", the previous natural language description instruction can be empty, and the current interface corresponding to the natural language description instruction to be executed can be empty.

[0057] For example, the instruction generation template could be:

[0058] Generate target automation instructions based on the following UI element information and instructions:

[0059] UI elements: AAAA

[0060] Current instruction: BBBB

[0061] Historical instructions: CCCC.

[0062] Please generate target automation instructions for DDDD.

[0063] Assume that the element information of at least one interface element in the login interface can be obtained as follows: [

[0065] {"id":"login_button","text":"Login","bounds":[100,200,200,300]},

[0066] {"id":"username_field","text":"Username","bounds":[50,400,300,450]},{"id":"password_field","text":"Password","bounds":[50,500,300,550]} ]

[0068] Fill in AAAA, fill in the current natural language description instruction to be executed as "click the login button" at BBBB, fill in the previous natural language description instruction as "open the APP" at CCCC, and fill in the information describing the interactive operation corresponding to the current natural language description instruction to be executed at DDDD, such as "the coordinates or element ID of the click operation", to generate the first prompt instruction. Input the first prompt instruction into the first artificial intelligence model, and the first artificial intelligence model can combine the knowledge base to generate the target automation instruction corresponding to the current natural language description instruction to be executed, such as the first artificial intelligence model can generate:

[0069] Target automation command: click_element(id="login_button").

[0070] The automation framework can then click the login button according to the target automation instruction to update the interface.

[0071] In some embodiments, based on the element information of at least one interface element, the natural language description instruction currently to be executed, and the previous natural language description instruction, using the first artificial intelligence big model in combination with the knowledge base, generating a target automation instruction corresponding to the natural language description instruction currently to be executed may include:

[0072] Based on the element information of at least one interface element, the natural language description instruction currently to be executed and the previous natural language description instruction, the first artificial intelligence big model is combined with the knowledge base to generate the target automation instruction and the expected interface state corresponding to the natural language description instruction currently to be executed.

[0073] For example, the instruction generation template could be:

[0074] Generate target automation instructions based on the following UI element information and instructions:

[0075] UI elements: AAAA

[0076] Current instruction: BBBB

[0077] Historical instructions: CCCC.

[0078] Please generate the target automation instructions for DDDD and describe the expected interface state after the operation.

[0079] The target automation instructions and expected interface states generated by the first artificial intelligence big model combined with the knowledge base can be:

[0080] Target automation command: click_element(id="login_button").

[0081] Expected interface status: the interface jumps to the home page.

[0082] After obtaining the updated interface, the method may further include: based on the element information of at least one interface element of the updated interface and the expected interface state, using a second artificial intelligence large model combined with a knowledge base to verify whether the updated interface is correct.

[0083] Returning to the execution order to determine whether the current natural language description instruction to be executed and the previous natural language description instruction are to be continued may include: when the interface after verification is correct, returning to the execution order to determine whether the current natural language description instruction to be executed and the previous natural language description instruction are to be continued. The current natural language description instruction to be executed is the next natural language description instruction of the executed natural language description instruction. According to the above example, the current natural language description instruction to be executed is "search for product 'mobile phone'".

[0084] Among them, the second artificial intelligence big model can be an LLM, which can be the same as or different from the first artificial intelligence big model.

[0085] Based on the element information of at least one interface element of the updated interface and the expected interface state, the second artificial intelligence large model is used in combination with the knowledge base to verify whether the updated interface is correct. The method can be as follows: based on the element information of at least one interface element of the updated interface, the expected interface state and the instruction verification template, a second prompt instruction is generated, and the second prompt instruction is input into the second artificial intelligence large model to verify whether the updated interface is correct by using the second artificial intelligence large model in combination with the knowledge base.

[0086] The updated interface is the interface updated after the automation framework executes the target automation instruction corresponding to the natural language description instruction to be executed currently.

[0087] For example, a directive validation template could be:

[0088] Verify whether the operation is successful based on the following UI element information and expected interface status:

[0089] UI elements: EEEE

[0090] Expected interface status: FFFF

[0091] Please determine whether the current interface meets your expectations.

[0092] According to the above example, the updated interface is the interface updated after the automation framework executes the target automation instruction click_element(id="login_button") corresponding to "click the login button". For example, the updated interface may be the product display homepage, and the element information of at least one interface element of the product display homepage may be: [

[0094] {"id":"home_title","text":"Home","bounds":[50,100,300,150]},

[0095] {"id":"search_box","text":"Search products","bounds":[50,200,300,250]} ]

[0097] Fill in EEEE, and fill in FFFF with the expected interface state "interface jumps to the home page" generated by the first artificial intelligence model in the above example to generate a second prompt instruction.

[0098] The second artificial intelligence large model combines with the knowledge base to verify whether the updated interface is correct. The output result of the second artificial intelligence large model is "verification correct" or "verification error".

[0099] The artificial intelligence big model can understand the semantics and context of the interface elements and verify whether the automated operation is in line with expectations. In this embodiment, the artificial intelligence big model can quickly analyze the interface information, which can improve the verification efficiency compared to manual inspection and verification. The interface information in the knowledge base can be compared with the information of the actual updated interface to improve the accuracy of the verification.

[0100] In some embodiments, the method may further include: in the case of an interface error after verification, based on element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction, using a third artificial intelligence large model in combination with a knowledge base to determine the error type of the updated interface, generating processing automation instructions for the error type, and using an automation framework to perform processing operations according to the processing automation instructions.

[0101] Among them, the third artificial intelligence big model can be LLM, can be the same as or different from the first artificial intelligence big model, and can be the same as or different from the second artificial intelligence big model.

[0102] Among them, based on the element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction, the third artificial intelligence big model is used in combination with the knowledge base to determine the error type of the updated interface, and the automatic processing instruction for the error type is generated. It can be: based on the element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction and the instruction correction template, a third prompt instruction is generated, and the third prompt instruction is input into the third artificial intelligence big model, so as to use the third artificial intelligence big model in combination with the knowledge base to determine the error type of the updated interface, and generate the automatic processing instruction for the error type.

[0103] In some embodiments, the third artificial intelligence model is combined with the knowledge base to determine the error type of the updated interface, generate a processing automation instruction for the error type, and use the automation framework to perform the processing operation according to the processing automation instruction, which may be:

[0104] The third artificial intelligence large model is used in combination with the knowledge base to determine that the error type of the updated interface is that the interface is blocked, identify the blocking elements and generate automatic closing instructions for the blocking elements; input the automatic closing instructions into the automation framework, and use the automation framework to close the blocking elements according to the automatic closing instructions.

[0105] Alternatively, the third artificial intelligence model is combined with the knowledge base to determine that the error type of the updated interface is an error interface, an automatic return to the previous step instruction is generated, and the automatic framework is used to return to the interface before the update according to the automatic return to the previous step instruction. The error interface may include a failure prompt interface, etc., and the failure prompt interface may include an interface that prompts a network connection failure or a server error, etc.

[0106] For example, the instruction error correction template can be:

[0107] Based on the following UI element information and instructions, analyze the cause of the operation failure and provide a correction solution:

[0108] UI element: XXXX

[0109] Current command: YYYY

[0110] Historical command: ZZZZ

[0111] Please analyze the error type and generate automatic instructions for handling the error type.

[0112] According to the above example, the natural language description instruction to be executed can be filled in at YYYY as "click the login button", and the previous natural language description instruction can be filled in at ZZZZ as "open the APP". The updated interface is the interface updated after the automation framework executes the target automation instruction click_element(id="login_button") corresponding to "click the login button". Assuming that the correct interface should be the product display homepage, the actual updated interface verification error is the failure prompt interface. The element information of at least one interface element of the failure prompt interface can be as follows: [

[0114] {"id":"error_popup","text":"Login failed","bounds":[100,300,300,400]},

[0115] {"id":"close_button","text":"Close","bounds":[250,350,300,400]}

[0116] ]Fill in to XXXX to generate the third prompt instruction.

[0117] Assume that the third prompt instruction is input into the third artificial intelligence model, and the third artificial intelligence model is used in combination with the knowledge base to determine that the error type of the updated interface is that the interface is blocked, identify the blocking elements and generate an automatic closing instruction for the blocking elements, such as the third artificial intelligence model can output:

[0118] Error type: The interface is blocked. Solution: click_element(id="close_button")

[0119] Furthermore, the automation framework may click the close button to close the blocking element according to click_element(id="close_button").

[0120] Assume that the third prompt instruction is input into the third artificial intelligence model, and the third artificial intelligence model is used in combination with the knowledge base to determine that the error type of the updated interface is an error interface, and an automatic return to the previous step instruction is generated, such as the third artificial intelligence model can output:

[0121] Error type: Error interface processing method: click_element(id="back_button")

[0122] Furthermore, the automation framework may click the back button according to click_element(id="back_button") to return to the interface before the update.

[0123] If the error type of the updated interface is that the interface is blocked, after closing the blocking element using the automation framework according to the automation closing instruction, the method can also: re-verify whether the updated interface is correct using the second artificial intelligence large model. If the updated interface is correct, return to the execution order to determine the current natural language description instruction to be executed and the step of the previous natural language description instruction to continue execution.

[0124] If the error type of the interface after the update is an error interface, using the automation framework to return to the interface before the update according to the automatic return to the previous step instruction may include: when the automatic return to the previous step instruction using the automation framework can return to the interface before the update, return to the execution order, determine the current natural language description instruction to be executed and the steps of the previous natural language description instruction to continue execution; when the automatic return to the previous step instruction using the automation framework cannot return to the interface before the update, use the first artificial intelligence large model to generate an automatic return to the initial state instruction, so that the automation framework can return to the initial state according to the automatic return to the initial state instruction. The initial state may refer to the state before the automation framework executes any target automation instruction. According to the above example, it may refer to the state when the automation framework does not execute the target automation instruction corresponding to "open the APP". Therefore, the automation framework can start from the first natural language description instruction and execute multiple target automation instructions corresponding to multiple natural language description instructions in sequence to realize the automated operation of the target application.

[0125] The knowledge base may include processing methods corresponding to a variety of error types. The error types included in the knowledge base are not limited to blocked interfaces and error interfaces. Therefore, the error types of the updated interface determined by the third artificial intelligence model in combination with the knowledge base are not limited to blocked interfaces and error interfaces, and the generated processing automation instructions for the error types are not limited to automated closing instructions and automated return to the previous step instructions.

[0126] In this embodiment, the artificial intelligence big model combined with the knowledge base can more accurately understand the interface information, error types and corresponding processing methods of the error types, reducing the possibility of misjudgment. Using the artificial intelligence big model to identify the error type and generate processing automation instructions can reduce the time for manual error troubleshooting and error repair, and improve the efficiency of automated operations. In the test scenario, the test cycle can be shortened, and the error type feedback from the third artificial intelligence big model can be used as a reference for developers to adjust the configuration of the interface, etc.

[0127] Figure 2 A schematic diagram of the structure of an automated operation device provided by an embodiment of the present invention, the device may include:

[0128] An acquisition module 201, configured to acquire at least one natural language description instruction for a target application in response to an instruction input operation;

[0129] A determination module 202, configured to determine element information of at least one interface element in a target application;

[0130] A generation module 203 is used to generate a target automation instruction corresponding to each natural language description instruction based on the element information of at least one interface element and at least one natural language description instruction by using the first artificial intelligence big model in combination with a knowledge base; the knowledge base includes the element information and interaction information of at least one interface element in the target application;

[0131] The execution module 204 is used to use the automation framework to sequentially execute at least one target automation instruction corresponding to at least one natural language description instruction to implement an automated operation on a target application.

[0132] In some embodiments, the determination module can also be used to determine the execution order of at least one natural language description instruction; according to the execution order, determine the natural language description instruction currently to be executed and the previous natural language description instruction of the natural language description instruction currently to be executed.

[0133] The generation module generates a target automation instruction corresponding to each natural language description instruction based on the element information of at least one interface element and at least one natural language description instruction, using the first artificial intelligence large model in combination with the knowledge base, which may include: obtaining the element information of at least one interface element in the current interface corresponding to the natural language description instruction to be executed; based on the element information of at least one interface element, the natural language description instruction to be executed currently and the previous natural language description instruction, using the first artificial intelligence large model in combination with the knowledge base, generating a target automation instruction corresponding to the natural language description instruction to be executed currently.

[0134] The execution module uses the automation framework to sequentially execute at least one target automation instruction corresponding to at least one natural language description instruction to implement automated operations on a target application, which may include: using the automation framework to execute the target automation instruction to perform the corresponding automated operations in the current interface of the target application to obtain an updated interface; returning to the execution order to determine the current natural language description instruction to be executed and the steps of the previous natural language description instruction to continue execution.

[0135] Among them, the generation module generates a target automation instruction corresponding to the natural language description instruction to be executed currently based on the element information of at least one interface element, the natural language description instruction to be executed currently and the previous natural language description instruction, using the first artificial intelligence big model in combination with the knowledge base, which may include: generating a first prompt instruction based on the element information of at least one interface element, the natural language description instruction to be executed currently, the previous natural language description instruction and the instruction generation template, and inputting the first prompt instruction into the first artificial intelligence big model, so as to use the first artificial intelligence big model in combination with the knowledge base to generate a target automation instruction corresponding to the natural language description instruction to be executed currently.

[0136] In some embodiments, based on the element information of at least one interface element, the natural language description instruction currently to be executed, and the previous natural language description instruction, the first artificial intelligence big model is combined with the knowledge base to generate the target automation instruction corresponding to the natural language description instruction currently to be executed, including:

[0137] Based on the element information of at least one interface element, the natural language description instruction currently to be executed and the previous natural language description instruction, the first artificial intelligence big model is combined with the knowledge base to generate the target automation instruction and the expected interface state corresponding to the natural language description instruction currently to be executed.

[0138] After obtaining the updated interface, the device may further include:

[0139] The verification module is used to verify whether the updated interface is correct based on the element information of at least one interface element of the updated interface and the expected interface state, using the second artificial intelligence large model in combination with the knowledge base.

[0140] The execution module returns to the execution order to determine the current natural language description instruction to be executed and the step of the previous natural language description instruction to continue execution, which may include: when the interface after verification and update is correct, returning to the execution order to determine the current natural language description instruction to be executed and the step of the previous natural language description instruction to continue execution. The current natural language description instruction to be executed is the next natural language description instruction of the executed natural language description instruction. According to the above example, the current natural language description instruction to be executed is "search for product 'mobile phone'".

[0141] Among them, the second artificial intelligence big model can be an LLM, which can be the same as or different from the first artificial intelligence big model.

[0142] The verification module verifies whether the updated interface is correct based on the element information of at least one interface element of the updated interface and the expected interface state, using the second artificial intelligence large model in combination with the knowledge base. The verification module can generate a second prompt instruction based on the element information of at least one interface element of the updated interface, the expected interface state and the instruction verification template, and input the second prompt instruction into the second artificial intelligence large model to verify whether the updated interface is correct by using the second artificial intelligence large model in combination with the knowledge base.

[0143] In some embodiments, the device may further include:

[0144] The error correction module is used to determine the error type of the updated interface based on the element information of at least one interface element of the updated interface, the current natural language description instruction to be executed and the previous natural language description instruction, using the third artificial intelligence large model in combination with the knowledge base, generate processing automation instructions for the error type, and use the automation framework to perform processing operations according to the processing automation instructions.

[0145] Among them, the error correction module determines the error type of the updated interface based on the element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction, using the third artificial intelligence large model in combination with the knowledge base, and generates an automated processing instruction for the error type. It can be: based on the element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction and the instruction error correction template, a third prompt instruction is generated, and the third prompt instruction is input into the third artificial intelligence large model, so as to use the third artificial intelligence large model in combination with the knowledge base to determine the error type of the updated interface, and generate an automated processing instruction for the error type.

[0146] In some embodiments, the error correction module uses the third artificial intelligence large model in combination with the knowledge base to determine the error type of the updated interface, generates a processing automation instruction for the error type, and uses the automation framework to perform the processing operation according to the processing automation instruction, which may be:

[0147] The third artificial intelligence large model is used in combination with the knowledge base to determine that the error type of the updated interface is that the interface is blocked, identify the blocking elements and generate automatic closing instructions for the blocking elements; input the automatic closing instructions into the automation framework, and use the automation framework to close the blocking elements according to the automatic closing instructions.

[0148] Alternatively, the third artificial intelligence large model is used in combination with the knowledge base to determine that the error type of the interface after the update is an error interface, an automatic return to the previous step instruction is generated, and the execution module uses the automation framework to return to the interface before the update according to the automatic return to the previous step instruction.

[0149] If the error type of the updated interface is that the interface is blocked, after the blocking element is closed by the automation framework according to the automation closing instruction, the verification module can also use the second artificial intelligence model to re-verify whether the updated interface is correct. If the updated interface is verified to be correct, the execution module can return to the execution order to determine the current natural language description instruction to be executed and the steps of the previous natural language description instruction to continue execution.

[0150] If the error type of the interface after the update is an error interface, the execution module uses the automation framework to return to the interface before the update according to the automatic return to the previous step instruction, which may include: when the automatic return to the previous step instruction using the automation framework can return to the interface before the update, return according to the execution order, determine the current natural language description instruction to be executed and the steps of the previous natural language description instruction to continue execution; when the automatic return to the previous step instruction using the automation framework cannot return to the interface before the update, the generation module can use the first artificial intelligence large model to generate an automatic return to the initial state instruction, and the execution module uses the automation framework to return to the initial state according to the automatic return to the initial state instruction.

[0151] Figure 2 The device shown can perform Figure 1 The automated operation method of the embodiment shown in the figure, for the parts not described in detail in this embodiment, please refer to Figure 1 The implementation process and technical effects of this technical solution refer to Figure 1 The description in the illustrated embodiment will not be repeated here.

[0152] In one possible design, Figure 1 The automated operation method provided in the illustrated embodiment may be applied in an electronic device, such as Figure 3 As shown, the electronic device may include: a processor 31 and a memory 32. The memory 32 is used to store the electronic device to perform the above Figure 1 The program of the method for generating burial point parameters provided in the illustrated embodiment, the processor 31 is configured to execute the program stored in the memory 32.

[0153] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the processor 31, the following steps can be implemented:

[0154] In response to the instruction input operation, obtaining at least one natural language description instruction for the target application;

[0155] Determining element information of at least one interface element in the target application;

[0156] Based on the element information of the at least one interface element and the at least one natural language description instruction, a target automation instruction corresponding to each natural language description instruction is generated by using the first artificial intelligence big model in combination with a knowledge base; the knowledge base includes the element information and interaction information of the at least one interface element in the target application;

[0157] The automation framework is used to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement an automated operation on the target application.

[0158] Optionally, the processor 31 is further configured to execute the aforementioned Figure 1 All or part of the steps in the illustrated embodiments.

[0159] The structure of the electronic device may further include a communication interface 33 for the electronic device to communicate with other devices or communication systems.

[0160] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by the above electronic device, which includes instructions for executing the above Figure 1 The procedures involved in the automated operation method shown.

[0161] In addition, an embodiment of the present invention provides a computer program product. The computer program product includes a computer program or an instruction. When the computer program or the instruction is executed by a processor, the processor is enabled to implement the above Figure 1 The steps or functions of the automated operation method shown.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated operation method, characterized in that: include: In response to the instruction input operation, obtaining at least one natural language description instruction for the target application; Determining element information of at least one interface element in the target application; Based on the element information of the at least one interface element and the at least one natural language description instruction, a target automation instruction corresponding to each natural language description instruction is generated by using the first artificial intelligence big model in combination with a knowledge base; the knowledge base includes the element information and interaction information of the at least one interface element in the target application; The automation framework is used to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement an automated operation on the target application.

2. The method according to claim 1, characterized in that: Also includes: determining an execution order of the at least one natural language description instruction; Determine, according to the execution order, the natural language description instruction to be currently executed and the previous natural language description instruction of the natural language description instruction to be currently executed; The generating of the target automation instruction corresponding to each natural language description instruction by using the first artificial intelligence big model in combination with the knowledge base based on the element information of the at least one interface element and the at least one natural language description instruction includes: Obtaining element information of at least one interface element in the current interface corresponding to the natural language description instruction to be executed currently; Based on the element information of the at least one interface element, the natural language description instruction to be executed currently and the previous natural language description instruction, a target automation instruction corresponding to the natural language description instruction to be executed currently is generated by using the first artificial intelligence big model in combination with the knowledge base; The using the automation framework to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement the automation operation on the target application includes: Executing the target automation instruction using an automation framework to perform a corresponding automation operation in the current interface of the target application to obtain an updated interface; Return to the execution order to determine the current natural language description instruction to be executed and the step of the previous natural language description instruction to continue execution.

3. The method according to claim 2, characterized in that The step of generating a target automation instruction corresponding to the natural language description instruction to be executed currently based on the element information of the at least one interface element, the natural language description instruction to be executed currently, and the previous natural language description instruction using the first artificial intelligence large model in combination with the knowledge base includes: Based on the element information of the at least one interface element, the natural language description instruction to be currently executed and the previous natural language description instruction, a target automation instruction and an expected interface state corresponding to the natural language description instruction to be currently executed are generated by using the first artificial intelligence big model in combination with a knowledge base; After obtaining the updated interface, the method further includes: Based on the element information of at least one interface element of the updated interface and the expected interface state, using the second artificial intelligence large model in combination with the knowledge base, verifying whether the updated interface is correct; The step of returning to determine the current natural language description instruction to be executed and the previous natural language description instruction to be executed in the execution order and continuing to execute includes: When the updated interface is verified to be correct, the execution sequence is returned to determine the current natural language description instruction to be executed and the step of the previous natural language description instruction to continue execution.

4. The method according to claim 3, characterized in that In the case of an interface error after verifying the update, the method further comprises: Based on the element information of at least one interface element of the updated interface, the natural language description instruction currently to be executed and the previous natural language description instruction, the third artificial intelligence large model is used in combination with the knowledge base to determine the error type of the updated interface, and a processing automation instruction for the error type is generated, and the automation framework is used to perform processing operations according to the processing automation instruction.

5. The method according to claim 4, characterized in that The using of the third artificial intelligence large model in combination with the knowledge base to determine the error type of the updated interface, generating a processing automation instruction for the error type, and using the automation framework to perform a processing operation according to the processing automation instruction includes: Determine using the third artificial intelligence model in combination with the knowledge base that the error type of the updated interface is that the interface is blocked, identify the blocking element and generate an automatic closing instruction for the blocking element; input the automatic closing instruction into the automation framework, and use the automation framework to close the blocking element according to the automatic closing instruction; or, The third artificial intelligence large model is used in combination with the knowledge base to determine that the error type of the interface after the update is an error interface, an automatic return to the previous step instruction is generated, and the automation framework is used to return to the interface before the update according to the automatic return to the previous step instruction.

6. The method according to claim 5, characterized in that The knowledge base also includes processing methods corresponding to a variety of error types.

7. The method according to claim 1, characterized in that The determining element information of at least one interface element in the target application includes: Acquire, using the automation framework, element information of at least one interface element in the target application in an extensible markup language format; Remove the element information corresponding to the interface elements that cannot be interacted with; The element information is parsed into element information of a key-value pair structure.

8. An automated operating device, characterized in that: include: An acquisition module, configured to acquire at least one natural language description instruction for a target application in response to an instruction input operation; A determination module, used to determine element information of at least one interface element in the target application; A generation module, configured to generate a target automation instruction corresponding to each natural language description instruction by using the first artificial intelligence big model in combination with a knowledge base based on the element information of the at least one interface element and the at least one natural language description instruction; the knowledge base includes the element information and interaction information of the at least one interface element in the target application; The execution module is used to use the automation framework to sequentially execute at least one target automation instruction corresponding to the at least one natural language description instruction to implement the automation operation on the target application.

9. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the automated operation method according to any one of claims 1 to 7.

10. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the automated operation method according to any one of claims 1 to 7.

Images