Business processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a business processing method and device, equipment and a storage medium. The method comprises the steps of obtaining business description information and an image of a target page of a target application; based on the business description information, the image and a reference document, an execution strategy for executing related operation on the target page is determined, and the reference document indicates functions of elements contained in the page of the application and an interaction result of interaction with the elements; and based on the execution strategy, executing related operation on the target page. In this way, the problem that the flexibility, adaptability and cross-application capability of an automatic agent are poor can be solved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, apparatuses, devices, and storage media for business processing. Background Art

[0002] An automated agent is an intelligent entity that can perceive the environment, make decisions, and execute actions. Like humans, automated agents have memory, logical analysis, the ability to decompose tasks and problems, and the ability to integrate and solve problems. However, automated agents have certain limitations. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for business processing is provided. The method includes: obtaining business description information and an image of a target page of a target application; determining an execution strategy for performing relevant operations on the target page based on the business description information, the image, and a reference document, wherein the reference document indicates the functions of elements contained in the page of the application and the results of interacting with the elements; and performing relevant operations on the target page based on the execution strategy.

[0004] In a second aspect of the present disclosure, a device for business processing is provided. The device includes: an acquisition module configured to acquire business description information and an image of a target page of a target application; a determination module configured to determine an execution strategy for performing relevant operations on the target page based on the business description information, the image, and a reference document, the reference document indicating the functions of elements contained in the page of the application and the results of interacting with the elements; and an execution module configured to perform relevant operations on the target page based on the execution strategy.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to perform the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, wherein the computer executable instructions implement the method according to the first aspect of the present disclosure when executed by a processor.

[0008] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart showing a process for business processing according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing an example of a process for processing user input according to some embodiments of the present disclosure;

[0013] Figure 4 A block diagram showing an apparatus for business processing according to some embodiments of the present disclosure; and

[0014] Figure 5 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", and these terms are used interchangeably in this article.

[0019] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called the mapping of input to output) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained from the training to determine the corresponding output.

[0020] As briefly mentioned above, an automated agent is an intelligent entity that can perceive the environment, make decisions, and execute actions. Like humans, an automated agent has memory, logical analysis, task decomposition, problem decomposition, and the ability to synthesize and solve problems.

[0021] In some scenarios, automation agents can be implemented based on rule-based automation, using predefined rules to simulate user interactions to handle predictable and standardized tasks. In other scenarios, automation agents can be implemented based on macro recording and playback tools, which can record user operations and generate scripts to achieve automated playback of tasks. In other scenarios, automation agents can be implemented based on no-code / low-code automation platforms, which allow users to build automation processes without coding through visual interfaces and drag-and-drop operations; or they can be implemented based on scripting languages ​​and programming libraries, giving developers the ability to write scripts to control applications.

[0022] In these solutions, the automated agents implemented in the above manner have certain limitations. For example, they rely on predefined rules, macro recording or scripting, which limits the flexibility and intuitiveness of the automated agents, resulting in limitations in the interactive methods of the automated agents. For example, they have poor adaptability when faced with new or unfamiliar scenarios (such as user interfaces). For example, the automated agents implement fixed actions through programs or rules, which results in the lack of generalization capabilities of the automated agents across applications.

[0023] In order to at least partially solve the above problems and other potential problems that may exist in traditional solutions, the present disclosure provides a solution for business processing. According to various embodiments of the present disclosure, first, business description information and an image of a target page of a target application are obtained. Further, based on the business description information, the image, and a reference document, an execution strategy for performing relevant operations on the target page is determined, and the reference document indicates the functions of the elements contained in the application page and the interaction results of interacting with the elements. Finally, based on the execution strategy, the relevant operations are performed on the target page. In this way, the flexibility, adaptability, and poor cross-application capabilities of the automation agent can be improved.

[0024] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, various types of applications 130 may be installed on a terminal device 110. A large language model (LLM) 140 is configured to process natural language and / or images, for example, to identify user intent based on business description information input by the user, or to identify elements in the image based on the image.

[0026] In some embodiments, the large language model 140 used may be deployed on the remote device 120. The terminal device 110 may communicate with the remote device 120 (e.g., via network communication) to utilize the large language model 140 stored thereon to perform the task of processing natural language and / or images. In some embodiments, the large language model 140 may also be partially or completely deployed locally on the terminal device 110 and run by the terminal device 110 to perform the task of processing natural language and / or images. The embodiments of the present disclosure do not specifically limit this.

[0027] The terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The remote device 120 can include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, a virtual machine, and the like. Although a single device is shown, the remote device 120 can include multiple physical devices. In addition, although only a single terminal device 110 is shown, the remote device 120 or the model 140 deployed therein can be accessed by multiple terminal devices 110 to provide the reasoning capability of the model 140.

[0028] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

[0029] Figure 2 1 is a flowchart of a process 200 for business processing according to some embodiments of the present disclosure. The process 200 may be implemented in the terminal device 110 or the remote device 120. For ease of description, the following description will be based on an example of the terminal device 110 executing the process 200. It should be understood that a similar process may also be executed by the remote device 120. Figure 1 The process 200 is described in detail.

[0030] In block 210, the terminal device 110 obtains the service description information and the image of the target page of the target application 130. In some embodiments, the service description information is information used to describe the relevant operations performed on the target application 130. For example, the service description information can be used to describe information that enables the target application 130 to implement one or more functions. In some embodiments, the service description information can include, but is not limited to: text information, voice information, pictures, videos, etc.

[0031] In some embodiments, the target application 130 can be determined by the service description information. For example, if the service description information is text information, such as "send an email to user A", the terminal device 110 can determine the email application as the target application 130 from multiple applications installed in the terminal device 110 based on the service description information.

[0032] Similarly, the target page of the target application 130 can also be determined by the business description information. Continuing with the above example, after determining that the target application 130 is an email application, the terminal device 110 can control the email application to open the corresponding page for sending emails as the target page. In some embodiments, the image of the target page can be a screenshot of the target page, or it can be an image of the target page captured by other devices and sent to the terminal device 110 via other devices.

[0033] It should be understood that the terminal device 110 can identify key information in the service description information (eg, the type of the target application 130 , the target page among multiple pages of the target application) through the large language model 140 .

[0034] In some embodiments, for the convenience of the user, the image of the target page may be a screenshot of the current page of the target application 130. During the operation of the target application 130, the terminal device 110 may use the current page of the target application 130 as the target page, and perform a screenshot operation on the current page of the target application 130 to obtain the image of the current page, so that the user can subsequently input relevant business description information for the current page to the terminal device 110, thereby avoiding the user from sending erroneous business description information due to unfamiliarity with the relevant pages of the target application 130, thereby improving the user experience.

[0035] In block 220 , the terminal device 110 determines an execution strategy for executing related operations on the target page based on the business description information, the image, and the reference document.

[0036] When determining the execution strategy for performing relevant operations on the target page, the terminal device 110 can implement the relevant process with the help of the trained large language model 140. In some embodiments, the terminal device 110 can identify the elements contained in the target page based on the image through the trained large language model 140. Further, the user intention for the business description information is determined based on the business description information through the trained large language model 140. Finally, the execution strategy is determined based on the elements contained in the target page, the user intention, the reference document, and the predetermined action space. For example, the terminal device 110 can input the business description information and the image of the target page into the large language model 140 respectively, and output the corresponding user intention and the elements contained in the target page.

[0037] In some embodiments, the elements included in the target page may be user interface (UI) elements, such as buttons, text boxes, menus, check boxes, drop-down lists, sliders, and other graphical interface elements, which are not limited in the embodiments of the present disclosure.

[0038] The reference document indicates the functions of the elements contained in the application page and the results of interacting with the elements. For example, the reference document can be a natural language description of the functions of UI elements in the relevant pages of various applications and the results of interacting with each UI element (e.g., clicking, long pressing, sliding, inputting text, etc.) (e.g., jumping to another page after clicking).

[0039] After obtaining the corresponding user intent and the elements included in the target page, the terminal device 110 can determine the functions of all elements in the target page and the interaction results of interacting with all elements based on the reference document. In combination with the user intent, the terminal device 110 can determine the relevant elements for the user intent among the elements included in the target page and the specific way to interact with the relevant elements, so that the corresponding action function for performing the relevant operation can be called from the predetermined action space, and the execution strategy for performing the relevant operation on the target page can be generated.

[0040] In some embodiments, the terminal device 110 can determine the order of executing corresponding action functions according to the user's intention, and generate an operation script as an execution strategy based on the corresponding action functions according to the order.

[0041] It should be understood that the terminal device 110 may also call the large language model 140 to enable the large language model 140 to implement the above process of determining the execution strategy.

[0042] The predetermined action space includes a set of all possible action functions that can be executed. For example, the predetermined action space includes action functions that perform related operations. In some embodiments, the action function may include at least one of the following: a tap action function, a long press action function, a swipe action function, a text input action function, a back action function, or an exit action function.

[0043] When the terminal device 110 calls the large language model 140 to implement the above process of determining the execution strategy, the large language model needs to be trained accordingly.

[0044] In some embodiments, the large language model 140 can be trained based on sample pages and predetermined operations. In some embodiments, the terminal device 110 can obtain corresponding sample pages of multiple sample applications and predetermined operations on the corresponding sample pages for training. During the training, the large language model 140 can observe the impact of the predetermined operation on the UI elements in the sample page and record it to complete the training. In some embodiments, the terminal device 110 can also obtain a video and train the large language model 140 based on the video. The video can be a video indicating the user's operation on the sample page and the operation result corresponding to the operation. The large language model 140 completes the training by observing the operation on the UI element in the page in the video and recording the operation result corresponding to the operation.

[0045] In some embodiments, during the training of the large language model 140, the large language model 140 may generate a reference document to record the functions of the sample elements contained in the sample page and the interaction results of interacting with the sample elements, so as to facilitate subsequent use as a reference for executing actions indicated by the user's intention.

[0046] In block 230 , the terminal device 110 performs relevant operations on the target page based on the execution policy.

[0047] The terminal device 110 may perform relevant operations on the target page based on the execution strategy generated by the large language model 140. In some embodiments, the terminal device 110 may obtain the operation result of performing the relevant operation on the target page, and update the reference document based on the operation result, so that the reference document can be continuously optimized to improve the accuracy and efficiency of subsequent execution tasks.

[0048] In some embodiments, after the terminal device 110 performs the relevant operation on the target page based on the execution strategy, it can also determine whether the operation result of the relevant operation meets the effect indicated in the user's intention based on the large language model 140. If it meets the requirements, the terminal device 110 can call the relevant action function to exit the target application 130. If it does not meet the requirements, the terminal device 110 can continue to call the relevant action function based on the above process until the operation result meets the effect indicated in the user's intention.

[0049] In order to more clearly explain the solutions for business processing provided by some embodiments of the present disclosure, a specific example is described below. Of course, it can be understood that the embodiments of the present disclosure are not limited to this application scenario.

[0050] Figure 3 A schematic diagram showing an example of a process 300 for processing user input according to some embodiments of the present disclosure. The process 300 may be implemented by the terminal device 110. Figure 1Process 300 is introduced.

[0051] It should be noted that the elements included in the page involved in this example are: a text box, a query button, and the user input is: query the customer information with the customer number XXX.

[0052] After acquiring the screenshot 310 and user input 320 of the above-mentioned page, the terminal device 110 inputs the screenshot 310 and user input 320 into the trained large language model 140, and outputs the elements 330 in the screenshot and the user intent 340. Further, the terminal device 110 can reorganize the recognition elements based on the user intent 340, the elements 330 in the screenshot, and the reference document to obtain the following information 350: Page 1, customer number, text box, input, input customer number; Page 1, query, button, click, query function. Further, the terminal device 110 generates a corresponding operation script 360 based on the user intent 340, the information 350, and the predetermined action space. Finally, the terminal device 110 executes the operation script 360 to realize the function of querying the customer information with the customer number XXX in the page.

[0053] In some embodiments, an automation agent application may be installed in the terminal device 110, so that the automation agent application implements the process 300. For related contents, reference may be made to the aforementioned contents, which will not be described in detail here. In some embodiments, when the automation agent application implements the process 300, the automation agent may be run on the command line interface (CLI) of the terminal device 110. In some embodiments, the automation agent application may receive a screenshot 310 of a page and an XML file of information of elements contained in the page.

[0054] In summary, the solution for business processing proposed in the present disclosure can improve the flexibility, adaptability and poor cross-application capabilities of automated agents, and can handle multiple input modes (e.g., text, image, voice, etc.), thereby realizing multimodal agents.

[0055] Figure 4 A block diagram of an apparatus 400 for business processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented in the remote device 120 and / or the terminal device 110. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware or any combination thereof.

[0056] The apparatus 400 includes an acquisition module 410 configured to acquire business description information and an image of a target page of a target application. The apparatus 400 also includes a determination module 420 configured to determine an execution strategy for performing related operations on the target page based on the business description information, the image, and a reference document, wherein the reference document indicates the functions of the elements contained in the page of the application and the interaction results of interacting with the elements. The apparatus 400 also includes an execution module 430 configured to perform related operations on the target page based on the execution strategy.

[0057] In some embodiments, the determination module 420 is also configured to identify elements contained in the target page based on the image through a trained large language model; determine the user intent for the business description information based on the business description information through a trained large language model; and determine the execution strategy based on the elements contained in the target page, the user intent, the reference document, and a predetermined action space, wherein the predetermined action space includes action functions for performing related operations.

[0058] In some embodiments, the predetermined action space includes at least one of the following action functions: a click action function, a long press action function, a slide action function, an input text action function, a return action function, or an exit action function.

[0059] In some embodiments, the image is a screenshot of the current page of the target application.

[0060] In some embodiments, the device 400 also includes an update module configured to obtain an operation result of performing a related operation on the target page; and update the reference document based on the operation result.

[0061] In some embodiments, the large language model is trained based on sample pages and predetermined operations.

[0062] In some embodiments, during the training of the large language model, a reference document is generated to record the functions of the sample elements contained in the sample page and the interaction results of interacting with the sample elements.

[0063] In some embodiments, the large language model is trained based on a video, which indicates the user's operations on the sample page and the corresponding operation results.

[0064] The units included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0065] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 remote device 120 and / or terminal device 110.

[0066] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0067] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 500.

[0068] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0069] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0070] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0071] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0072] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0073] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0076] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the implementations disclosed herein.

Claims

1. A method for business processing, comprising: Obtaining business description information and an image of a target page of a target application; Determining an execution strategy for executing related operations on the target page based on the business description information, the image, and a reference document, wherein the reference document indicates functions of elements included in the application page and results of interactions with the elements; as well as Based on the execution strategy, the related operation is performed on the target page.

2. The method of claim 1 , wherein determining the execution strategy comprises: Using a trained large language model, based on the image, identify elements included in the target page; Determining the user intention for the service description information based on the service description information by using the trained large language model; as well as The execution strategy is determined based on the elements included in the target page, the user intention, the reference document, and a predetermined action space, wherein the predetermined action space includes an action function for executing the related operation.

3. The method according to claim 2, wherein the predetermined action space includes at least one of the following action functions: Click on the action function. Long press action function, Slide action function, Input text action function, Returns an action function, or Exit the action function. The method according to claim 1 , wherein the image is a screenshot of a current page of the target application.

5. The method according to claim 1, further comprising: Obtaining the result of performing the related operation on the target page; as well as Based on the operation result, the reference document is updated.

6. The method according to claim 1, wherein the large language model is trained based on sample pages and predetermined operations.

7. The method according to claim 6, wherein during the training of the large language model, the reference document is generated to record the functions of the sample elements contained in the sample page and the interaction results of interacting with the sample elements.

8. The method according to claim 1, wherein the large language model is trained based on a video, wherein the video indicates a user's operation on a sample page and an operation result corresponding to the operation.

9. A device for business processing, comprising: An acquisition module, configured to acquire business description information and an image of a target page of a target application; a determination module configured to determine an execution strategy for performing relevant operations on the target page based on the business description information, the image, and a reference document, wherein the reference document indicates functions of elements included in the application page and interaction results of interacting with the elements; as well as The execution module is configured to execute the relevant operation on the target page based on the execution strategy.

10. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images