Method, device and electronic equipment for executing operation instructions of financial software

Abstract

The application discloses an execution method and device of operation instructions of financial software and electronic equipment, relates to the field of financial technology or other related technical fields, and comprises the following steps: collecting a screen image of the financial software and a multi-modal instruction input by a user, wherein the multi-modal instruction is used for representing the operation intention of the user, and the multi-modal instruction comprises a voice instruction and / or a text instruction; generating a target topology graph based on the screen image, wherein the target topology graph represents L operation elements in the screen image and the association relationship between the L operation elements in the form of a directed acyclic graph, and L is a positive integer; determining a target instruction sequence based on the multi-modal instruction and the target topology graph, wherein the target instruction sequence comprises M planning instructions, and M is a positive integer; and sequentially executing the M planning instructions in the target instruction sequence. The application solves the technical problem that the financial mobile terminal in the prior art cannot realize full-automatic process operation and business handling through user instructions.

Description

Technical Field

[0001] This application relates to the field of financial technology or other related technical fields. Specifically, it relates to a method, apparatus, and electronic device for executing operation instructions of financial software. Background Technology

[0002] Currently, financial mobile applications have become the main channel for users to conduct financial business. However, most apps have complex business processes and frequent page jumps, making it impossible to achieve fully automated process operations and business processing through natural user commands. At the same time, the existing app interaction design is not fully adapted to special groups such as elderly users and disabled users, with high operating thresholds, making it difficult to provide convenient and efficient services. Intelligent customer service has inaccurate responses to user commands, requiring users to ask questions repeatedly, which reduces the service experience.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for executing operation instructions of financial software, so as to at least solve the technical problem that existing financial mobile terminals cannot achieve fully automated process operation and business processing through user instructions.

[0005] According to one aspect of this application, a method for executing operation instructions of financial software is provided, comprising: acquiring a screen image of the financial software and multimodal instructions input by a user, wherein the multimodal instructions are used to represent the user's operation intention, and the multimodal instructions include voice instructions and / or text instructions; generating a target topology map based on the screen image, wherein the target topology map represents L operation elements in the screen image and the relationships between the L operation elements in a directed acyclic form, where L is a positive integer; determining a target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; and sequentially executing the M planning instructions in the target instruction sequence.

[0006] Optionally, generating a target topology map based on a screen image includes: detecting N operation elements in the screen image and the page identifier corresponding to the screen image, where N is a positive integer greater than or equal to L, and the page identifier is the identifier of the front-end page corresponding to the screen image; performing data cleaning on the N operation elements to obtain L operation elements, wherein the data cleaning is used to remove operation elements without semantic information at least; querying the database of the financial software based on the page identifier to obtain P operation records, where P is a positive integer, and the P operation records are used to represent the historical operation information of the operation elements in the front-end page indicated by the page identifier; generating a target topology map based on the L operation elements, the element information of each of the L operation elements, and the P operation records, wherein the element information includes at least element type, element text, and element coordinates.

[0007] Optionally, generating a target topology graph based on L operation elements, element information of each of the L operation elements, and P operation records includes: materializing the L operation elements to obtain L nodes corresponding to the L operation elements; determining the node attributes of each of the L nodes based on the element information of each of the L operation elements; determining the association relationships between the L nodes based on the P operation records; connecting the L nodes with directed edges according to the association relationships between the L nodes to obtain an initial topology graph, wherein the weight of each directed edge is used to characterize the probability that the user will transfer from the operation element corresponding to the starting point of the edge to the operation element corresponding to the ending point of the edge; after performing a detection operation on the initial topology graph, embedding a preset cyclic gating unit into the initial topology graph that has passed the detection operation to obtain the target topology graph, wherein the preset cyclic gating unit is used to perform information propagation operations along the directed edges.

[0008] Optionally, a detection operation is performed on the initial topology graph, including: performing a first detection operation, wherein the first detection operation is used to detect whether there is a loop in the initial topology graph; performing a second detection operation, wherein the second detection operation is used to detect whether the direction of the directed edge in the initial topology graph has a logical conflict with a preset business rule; and determining that the initial topology graph passes the detection operation if the initial topology graph passes the first detection operation and the second detection operation.

[0009] Optionally, determining the target instruction sequence based on multimodal instructions and the target topology graph includes: extracting features from the multimodal instructions using a pre-set large model to obtain instruction features, wherein the instruction features are used to characterize keyword information and context information in the multimodal instructions; performing information propagation operations along the directed edges in the target topology graph using a pre-set loop gating unit based on the instruction features and the node attributes of each node in the target topology graph to obtain the user's operation timing features, wherein the operation timing features are used to characterize the user's access order, dwell pattern, and jump rhythm of operation elements in the financial software; and determining the target instruction sequence based on the operation timing features.

[0010] Optionally, determining the target instruction sequence based on operation timing features includes: generating M intermediate operation tasks based on operation timing features, wherein each of the M intermediate operation tasks corresponds to an atomic operation; obtaining the operation target corresponding to each intermediate operation task, wherein the operation target is the operation element that executes the intermediate operation task; generating planning instructions corresponding to each intermediate operation task based on each intermediate operation task and the operation target corresponding to each intermediate operation task, thereby obtaining M planning instructions; and sorting the M planning instructions based on operation timing features to obtain the target instruction sequence.

[0011] Optionally, the M planning instructions in the target instruction sequence are executed sequentially, including: after the i-th planning instruction in the M planning instructions is executed, the execution result corresponding to the i-th planning instruction is collected; if the execution result corresponding to the i-th planning instruction is consistent with the expected result corresponding to the i-th planning instruction, the (i+1)-th planning instruction in the M planning instructions is executed; if the execution result corresponding to the i-th planning instruction is inconsistent with the expected result corresponding to the i-th planning instruction, a rollback operation is performed, and the target instruction sequence is regenerated.

[0012] According to another aspect of this application, an execution device for operation instructions of financial software is also provided, comprising: an acquisition unit for acquiring screen images of the financial software and multimodal instructions input by the user, wherein the multimodal instructions are used to represent the user's operation intention, and the multimodal instructions include voice instructions and / or text instructions; a target generation unit for generating a target topology map based on the screen images, wherein the target topology map represents L operation elements in the screen images and the relationships between the L operation elements in a directed acyclic form, where L is a positive integer; a target determination unit for determining a target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; and an instruction execution unit for sequentially executing the M planning instructions in the target instruction sequence.

[0013] According to another aspect of this application, a computer program product is also provided, which stores a computer program, wherein a method for executing operation instructions of financial software that controls the computer program product to execute any of the above-mentioned items is provided when the computer program is running.

[0014] According to another aspect of this application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for executing the operation instructions of the financial software described above.

[0015] In this application, the screen image of the financial software and the multimodal instructions input by the user are first collected. The multimodal instructions are used to represent the user's operation intention and include voice instructions and / or text instructions. Then, this application generates a target topology map based on the screen image. The target topology map represents the L operation elements in the screen image and the relationship between the L operation elements in the form of directed acyclic graphs, where L is a positive integer. Then, this application determines a target instruction sequence based on the multimodal instructions and the target topology map. The target instruction sequence includes M planning instructions, where M is a positive integer. Subsequently, this application executes the M planning instructions in the target instruction sequence sequentially.

[0016] As described above, this application employs a directed acyclic graph (DAG) modeling approach to model the relationships between operational elements in the software interface of a financial mobile terminal. By collecting screen images from the financial mobile terminal to generate a target topology graph, and combining this with multimodal commands for user intent parsing and command sequence planning, it achieves the goal of accurately identifying user operational intent and automatically generating executable operation sequences. Based on the DAG structure representing the browsing and jumping relationships (i.e., association relationships) between interface elements, and combined with multimodal input commands for command sequence reasoning, this application improves the accuracy and contextual consistency of the system's executed operation commands. It avoids the problems of misoperation and process interruption caused by interface changes or complex interactions in traditional solutions, thereby achieving the technical effect of automatically completing financial business process operations based on interface status and natural language commands. This solves the technical problem that existing financial mobile terminals cannot achieve fully automated process operations and business processing through user natural commands. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an operation instruction execution method for financial software, according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of an optional method for executing operation instructions of financial software according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of an optional directed acyclic graph cyclic gated unit network structure according to an embodiment of this application;

[0021] Figure 4 This is a flowchart of an optional MobileAgent design method according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an execution device for operation instructions of an optional financial software according to an embodiment of this application;

[0023] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] It should also be noted that all information and data (including but not limited to information used for display and analysis) involved in this application are authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send a request to the aforementioned user or organization through the interface, and obtain the relevant information only after receiving consent from the aforementioned user or organization.

[0027] Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and data involved in this application all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary security measures have been taken. They do not violate public order and good morals. In addition, this application provides corresponding operation entry points for users to choose to agree to authorization or refuse authorization. If the user chooses to refuse authorization, the corresponding expert decision-making process will be initiated.

[0028] In one optional embodiment, a mobile application solution based on a multimodal large model, MobileAgent, is provided. MobileAgent can accurately identify multimodal commands such as user voice and text, and convert user intent into automatic operations of the mobile APP. Relying on the visual perception and intent understanding capabilities of the multimodal large model, MobileAgent can realize functions such as cross-application operation and autonomous task planning. It does not require the APP to write XML operation documents, but can complete operation positioning and execution solely through visual perception. Its core advantage lies in its ability to simulate the user's operation logic, autonomously understand user needs, plan operation paths, execute operation actions, and provide feedback results, thereby effectively simplifying the APP operation process and lowering the usage threshold for special groups.

[0029] Compared to traditional app-based intelligent customer service, MobileAgent offers greater autonomy and interactivity, significantly improving user business processing efficiency. While MobileAgent currently relies on a multimodal big model to automate basic operations, it is prone to errors in locating clickable elements in complex app interfaces. Furthermore, MobileAgent cannot accurately model user eye movements and browsing navigation, leading to issues such as accidental touches and process lag, making it difficult to meet the accuracy requirements of financial business processing.

[0030] To address the technical problems of insufficient operation positioning accuracy and inaccurate user behavior modeling in existing MobileAgent applications in APP scenarios, this invention proposes a technical solution based on Directed Acyclic Graph (DAG) Gated Recurrent Unit (GRU). The aim is to optimize the perception and planning capabilities of MobileAgent through a pre-designed DAG-GRU network, improve its operation positioning accuracy and process execution accuracy, and further promote the large-scale and standardized application of MobileAgent in mobile scenarios.

[0031] The present invention will now be described in detail with reference to various embodiments.

[0032] Example 1

[0033] According to an embodiment of this application, an embodiment of a method for executing operation instructions of financial software is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) according to an optional embodiment of this application for implementing a method for executing operation instructions of financial software. For example... Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the execution method of operation instructions of financial software in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned execution method of operation instructions of financial software. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be configured as a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Under the aforementioned operating environment, this application provides an execution system (hereinafter referred to as the execution system) for executing the operation instructions of the financial software described in this application. Figure 2 This is a flowchart of an optional method for executing operation instructions of financial software according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0040] Step S201: Collect the screen image of the financial software and the multimodal instructions input by the user. The multimodal instructions are used to represent the user's operation intention and include voice instructions and / or text instructions.

[0041] Optionally, after the user launches the mobile app, the system control perception unit continuously calls the app's screenshot interface to acquire the screen image of the app's current interface in real time. At the same time, it collects the user's voice commands (such as "I want to transfer money to Zhang San") through the voice recognition module, and / or receives the user's text commands (such as "Zhang San's account is: XXX") through the text input box. The voice recognition module and the text input box can be used in combination to form a multimodal command input. The acquired screen image and multimodal commands serve as the input source for recognizing the user's operation intention.

[0042] Optionally, screen image refers to the pixel-level image of the APP's current display interface, containing visual information of all operation elements (such as buttons, input boxes, text, and icons); multimodal instruction refers to natural language instructions expressed by the user in the form of voice or text, used to drive the system to perform operations; operation intent refers to the specific business action that the user wants to complete through multimodal instructions, such as "transfer money", "check balance", "change password", which needs to be parsed by the execution system from natural language to extract the executable semantic goal.

[0043] Optionally, by acquiring screen images and multimodal instructions, the execution system overcomes the limitations of traditional apps that rely on fixed menus or voice commands. Based on the acquired natural voice and / or text instructions expressed by real users, it eliminates the need for users to memorize complex app operation paths, thereby lowering the operational threshold of the app and making it more user-friendly for older users and those with limited app operation functions. At the same time, the screen images provide a realistic interface state for the visual perception of subsequent large models, avoiding the adaptation failure problem caused by relying solely on structured UI (User Interface) documents, and enhancing the universality of the execution system for different app interfaces.

[0044] Step S202: Generate a target topology graph based on the screen image. The target topology graph represents the L operation elements in the screen image and the relationship between the L operation elements in a directed acyclic form, where L is a positive integer.

[0045] Optionally, the execution system inputs the collected screen images into the OCR (Optical Character Recognition) module and icon recognition model in the perception unit to identify all clickable elements and corresponding text information in the current APP software interface, obtaining a total of L operation elements and element information corresponding to each element. Subsequently, based on historical user operation logs, the execution system constructs element nodes and directed edges between nodes: for example, after a user clicks "Transfer" on the "Home" page, they are usually redirected to the "Recipient Selection Page" and then enter the "Amount Input Page". Based on this, the system establishes a directed edge between the "Home" node and the "Transfer" node, and then connects it to the "Recipient Selection" node to form a directed path. The nodes corresponding to the L operation elements and the directed edges constitute an acyclic structure (without circular dependencies), i.e., the target topology graph.

[0046] Optionally, the target topology graph refers to the graph structure dynamically constructed for the current APP software interface, which expresses the operation elements and their logical relationships in the form of a directed acyclic graph; operation elements refer to visual objects in the screen image that users can interact with, such as buttons, links, input boxes, icons, etc.; relationships refer to the sequential dependencies or jump relationships between operation elements in the user's usage flow.

[0047] Optionally, the execution system models user operation behavior as a directed acyclic graph (DAG), breaking through the limitations of traditional MobileAgent which relies solely on pixel coordinates or static layout trees for positioning. The DAG structure accurately captures the essential characteristics of operation path dependencies and process sequence in the APP, such as business rules like "the recipient must be selected before the amount is entered." Compared to unstructured visual detection, the DAG clearly expresses the logical jump relationships between elements, enabling the execution system to still locate users through path logic rather than absolute coordinates when faced with changes in interface layout (such as button position changes), thereby improving robustness. At the same time, the acyclic structure avoids infinite loops in planning, ensuring that the path can terminate and guaranteeing stability.

[0048] Step S203: Determine the target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer.

[0049] Optionally, after creating the target topology graph, information is propagated along each directed edge of the graph by embedding a cyclic gating unit in the target topology graph. Combined with the GRU's ability to model temporal paths, the operation timing features corresponding to the user's intent are inferred. Then, the execution system generates M intermediate operation tasks based on the operation timing features. Based on each intermediate operation task and the operation target corresponding to each intermediate operation task, planning instructions are generated to obtain M planning instructions. Then, the M planning instructions are sorted to obtain the target instruction sequence.

[0050] Optionally, the target instruction sequence refers to a set of M atomic operation instructions generated in the order of execution to fulfill the user's intent, each instruction corresponding to a low-level action (i.e., atomic operation) that can be recognized by the executable unit; the planned instruction refers to the smallest operation unit that can be directly executed by the APP, decomposed from the user's operation intent at a higher level.

[0051] Optionally, the execution system determines the target instruction sequence based on multimodal instructions and the target topology graph, achieving a deep integration of semantic understanding and structural reasoning. Traditional methods rely solely on semantic models to match keywords, which can easily lead to misjudgments in business operations (e.g., "transfer" may be mistaken for "payment"). In contrast, this application uses the target topology graph to constrain business operation paths and models users' historical behavioral preferences in similar operations using GRU, thereby improving the accuracy of target instruction sequence planning.

[0052] Step S204: Execute the M planning instructions in the target instruction sequence sequentially.

[0053] Optionally, the execution unit calls the APP's automation control interface one by one according to the order of the target instruction sequence, and executes each planned instruction in sequence. After each instruction is executed, the system verifies whether the operation element corresponding to the planned instruction has been successfully changed by comparing the screen image in real time. If it fails, the reflection unit is triggered to retry or correct the planned instruction. Otherwise, the next instruction is executed until the target instruction sequence is completed.

[0054] Optionally, the execution system ensures that each operation is performed on the basis of the previous step's success by planning the serial execution of instructions and verifying the execution results, thus avoiding process crashes caused by "jumping and breaking". Combined with the reflection unit, if the execution fails, it can roll back or adjust the path in real time instead of directly reporting an error and exiting, thereby improving the success rate of user instructions.

[0055] As described above, this application employs a directed acyclic graph (DAG) modeling approach to model the relationships between operational elements in the software interface of a financial mobile terminal. By collecting screen images from the financial mobile terminal to generate a target topology graph, and combining this with multimodal commands for user intent parsing and command sequence planning, it achieves the goal of accurately identifying user operational intent and automatically generating executable operation sequences. Based on the DAG structure representing the browsing and jumping relationships (i.e., association relationships) between interface elements, and combined with multimodal input commands for command sequence reasoning, this application improves the accuracy and contextual consistency of the system's executed operation commands. It avoids the problems of misoperation and process interruption caused by interface changes or complex interactions in traditional solutions, thereby achieving the technical effect of automatically completing financial business process operations based on interface status and natural language commands. This solves the technical problem that existing financial mobile terminals cannot achieve fully automated process operations and business processing through user natural commands.

[0056] In one optional embodiment, the execution system first detects N operation elements in the screen image and the page identifier corresponding to the screen image, where N is a positive integer greater than or equal to L, and the page identifier is the identifier of the front-end page corresponding to the screen image. Then, the execution system performs data cleaning on the N operation elements to obtain L operation elements, where data cleaning is used to remove operation elements without semantic information. Then, the execution system queries the database of the financial software based on the page identifier to obtain P operation records, where P is a positive integer, and the P operation records are used to represent the historical operation information of the operation elements in the front-end page indicated by the page identifier. Subsequently, the execution system generates a target topology map based on the L operation elements, the element information of each of the L operation elements, and the P operation records, where the element information includes at least element type, element text, and element coordinates.

[0057] Optionally, N operation elements refer to all interactive controls detected from the screen image via OCR, such as buttons, input boxes, links, icons, etc.; P operation records refer to historical operation data matching the current page identifier, with each record containing the elements that the user clicked or entered on the page and their operation order; page identifier is used to uniquely identify the front-end page corresponding to the current screen, such as "Transfer Homepage", "Repayment Confirmation Page", "Investment Details Page".

[0058] Optionally, by generating the target topology map through the steps in the above embodiments, the execution system can achieve the following technical effects:

[0059] (1) Improve the data quality of the set of operational elements: Through data cleaning, the execution system retains only L truly operable and meaningful operational elements, and eliminates interference elements such as advertisements and decorative text, thereby avoiding subsequent planning units from generating incorrect execution paths due to misidentification of invalid elements, and thus reducing the possibility of invalid operations.

[0060] (2) Enhance the semantic consistency of element identification and positioning: Each operation element is bound to at least three types of information: element type, element text and element coordinates, so that the execution system can distinguish operation elements that are semantically similar but have different positions (such as the "confirm" button corresponding to two business functions). Optionally, the execution system can also verify its usage frequency and order through historical records to avoid mismatch caused by coordinate or text matching alone.

[0061] In one optional embodiment, the execution system first materializes L operation elements to obtain L nodes corresponding to the L operation elements. Then, based on the element information of each of the L operation elements, the execution system determines the node attributes of each of the L nodes. Next, the execution system determines the association relationship between the L nodes based on P operation records. Subsequently, based on the association relationship between the L nodes, the execution system connects the L nodes through directed edges to obtain an initial topology graph. The weight of each directed edge is used to characterize the probability that the user will transfer from the operation element corresponding to the starting point of the edge to the operation element corresponding to the ending point of the edge. After performing a detection operation on the initial topology graph, a preset cyclic gating unit is embedded into the initial topology graph that has passed the detection operation to obtain a target topology graph. The preset cyclic gating unit is used to perform information propagation operations along the directed edges.

[0062] Optionally, the information propagation operation refers to the GRU passing and updating the node state layer by layer along the directed edges of the DAG, integrating local attributes and the temporal information of upstream nodes, so that the final state of each node reflects its semantic importance and contextual dependence in the user's operation path; the preset cyclic gated unit, namely the gated cyclic unit (GRU), is a neural network structure with memory capabilities used to process sequence information.

[0063] Optionally, by executing the steps in the above embodiments, the execution system can achieve the following technical effects:

[0064] (1) Achieve structured mapping from operation elements to graph structure: By materializing visual elements into nodes, the original unstructured interface image is transformed into a computable graph model, providing standardized input for subsequent reasoning.

[0065] (2) Achieve the perceptibility of node semantics and position: Each node is bound to node attributes such as element type, element text, and element coordinates, so that the execution system can not only identify elements, but also distinguish the type, position and related functional information of elements, thereby providing the execution unit with accurate operation basis.

[0066] (3) Achieve quantitative modeling of user behavior: The weight of each directed edge in the initial topology graph is obtained by statistical analysis of historical operation records, which can reflect the user's operation preferences on the page, thus completing the leap from static UI structure to behavior-oriented topology graph.

[0067] (4) Achieve dynamic enhancement of path dependency: By embedding GRU into DAG, the execution system can propagate the node state along the path. For example, if a user clicks "Select Account" multiple times on the "Transfer Page" before entering the amount, GRU will enhance the hidden state of the "Select Account" node, making it more prioritized in subsequent reasoning, thereby improving the ability to model complex behaviors.

[0068] In one optional embodiment, the execution system first performs a first detection operation, wherein the first detection operation is used to detect whether there is a loop in the initial topology graph. Then, the execution system performs a second detection operation, wherein the second detection operation is used to detect whether the direction of the directed edge in the initial topology graph has a logical conflict with the preset business rules. Subsequently, if the initial topology graph passes the first and second detection operations, it is determined that the initial topology graph passes the detection operation.

[0069] Optionally, a logical conflict refers to an operation sequence represented by a directed edge that violates the above-mentioned preset rules. For example, there may be a path of "select account → enter amount" but missing "enter amount → confirm", or there may be a reverse error of "confirm payment → select account".

[0070] Optionally, by executing the steps in the above embodiments, the execution system can achieve the following technical effects:

[0071] (1) Preventing infinite loops caused by loops: The business operation process in financial scenarios is linear or tree-like, and there is usually no loop structure. Abnormal execution of system business (such as due to test log pollution) may cause loops in the topology graph. If the system is planned based on this, it may fall into an infinite loop of repeated page jumps. The first detection operation helps to ensure the stability of the APP by actively detecting loops;

[0072] (2) Avoid operation paths that violate the rules of conventional financial business: Financial business has strict restrictions on the order of operation, such as not being able to confirm without filling in the amount, not being able to transfer funds without binding an account, etc. If the reverse operation such as "confirm payment → select account" appears in the figure due to abnormal user behavior (such as accidental click), if the execution system directly adopts it, it will cause the execution unit to issue an incorrect instruction (such as clicking confirm when there is no amount), thereby causing fund errors or security risks. The second detection operation compares the edge direction with the preset rules through the rule engine to actively filter such erroneous connections and ensure the compliance of the execution order of the final generated planning instructions.

[0073] In one optional embodiment, the execution system first extracts features from the multimodal instructions using a preset large model to obtain instruction features, wherein the instruction features are used to characterize the keyword information and context information in the multimodal instructions. Then, the execution system uses a preset loop gating unit to perform information propagation operations along the directed edges of the target topology graph based on the instruction features and the node attributes of each node in the target topology graph to obtain the user's operation timing features, wherein the operation timing features are used to characterize the user's access order, dwell mode and jump rhythm of operation elements in the financial software. Finally, the execution system determines the target instruction sequence based on the operation timing features.

[0074] For example, the access sequence, such as users usually clicking "Transfer" first and then "Select Account"; the dwell mode, such as users lingering for 30 seconds in the "Amount Input Box" after entering an amount; and the jump rhythm, such as the event interval between "Select Account" and "Confirm" being 10 seconds.

[0075] Optionally, by executing the steps in the above embodiments, the execution system can achieve the following technical effects:

[0076] (1) Achieve accurate understanding of user intent: By using a pre-set large model to extract features from multimodal instructions, the ambiguity caused by traditional keyword matching technology can be avoided (such as "transfer" being misjudged as "payment").

[0077] (2) Capturing subtle user behavior patterns to improve the accuracy of path selection: The operation sequence features generated by the execution system not only record the click order, but also include dynamic information such as pause patterns and jump rhythms. For example, if historical data shows that users often pause for 2 seconds before "confirming", the system can infer that they are hesitant and prioritize recommending the "preview" operation rather than executing it directly. This fine-grained modeling makes the planning results more reliable and reduces erroneous operations.

[0078] (3) Achieving collaborative decision-making between semantic-driven and behavior-driven approaches: Traditional methods rely solely on semantic matching of elements or on recommending actions based solely on historical paths, which can easily lead to situations where the semantic matching is correct but the path is incorrect, or the path is reasonable but the semantics are not consistent. The execution system integrates the two through GRU, thus achieving collaborative decision-making between semantic-driven and behavior-driven approaches.

[0079] In one optional embodiment, the execution system first generates M intermediate operation tasks based on operation timing characteristics, wherein each of the M intermediate operation tasks corresponds to an atomic operation. Then, the execution system obtains the operation target corresponding to each intermediate operation task, wherein the operation target is the operation element for executing the intermediate operation task. Then, the execution system generates planning instructions corresponding to each intermediate operation task based on each intermediate operation task and the operation target corresponding to each intermediate operation task, thereby obtaining M planning instructions. Subsequently, the execution system sorts the M planning instructions based on the operation timing characteristics to obtain a target instruction sequence.

[0080] Optionally, an atomic operation refers to an indivisible basic action in mobile operations, including clicking, swiping, text input, long pressing, etc.; the operation target refers to the interface element targeted to perform a certain intermediate operation task, such as a "transfer button", "amount input box", "confirm submission icon", etc. The operation target is determined by the perception unit through the DAG-GRU network and is the specific object of the atomic operation.

[0081] Optionally, by executing the steps in the above embodiments, the execution system can achieve the following technical effects:

[0082] (1) Make the task decomposition fit the actual usage habits: Based on the operation timing characteristics, generate M intermediate operation tasks to ensure that each task is not a virtual assumption, but is generated according to the user's real operation path learned from the target topology map, thus avoiding the problem of generating irrelevant or redundant operations due to misunderstanding of user intentions in traditional methods.

[0083] (2) Improve operational accuracy: By obtaining the operation target corresponding to each intermediate operation task, and based on the DAG-GRU network's ability to accurately locate interface elements, each task is bound to the correct UI element, thereby solving the problem of accidental clicks and wrong button clicks caused by element recognition errors in the existing MobileAgent.

[0084] (3) Avoid ambiguity in subsequent execution: The execution system generates planning instructions based on tasks and objectives, transforming abstract tasks into structured instructions that can be executed by the execution unit. This makes the output of the planning unit clearly executable, avoiding ambiguous instructions or syntax errors.

[0085] In one optional embodiment, after the i-th planning instruction among the M planning instructions is executed, the execution system collects the execution result corresponding to the i-th planning instruction. Then, if the execution result corresponding to the i-th planning instruction is consistent with the expected result corresponding to the i-th planning instruction, the execution system executes the (i+1)-th planning instruction among the M planning instructions. If the execution result corresponding to the i-th planning instruction is inconsistent with the expected result corresponding to the i-th planning instruction, the execution system performs a rollback operation and regenerates the target instruction sequence.

[0086] Optionally, rollback operation refers to actively restoring the system state to the previous known correct and stable state when a planned instruction fails to achieve the expected effect during the execution of the system (such as the interface not jumping as expected, elements not responding correctly, operation failing, or an abnormal state occurring), in order to avoid the error from continuing to propagate or causing more serious consequences.

[0087] Optionally, the execution system, based on the original architecture of MobileAgent, introduces a closed-loop feedback mechanism based on execution result verification. After each planned instruction is executed, the execution system actively collects the actual interface feedback (execution result) and compares it with the expected result predicted by the DAG-GRU model. This confirms whether each step of the operation has truly taken effect, rather than simply considering it complete after issuing the instruction. This solves the instruction execution error problem caused by interface response delay and element recognition deviation in traditional MobileAgent. When the execution result is inconsistent with the expectation, the system immediately triggers a rollback and regenerates the instruction sequence, effectively avoiding the accumulation of errors caused by accidental touches, element misalignment, and abnormal interface loading. Through the closed-loop mechanism of "execution-verification-rollback-replanning", the execution system enables MobileAgent to correct operational deviations in real time, avoid process interruptions, and prevent the spread of misoperations in scenarios such as APPs where accuracy requirements are extremely high. This improves the accuracy, stability, and fault tolerance of the operation.

[0088] Optionally, Figure 3 This is a schematic diagram of an optional directed acyclic graph cyclic gated unit network structure according to an embodiment of this application, such as... Figure 3 As shown, the Directed Acyclic Graph Cyclic Gated Unit (DAG-GRU) network proposed in this application is used to model the topological relationships between elements of a bank's mobile APP interface and the temporal behavioral characteristics of user operations. The DAG-GRU network consists of two layers: the top layer is the DAG structure layer, and the bottom layer is the Cyclic Gated Unit (GRU) computation layer.

[0089] In the DAG structure layer, each node corresponds to an interactive element (such as a button, input box, link, icon, etc.) on the current page of the bank APP. The directed edges between nodes represent the user's gaze shift path or operation jump flow during actual use. Its directionality reflects the logical order of user operations. Moreover, the DAG is an acyclic structure, ensuring that it will not fall into an infinite loop operation path, which conforms to the unidirectional characteristics of real business processes.

[0090] In the GRU computation layer, each DAG node is bound to an independent GRU unit. The GRU propagates information layer by layer along the directed edges of the DAG, realizing dynamic modeling of the user's sequential browsing behavior within local blocks on the page. The update gate and reset gate mechanism of the GRU effectively filters key temporal information and suppresses noise interference, thereby improving the convergence speed and long-term dependency capture capability.

[0091] As can be seen from the above, through the collaborative structure of DAG-GRU, this application dynamically models the interface topology and user operation sequence, providing MobileAgent with a high-precision joint representation of "interface semantics + behavioral features", thereby improving the accuracy of predicting user intent and the robustness of locating operation elements.

[0092] Optionally, Figure 4 This is a flowchart of an optional MobileAgent design method according to an embodiment of this application, such as... Figure 4 As shown, the MobileAgent in this application includes a perception unit, a planning unit, an execution unit, and a reflection unit. The functions of each unit are as follows:

[0093] The process begins at the perception unit: the user inputs natural language commands via voice and / or text (such as "I want to transfer 5,000 yuan to Zhang San"), which calls a multimodal large model to analyze the current APP screenshot, combining OCR to recognize text content and icon classification models to recognize functional controls, generating a structured interface state diagram; subsequently, this state diagram is input into the DAG-GRU network, which automatically matches the current interface element nodes based on the pre-trained DAG model of the bank APP page, extracts the semantic features and contextual association weights of each element, and outputs a set of operable elements with temporal priority.

[0094] The perception results are transmitted to the planning unit. Based on the "operation element sequence" and "user behavior pattern" output by DAG-GRU, and combined with the task semantic parsing results, the planning unit calls the graph search algorithm to find a suitable execution path in DAG, generates an atomic operation sequence, and performs path rationality verification (such as whether it crosses the permission boundary or whether it misses the secondary verification) to ensure that the process complies with security specifications.

[0095] Planning instructions are issued to the execution unit: The execution unit converts the atomic operation sequence into automated instructions for the mobile terminal, and performs operations such as clicking, swiping, and input in the real APP environment, reporting screen status changes in real time during the operation process.

[0096] After the operation is completed, the reflection unit compares the real-time interface status with the expected target status (preset by the planning unit). If a deviation occurs (such as not jumping to the success page or popping up an error message), the error diagnosis mechanism is triggered. The activation history of relevant nodes in the DAG-GRU network is traced back to determine whether it is a perception misjudgment, a path planning error or external interference (such as a verification code pop-up). The timing weights of the GRU are dynamically adjusted or the planned path is regenerated to achieve online self-correction.

[0097] As described above, this application improves the operational accuracy of the mobile agent in banking app scenarios by deeply integrating a Directed Acyclic Graph Gated Unit (DAG-GRU) network with the MobileAgent. The DAG structure accurately models the association between banking app interface elements and the user's browsing path, while the GRU unit efficiently captures the timing features of user operations. This effectively solves the technical problems of positioning deviation and inaccurate behavior modeling in existing MobileAgents, reduces the rate of accidental touches, and thus enables accurate execution of user instructions, simplifying banking business processes. At the same time, this application improves the adaptability to elderly and disabled users, lowers the usage threshold, and takes into account the operating efficiency of mobile devices. While ensuring operational accuracy, it achieves stable operation of the MobileAgent.

[0098] Example 2

[0099] This application embodiment can also provide an execution device for operation instructions of financial software. It should be noted that the execution device for operation instructions of financial software in this application embodiment can be used to execute the execution method for operation instructions of financial software provided in this application embodiment. The execution device for operation instructions of financial software provided in this application embodiment will be described below.

[0100] According to an embodiment of this application, an apparatus for executing the operation instructions of the aforementioned financial software is also provided. Figure 5 This is a schematic diagram of an execution device for operation instructions of optional financial software according to an embodiment of this application, such as... Figure 5 As shown, the device includes: a data acquisition unit 501, a target generation unit 502, a target determination unit 503, and an instruction execution unit 504.

[0101] Optionally, the acquisition unit 501 is used to acquire screen images of the financial software and multimodal instructions input by the user, wherein the multimodal instructions are used to represent the user's operation intentions, and the multimodal instructions include voice instructions and / or text instructions; the target generation unit 502 is used to generate a target topology map based on the screen images, wherein the target topology map represents L operation elements in the screen images and the relationships between the L operation elements in a directed acyclic form, where L is a positive integer; the target determination unit 503 is used to determine a target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; and the instruction execution unit 504 is used to sequentially execute the M planning instructions in the target instruction sequence.

[0102] In one optional embodiment, the target generation unit 502 includes: a detection subunit, a cleaning subunit, a query subunit, and a target generation subunit.

[0103] Optionally, the detection subunit is used to detect N operation elements in the screen image and the page identifier corresponding to the screen image, where N is a positive integer greater than or equal to L, and the page identifier is the identifier of the front-end page corresponding to the screen image; the cleaning subunit is used to perform data cleaning on the N operation elements to obtain L operation elements, where data cleaning is used to remove operation elements without semantic information at least; the query subunit is used to query the database of the financial software based on the page identifier to obtain P operation records, where P is a positive integer, and the P operation records are used to represent the historical operation information of the operation elements in the front-end page indicated by the page identifier; the target generation subunit is used to generate a target topology map based on the L operation elements, the element information of each of the L operation elements, and the P operation records, where the element information includes at least element type, element text, and element coordinates.

[0104] In one optional embodiment, the target generation subunit includes: a materialization module, a first determination module, a second determination module, a connection module, and an embedding module.

[0105] Optionally, the entityization module is used to entityize L operation elements to obtain L nodes corresponding to the L operation elements; the first determination module is used to determine the node attributes of each of the L nodes based on the element information of each operation element in the L operation elements; the second determination module is used to determine the association relationship between the L nodes based on P operation records; the connection module is used to connect the L nodes through directed edges according to the association relationship between the L nodes to obtain an initial topology graph, wherein the weight of each directed edge is used to characterize the probability that the user will transfer from the operation element corresponding to the starting point of the edge to the operation element corresponding to the ending point of the edge; the embedding module is used to embed a preset loop gating unit into the initial topology graph that has passed the detection operation after performing a detection operation on the initial topology graph to obtain a target topology graph, wherein the preset loop gating unit is used to perform information propagation operation along the directed edges.

[0106] In one optional embodiment, the target generation subunit further includes: a first execution module, a second execution module, and a third determination module.

[0107] Optionally, the first execution module is used to execute a first detection operation, wherein the first detection operation is used to detect whether there is a loop in the initial topology graph; the second execution module is used to execute a second detection operation, wherein the second detection operation is used to detect whether the direction of the directed edge in the initial topology graph has a logical conflict with the preset business rules; and the third determination module is used to determine that the initial topology graph passes the detection operation if the initial topology graph passes the first detection operation and the second detection operation.

[0108] In one optional embodiment, the target determination unit 503 includes a feature extraction subunit, a propagation subunit, and a target determination subunit.

[0109] Optionally, the feature extraction subunit is used to extract features from multimodal instructions using a preset large model to obtain instruction features, wherein the instruction features are used to characterize keyword information and context information in multimodal instructions; the propagation subunit is used to perform information propagation operations along directed edges in the target topology graph based on the instruction features and the node attributes of each node in the target topology graph using a preset loop gating unit to obtain user operation timing features, wherein the operation timing features are used to characterize the user's access order, dwell pattern, and jump rhythm of operation elements in the financial software; and the target determination subunit is used to determine the target instruction sequence based on the operation timing features.

[0110] In one optional embodiment, the target determination subunit includes: a first generation module, an acquisition module, a second generation module, and a sorting module.

[0111] Optionally, the first generation module is used to generate M intermediate operation tasks based on operation timing features, wherein each of the M intermediate operation tasks corresponds to an atomic operation; the acquisition module is used to acquire the operation target corresponding to each intermediate operation task, wherein the operation target is the operation element that executes the intermediate operation task; the second generation module is used to generate planning instructions corresponding to each intermediate operation task based on each intermediate operation task and the operation target corresponding to each intermediate operation task, thereby obtaining M planning instructions; and the sorting module is used to sort the M planning instructions based on the operation timing features, thereby obtaining a target instruction sequence.

[0112] In one optional embodiment, the instruction execution unit 504 includes: a collection subunit, a first execution subunit, and a second execution subunit.

[0113] Optionally, the acquisition subunit is used to acquire the execution result corresponding to the i-th planning instruction after the i-th planning instruction among the M planning instructions has been executed; the first execution subunit is used to execute the (i+1)-th planning instruction among the M planning instructions if the execution result corresponding to the i-th planning instruction is consistent with the expected result corresponding to the i-th planning instruction; and the second execution subunit is used to perform a rollback operation and regenerate the target instruction sequence if the execution result corresponding to the i-th planning instruction is inconsistent with the expected result corresponding to the i-th planning instruction.

[0114] It should be noted that the acquisition unit 501, target generation unit 502, target determination unit 503, and instruction execution unit 504 mentioned above correspond to steps S201 to S204 in the method embodiment. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in the embodiment.

[0115] Example 3

[0116] Embodiments of this application can also provide an electronic device. Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, the electronic device includes: one or more ( Figure 6 (Only one is shown) Processor 602, memory 604, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0117] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the above-mentioned method for executing the operation instructions of the financial software.

[0118] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0119] The processor can access information and applications stored in memory via a transmission device to execute the following steps: acquiring screen images of the financial software and multimodal instructions input by the user, wherein the multimodal instructions represent the user's operational intent and include voice instructions and / or text instructions; generating a target topology map based on the screen images, wherein the target topology map represents L operational elements in the screen images and the relationships between the L operational elements in a directed acyclic form, where L is a positive integer; determining a target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; and sequentially executing the M planning instructions in the target instruction sequence.

[0120] This application provides an execution scheme for operation instructions in financial software. It employs a directed acyclic graph (DAG) model to model the relationships between operation elements in the software interface of a financial mobile terminal. By collecting screen images from the financial mobile terminal to generate a target topology graph, and combining this with multimodal instructions for user intent parsing and instruction sequence planning, it achieves the goal of accurately identifying user operation intentions and automatically generating executable operation sequences. Based on the DAG structure representing the browsing and navigation relationships (i.e., association relationships) between interface elements, and combined with multimodal input instructions for instruction sequence reasoning, this application improves the accuracy and contextual consistency of the executed operation instructions, avoiding the problems of misoperation and process interruption caused by interface changes or complex interactions in traditional solutions. This achieves the technical effect of automatically completing financial business process operations based on interface status and natural language instructions, thereby solving the technical problem that existing financial mobile terminals cannot achieve fully automated process operations and business processing through user natural instructions.

[0121] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, PDAs, mobile internet devices, PADs, and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 6 The different configurations shown.

[0122] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0123] Example 4

[0124] Embodiments of this application may also provide a storage medium.

[0125] Optionally, in this embodiment of the application, the storage medium can be used to store the program code executed by the method for executing the operation instructions of the financial software provided in the above method embodiment.

[0126] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0127] This application also provides a computer program product, which, when executed on a data processing device, is a program suitable for executing the operation instructions of financial software.

[0128] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0134] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for executing operation instructions in financial software, characterized in that, include: The system collects screen images of financial software and multimodal commands input by the user, wherein the multimodal commands are used to represent the user's operational intentions and include voice commands and / or text commands. A target topology graph is generated based on the screen image, wherein the target topology graph represents the L operation elements in the screen image and the association relationship between the L operation elements in a directed acyclic form, where L is a positive integer; The target instruction sequence is determined based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; The M planning instructions in the target instruction sequence are executed sequentially.

2. The method for executing operation instructions of financial software according to claim 1, characterized in that, Generating a target topology map based on the screen image includes: Detect N operation elements in the screen image and the page identifier corresponding to the screen image, where N is a positive integer greater than or equal to L, and the page identifier is the identifier of the front-end page corresponding to the screen image; Data cleaning is performed on the N operation elements to obtain the L operation elements, wherein the data cleaning is at least used to remove operation elements without semantic information; Based on the page identifier, P operation records are obtained by querying the database of the financial software, where P is a positive integer. The P operation records are used to represent the historical operation information of the operation elements in the front-end page indicated by the page identifier. The target topology map is generated based on the L operation elements, the element information of each of the L operation elements, and the P operation records, wherein the element information includes at least element type, element text, and element coordinates.

3. The method for executing operation instructions of financial software according to claim 2, characterized in that, The target topology map is generated based on the L operation elements, the element information of each of the L operation elements, and the P operation records, including: The L operation elements are materialized to obtain L nodes corresponding to the L operation elements; Based on the element information of each of the L operation elements, determine the node attributes of each of the L nodes; The association relationships between the L nodes are determined based on the P operation records; Based on the association between the L nodes, the L nodes are connected by directed edges to obtain an initial topology graph. The weight of each directed edge is used to characterize the probability that the user will move from the operation element corresponding to the starting point of the edge to the operation element corresponding to the ending point of the edge. After performing a detection operation on the initial topology graph, a preset loop gating unit is embedded into the initial topology graph that has passed the detection operation to obtain the target topology graph, wherein the preset loop gating unit is used to perform information propagation operation along the directed edge.

4. The method for executing operation instructions of financial software according to claim 3, characterized in that, The detection operation on the initial topology graph includes: Perform a first detection operation, wherein the first detection operation is used to detect whether there is a loop in the initial topology graph; Perform a second detection operation, wherein the second detection operation is used to detect whether the direction of the directed edge in the initial topology graph has a logical conflict with the preset business rules; If the initial topology map passes both the first and second detection operations, it is determined that the initial topology map passes the detection operations.

5. The method for executing operation instructions of financial software according to claim 1, characterized in that, Determining the target instruction sequence based on the multimodal instructions and the target topology map includes: The multimodal instructions are subjected to feature extraction using a pre-defined large model to obtain instruction features, wherein the instruction features are used to characterize the keyword information and context information in the multimodal instructions; By using a preset loop gating unit to perform information propagation operations along the directed edges of the target topology graph based on the instruction features and the node attributes of each node in the target topology graph, the user's operation timing features are obtained. The operation timing features are used to characterize the user's access order, dwell mode, and jump rhythm of operation elements in the financial software. The target instruction sequence is determined based on the aforementioned operation timing characteristics.

6. The method for executing operation instructions of financial software according to claim 5, characterized in that, Determining the target instruction sequence based on the aforementioned operation timing characteristics includes: Based on the operation timing features, M intermediate operation tasks are generated, wherein each of the M intermediate operation tasks corresponds to an atomic operation. Obtain the operation target corresponding to each intermediate operation task, wherein the operation target is the operation element that executes the intermediate operation task; Based on each intermediate operation task and the operation target corresponding to each intermediate operation task, a planning instruction corresponding to each intermediate operation task is generated to obtain M planning instructions; The M planning instructions are sorted based on the operation timing characteristics to obtain the target instruction sequence.

7. The method for executing operation instructions of financial software according to claim 1, characterized in that, Execute M planning instructions from the target instruction sequence sequentially, including: After the i-th planning instruction among the M planning instructions is executed, the execution result corresponding to the i-th planning instruction is collected; If the execution result of the i-th planning instruction is consistent with the expected result of the i-th planning instruction, then the (i+1)-th planning instruction among the M planning instructions is executed. If the execution result of the i-th planning instruction is inconsistent with the expected result of the i-th planning instruction, a rollback operation is performed, and the target instruction sequence is regenerated.

8. A device for executing operation instructions of financial software, characterized in that, include: The acquisition unit is used to acquire screen images of financial software and multimodal commands input by the user, wherein the multimodal commands are used to represent the user's operation intention, and the multimodal commands include voice commands and / or text commands; A target generation unit is configured to generate a target topology graph based on the screen image, wherein the target topology graph represents the L operation elements in the screen image and the association relationships between the L operation elements in a directed acyclic form, where L is a positive integer; The target determination unit is used to determine a target instruction sequence based on the multimodal instructions and the target topology map, wherein the target instruction sequence includes M planning instructions, where M is a positive integer; The instruction execution unit is used to sequentially execute the M planned instructions in the target instruction sequence.

9. A computer program product, characterized in that, The computer program product includes a computer program, wherein, during the execution of the computer program, a method for controlling the computer program product to execute the operation instructions of the financial software according to any one of claims 1 to 7 is provided.

10. An electronic device, characterized in that, The device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method for executing operating instructions of the financial software according to any one of claims 1 to 7.

Images