Automatic operation and collaborative execution method and system for heterogeneous application software

Abstract

The invention relates to an automatic operation and cooperative execution method and system for heterogeneous application software, and belongs to the technical field of computer software automation, the system adopts a three-layer architecture, and comprises a UI element identification and interaction layer, an intelligent business process arrangement and cooperative execution engine and a unified control and task management platform. The UI element identification and interaction layer provides standardized UI operation capability for heterogeneous applications such as Web, Java and C # through an adapter mode; the engine is responsible for analyzing structured process configuration, scheduling the UI interaction layer to execute operation and managing cross-application data flow and state flow; the platform provides a visual interface for flow configuration, task control and state monitoring. The problem of tool fragmentation is solved through a unified interaction interface, cross-application data automatic circulation is realized through an intelligent scheduling engine, the use threshold is reduced through a visual platform, and three core pain points of non-unified interfaces, no mechanism in cooperation and no means in management and control in the prior art are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software automation, and more particularly, to an automatic operation and collaborative execution method and system for heterogeneous application software. BACKGROUND

[0002] In modern enterprise operation and government affairs processing, a business process often needs to cross multiple heterogeneous application systems to be completed. For example, a financial staff needs to first fill in a reimbursement form in a Web OA system, then complete the approval operation in a C# developed financial approval tool, and finally generate summary data through a Java report system; a government worker needs to enter application information in a Web government platform, verify materials through a desktop special tool, and complete data archiving through an archiving system. Such cross-application business processes have become the norm of enterprise digital operation.

[0003] However, the existing automation technical solutions have significant bottlenecks when facing heterogeneous application collaborative scenarios, and the bottlenecks restrict each other to form a chain reaction: first, there is a tool fragmentation problem at the interaction level. Different technology stacks of applications need to rely on different automation tools (such as Web applications need Selenium, Java desktop applications need PyJab, and C# applications need UIAutomation), the interfaces of each tool are not standardized, developers need to master multiple technical systems, and need to manually adapt the interfaces when integrating cross-application scripts, which significantly reduces the development efficiency. Secondly, there is a lack of effective scheduling mechanism at the collaboration level. Various tool scripts run independently, and lack unified timing scheduling and data flow transfer capabilities. For example, the business document identifier generated by the OA system needs to be manually copied to the financial tool, and cannot be automatically transferred; when an application in the process appears an exception, other related applications cannot perceive and collaboratively process, resulting in prominent process interruption problems. Thirdly, at the technical implementation level, the traditional image recognition positioning method is easily affected by factors such as interface theme and display resolution, and the positioning method based on control attributes has syntax barriers due to technology stack differences, both of which are difficult to guarantee the consistency of cross-scene positioning, increasing the maintenance cost of scripts. Finally, at the control level, there is a lack of visual means. Automation scripts are mostly run in command line form, and lack intuitive process configuration and state monitoring interfaces. New personnel need a long time to learn before they can adjust the process logic, and the process execution state needs to be confirmed by manually checking log files, which is inefficient for exception positioning and problem tracing.

[0004] The above multi-dimensional problems result in high cost, poor stability, and great difficulty in operation and maintenance of cross-application automation solutions, and there is an urgent need for a complete solution that integrates "unified interaction + intelligent collaboration + visual control" to solve the core pain points of multi-technology stack application collaborative execution. SUMMARY

[0005] To address the aforementioned technical problems, this invention proposes an automated operation and collaborative execution method and system for heterogeneous application software.

[0006] The technical solution of this invention is as follows: This invention proposes an automated operation and collaborative execution system for heterogeneous application software, comprising: a UI element recognition and interaction layer, an intelligent business process orchestration and collaborative execution engine, and a unified control and task management platform; wherein: The UI element recognition and interaction layer is used to provide standardized UI element positioning and interaction capabilities for different types of heterogeneous applications through the adapter pattern. Its operation results and status data are output to the intelligent business process orchestration and collaborative execution engine. The intelligent business process orchestration and collaborative execution engine is used to receive and parse structured process configurations from the unified control and task management platform, schedule UI element recognition and interaction layer to execute specific application operations according to process logic, and manage cross-application data streams and state streams generated during process execution. The unified control and task management platform provides users with a visual interface to receive process configuration instructions and task control instructions and send them to the intelligent business process orchestration and collaborative execution engine. It also displays the process execution status, data flow details and exception logs received from the engine in real time.

[0007] Preferably, the UI element recognition and interaction layer includes a unified abstract interface, multiple specialized adapters, and adaptation scheduling logic; A unified abstract interface is used to define a set of standard UI operation methods that are independent of the technology stack; Multiple specialized adapters, corresponding to different types of heterogeneous applications, are used to implement the unified abstract interface and convert standard operation methods into native interactive instructions for specific technology stacks; The adaptation scheduling logic is used to route requests to the corresponding special adapters for execution based on the application type identifier in the operation requests issued by the intelligent business process orchestration and collaborative execution engine.

[0008] Preferably, the specialized adapters include a Web application adapter, a Java desktop application adapter, and a C# / .NET desktop application adapter; The Web application adapter integrates Selenium WebDriver for interacting with web browser applications; The Java desktop application adapter is based on the PyJab library and interfaces with the Java Accessibility API for interacting with Java desktop applications; The C# / .NET desktop application adapter uses pywin32 to call the Win32 API or the WindowsUIAutomation framework to interact with the C# desktop application.

[0009] Preferably, the intelligent business process orchestration and collaborative execution engine includes a process parsing module, a collaborative scheduling module, a global data bus module, and a status and exception management module; The process parsing module is used to verify and parse the structured process configuration, and generate a process object model that includes the execution order of steps, dependencies, and data reference relationships defined by variable placeholders. The collaborative scheduling module is used to call the UI element recognition and interaction layer to execute each step operation in sequence according to the process object model, and to manage the temporal dependencies between steps. The global data bus module is used to centrally store and manage variable data generated during process execution, enabling data flow across steps and applications; The status and exception management module is used to maintain and update the execution status of processes and steps in real time, capture and handle exceptions during execution, and synchronize status and exception information to the unified control and task management platform.

[0010] Preferably, the structured process configuration defines a sequence of process steps, a target application type for each step, an operation sequence, input variables, and output variables.

[0011] Preferably, the unified control and task management platform includes a user authentication and permission control module, a process management module, a task control module, and a log and report module; The user authentication and access control module is used to implement user login verification and role-based access control. The process management module is used to provide process library management, visual process editor and process parameter configuration functions; The task control module is used to receive the user's task control instructions and send them to the engine, while displaying the task execution progress and status in real time. The log and report module is used to receive, hierarchically display and export system execution logs, and generate execution reports.

[0012] Preferably, the system adopts a multi-threaded architecture, with the unified control and task management platform running on the main thread, responsible for interface response; the intelligent business process orchestration and collaborative execution engine and the task execution logic of the UI element recognition and interaction layer running on the slave thread; the main thread and slave thread communicate through a signal-slot mechanism to decouple interface operation from task execution.

[0013] On the other hand, the present invention also provides a method for automated operation and collaborative execution of heterogeneous application software, comprising the following steps: Step S1: Receive the user-configured structured process definition and task start command through the unified control and task management platform; Step S2: The intelligent business process orchestration and collaborative execution engine parses the structured process definition and generates an executable process object model; Step S3: The engine schedules the UI element recognition and interaction layer to perform an operation sequence on the specified heterogeneous application according to the logical order of the process object model; Step S4: During the execution process, the UI element recognition and interaction layer uploads the output data defined in the operation result to the global data bus of the engine for storage; Step S5: When the engine executes subsequent steps, it obtains the output data stored in the previous steps from the global data bus and passes it as an input parameter to the UI element recognition and interaction layer to realize cross-application data flow. Step S6: The engine feeds back the execution status of the entire process, data flow details and abnormal information to the unified control and task management platform in real time for visual monitoring and recording.

[0014] The present invention has the following beneficial effects: 1. The adapter pattern unifies the interaction interfaces of applications with different technology stacks, encapsulating complex underlying technical differences within specialized adapters and providing standardized UI operation capabilities to upper layers. Developers no longer need to master multiple automation tool APIs; they can operate various applications such as Web, Java, and C# through a unified configuration interface, significantly reducing the development and maintenance costs of cross-application scripts and improving development efficiency.

[0015] 2. Through an intelligent business process orchestration and collaborative execution engine, automated time-series scheduling and cross-application data flow of processes are achieved. Business processes are defined using declarative structured configuration, supporting various process logics such as operation nodes, decision nodes, and loop nodes; automatic data transfer between heterogeneous applications is achieved through a global data bus and variable placeholder mechanisms; and intelligent handling of process anomalies is ensured through a status and exception management module, significantly reducing manual intervention and the risk of process interruption.

[0016] 3. The positioning method based on the native properties of controls, combined with an intelligent waiting mechanism, effectively overcomes the drawbacks of traditional image recognition positioning being affected by interface theme and resolution, while also avoiding syntax barriers caused by differences in technology stacks. Element caching and window management mechanisms further ensure the accuracy and consistency of UI interaction, significantly reducing script maintenance costs.

[0017] This invention has wide application value in cross-application scenarios such as financial reimbursement, government approval, and production data processing. It can realize full-process automation from data entry to business processing and result generation, significantly shorten the business process execution cycle, reduce human error, and improve data processing efficiency and accuracy. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system structure of the present invention; Figure 2 This is a sample code image showing the core content of the expense_flow.json file. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0021] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0023] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0024] Example 1: To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0025] This invention adopts a three-layer architecture of "standardized bottom-layer interaction + intelligent mid-layer scheduling + visualized top-layer management and control." Through deep integration of the UI element recognition and interaction layer (bottom layer), the intelligent business process orchestration and collaborative execution engine (mid layer), and the unified control and task management platform (top layer), it achieves automated collaborative execution of heterogeneous applications. The core logical loop of the three-layer architecture is as follows: the top-layer management and control platform receives user operation instructions and sends them to the mid-layer engine; the mid-layer engine parses the process configuration and schedules the adaptation capabilities of the bottom-layer interaction layer; the bottom-layer interaction layer completes the actual interaction operation with the target application, and simultaneously synchronizes the execution status and data back to the mid-layer engine; finally, the mid-layer engine feeds back the data to the top-layer platform for display, forming a complete chain of "instruction issuance - operation execution - status feedback."

[0026] To address the problems of existing technologies, this invention provides an automated operation and collaborative execution system for heterogeneous application software, comprising: a UI element recognition and interaction layer, an intelligent business process orchestration and collaborative execution engine, and a unified control and task management platform; wherein: The UI element recognition and interaction layer is used to provide standardized UI element positioning and interaction capabilities for different types of heterogeneous applications through the adapter pattern. Its operation results and status data are output to the intelligent business process orchestration and collaborative execution engine. This layer is the core execution unit for the interaction between the system and the target application. The core design concept is "unified interface encapsulation + specialized adaptation and implementation". By using the adapter pattern, it shields the interaction differences between different technology stacks and provides the mid-layer engine with standardized UI operation capabilities without any differences.

[0027] Architectural Logic and Component Collaboration The layer adopts a three-level adaptation logic of "abstract interface - special adapter - target application", and the components work together to form standardized interaction capabilities.

[0028] The unified abstract interface (IUIElementOperator) defines a set of UI operation methods independent of the technology stack, serving as the sole entry point for calls from upper-layer modules. Core methods include: `find_element(locator)`, which locates the target UI element using a standardized locator (JSON format, including location type, location value, and alternative location strategies); `click(element)`, which performs a click operation on the located element; `input_text(element, content)`, which passes specified text content to an input element; `get_text(element)`, which extracts the text information of the target element; and `wait_for_element(locator, timeout)`, which intelligently waits for the element to reach an interactive state (e.g., visible or clickable). This interface only defines the operation specifications and does not contain specific implementation logic, ensuring consistency and compatibility for upper-layer calls.

[0029] Specialized Adapters: Serving as an "adaptation bridge" between the unified interface and the target application, these adapters implement the unified abstract interface and encapsulate the native interaction capabilities of the corresponding technology stack. The adaptation logic of the three core adapters is as follows: Web Application Adapter: Integrates Selenium WebDriver, converting the unified interface's `find_element` method into Selenium's native location methods (such as `find_element(By.XPATH, value)`), while also supporting web application-specific interaction scenarios such as DOM parsing and JavaScript execution, and adapting to mainstream browsers; Java Desktop Application Adapter: Based on the PyJab library and connecting to the Java Accessibility API, converting the unified interface methods into logic for locating Swing / AWT controls through native attributes such as `accessibilityName` and `className`, with the location effect unaffected by the interface theme and resolution; C# / .NET Desktop Application Adapter: Calls the Win32 API via pywin32 or connects to the Windows UI Automation framework, converting the unified interface methods into location logic for traversing the control tree through the window handle (HWND), adapting to mainstream C# application architectures such as WinForms / WPF.

[0030] Adaptive scheduling logic: When the upper-layer module calls the unified interface, it only needs to pass in the type identifier of the target application (such as "WEB", "JAVA", "C_SHARP") and the standardized locator. The layer automatically matches the corresponding special adapter and performs the operation based on the type identifier. For example, when calling the click method, passing in the "WEB" identifier will trigger the Selenium click logic of the Web adapter, and passing in the "JAVA" identifier will trigger the PyJab click logic of the Java adapter. The upper-layer module does not need to be aware of the underlying technology differences.

[0031] Core technology support – ensuring interaction stability To support the accuracy and stability of standardized interactions, two core technical mechanisms are integrated within the layer: Intelligent waiting and element caching mechanism: The wait_for_element method determines the element status by polling, replacing the traditional fixed delay strategy, effectively reducing interaction failures caused by elements not being fully loaded; at the same time, it has a built-in element cache pool, and successfully located elements will be temporarily stored in memory, and the cached results will be reused directly when repeated operations are performed, reducing the overhead of repeated location.

[0032] Cross-application window management mechanism: Through the window management interface of Windows API or browser driver, the window handle of the target application is accurately obtained and activated, ensuring that operation commands are accurately delivered to the target window when switching between multiple applications, avoiding "misoperation" or "unresponsiveness" problems caused by loss of window focus.

[0033] The intelligent business process orchestration and collaborative execution engine is used to receive and parse structured process configurations from the unified control and task management platform, schedule UI element recognition and interaction layer to execute specific application operations according to process logic, and manage cross-application data streams and state streams generated during process execution. This layer is the "core scheduling hub" of the system, responsible for parsing business process logic, scheduling underlying interaction capabilities, and managing cross-application collaboration rules. It is the core solution to the problem of "how multiple heterogeneous applications can be executed in an orderly manner according to business logic and collaboratively handle exceptions", and realizes full-link automation of "process definition-parsing-scheduling-execution-feedback".

[0034] Process definition mechanism – making business logic configurable and easily adjustable The design employs a "declarative structured configuration" approach, freeing business processes from code and transforming them into editable configuration files. This lowers the operational threshold for non-technical personnel. The core design includes: Structured configuration carrier: Using JSON / YAML as the standard configuration format, it includes three core modules: basic process information (unique ID, business name), global variables (such as "username" and "business document type"), and a step list. Each step in the step list must specify four key elements: "target application type", "preconditions", "operation sequence", and "output variable". For example, the "OA form filling" step in the expense reimbursement process must be configured with the application type as "WEB", the precondition as "user has completed login", the operation sequence as "enter amount → upload voucher → click submit", and the output variable as "unique identifier of expense reimbursement form".

[0035] Flexible node types: Supports four types of core process nodes, covering the logical expression needs of complex business scenarios. Operation nodes define continuous atomic operation combinations for a single application (such as "filling out expense reimbursement forms in the OA system" or "approving documents using C# tools"). Decision nodes implement branching logic based on variable values ​​or execution results (such as "triggering multi-level approval if the reimbursement amount exceeds the threshold, otherwise automatically passing"). Loop nodes repeatedly execute specified operations based on preset conditions (such as "looping through approval status until a 'processed' result is returned"). Sub-process nodes encapsulate common processes (such as "system login" or "data archiving") into independent sub-processes, supporting cross-main process reuse and improving process configuration efficiency.

[0036] Cross-application data transfer mechanism: The "variable placeholder" syntax is used to realize cross-step and cross-application data transfer. For example, the "unique identifier of expense report" variable output by the "OA form filling" step can be directly referenced in the subsequent "C# approval" step by "${unique identifier of expense report}". The variable value is stored in the global data bus for adapters of different applications to read in real time, realizing seamless data transfer between heterogeneous applications.

[0037] Engine core architecture and coordination logic The engine adopts a modular design, including a process parsing module, a collaborative scheduling module, a global data bus module, and a status and exception management module. Each module works together to achieve intelligent execution of the process.

[0038] The process parsing module first verifies the syntactic correctness and logical rationality of the configuration file (such as whether the step dependencies are closed loops and whether the variable definitions are complete), and then converts the configuration file into a "process object model" in memory (including the step execution order, dependencies, and variable mapping rules) to provide structured input for the scheduling module.

[0039] The collaborative scheduling module, as the core execution unit of the engine, implements three core scheduling functions based on the process object model: First, sequential scheduling executes operations in the order defined by the steps. If a step contains preconditions (such as "the previous step must be executed successfully"), it waits for the preconditions to be met before triggering execution. Second, adapter routing extracts the "target application type" from the steps, initiates a call request to the underlying interaction layer, automatically matches the corresponding special adapter, and executes the operation. Third, application lifecycle management automatically starts the target application according to the step requirements (such as automatically starting the financial approval tool when calling the C# adapter), and releases application resources after the process is completed to avoid resource occupation.

[0040] The global data bus module adopts a thread-safe storage mechanism to centrally manage the assignment, reading, and updating of process variables, providing support for cross-application data flow. For example, after the "unique identifier of expense report" extracted by the Web adapter is stored in the data bus, the C# adapter can directly read the variable from the bus and pass it to the approval tool.

[0041] Status and Exception Management Module: Maintains the execution status (idle / running / success / failure / paused) of the process and each step in real time, and synchronizes it to the upper-level management platform; supports pausing, stopping, and resuming the process from a failed step; captures various exceptions in the underlying interaction process (such as element not found, operation timeout, application crash), and automatically executes corresponding operations according to the pre-set handling strategies (retry / skip / terminate the process); at the same time, it records exception context information (execution time, current variable value, locator parameters), providing complete data support for problem tracing.

[0042] The unified control and task management platform provides users with a visual interface to receive process configuration instructions and task control instructions and send them to the intelligent business process orchestration and collaborative execution engine. It also displays the process execution status, data flow details and exception logs received from the engine in real time.

[0043] This layer is the only "entry and exit" for user interaction with the system. It uses PyQt5 to build a visual desktop interface, encapsulating the underlying interactive capabilities and the middle-layer engine scheduling logic into intuitive operation functions, and realizing full lifecycle management of "process configuration - task execution - status monitoring - log traceability".

[0044] Core functional modules and interaction logic The platform's functions are deeply integrated with the mid-level engine, forming an interactive closed loop of "user operation - platform forwarding - engine execution - platform display." Core modules include: User authentication and access control module: Supports username / password login verification and integrates RBAC (role-based access control) mechanism - administrators have full operation permissions such as process editing and permission configuration, while operators only have limited permissions such as process execution and log viewing, ensuring the security and standardization of system operation.

[0045] The process management module includes: a process library that displays all structured configuration files categorized by business type and supports quick retrieval functions such as keyword search and tag classification; a visual editor that generates process configuration files by dragging and dropping nodes (operations / judgments / loops) and filling in parameter forms (application type, operation sequence), which verifies syntax and logic errors in real time, replacing the complex operation of manually writing JSON / YAML; and parameter configuration that assigns values ​​to global variables of the process (such as inputting "username" and "business processing date"), and the assigned parameters are passed to the engine's global data bus through the platform.

[0046] Task control module: Provides core operation buttons such as "start / pause / stop". After the user triggers the operation, the platform sends the corresponding instruction to the engine's collaborative scheduling module. At the same time, it displays the task execution progress in real time (such as "Executing step 2 of 3") and the current operation object (such as "Operating C# financial approval tool"). The status data is synchronized from the engine's status management module in real time.

[0047] Log and Report Module: Receives log information pushed by the engine exception handling module and displays it in DEBUG / INFO / WARNING / ERROR categories; supports exporting logs to TXT / HTML format and automatically generates execution reports (including step execution results and time distribution), enabling traceability and analysis of the execution process.

[0048] Performance optimization design The "master-slave thread separation" architecture is adopted to avoid UI lag issues: the task execution logic (calling the engine and the underlying interaction layer) is deployed on the slave thread, while the master thread is only responsible for UI updates and user operation responses; the threads communicate securely through the signal-slot mechanism of PyQt5 to ensure that data such as execution status and log information are synchronized to the interface in real time, and the execution dynamics are displayed without delay.

[0049] Example 2: This embodiment uses a typical "cross-application automated financial reimbursement process" as an example to explain in detail the implementation process of the method of the present invention. This scenario requires the collaboration of three types of heterogeneous applications: a web-based OA system (for filling out reimbursement forms), a C# financial approval tool (for document approval), and a Java report generator (for summary reports). The complete business chain is "OA form filling → C# approval → Java report generation".

[0050] This embodiment provides a method for automating and collaboratively executing heterogeneous application software, including the following steps: Step S1: Receive the user-configured structured process definition and task start command through the unified control and task management platform; After logging into the system, users select a predefined "Financial Reimbursement Process" from the platform's process library. The structured configuration file (expense_flow.json) for this process is already stored in the system. Users assign values ​​to global variables in the parameter configuration interface, for example: username: "admin", password: "******", expense_amount: "500". After assigning values, users click the "Start Task" button, and the platform sends this start command along with the assigned process configuration to the intelligent business process orchestration and collaborative execution engine.

[0051] Step S2: The intelligent business process orchestration and collaborative execution engine parses the structured process definition and generates an executable process object model; The engine's process parsing module receives and parses the expense_flow.json file. The core content of this file is shown below. Figure 2 As shown; The parsing module first performs syntax and logic checks, such as checking whether the step dependencies are closed loops. Then, it converts the configuration file into an in-memory process object model, which specifies the execution order of the steps (WEB-OA-SUBMIT first, then C#-APPROVE), dependencies (the second step depends on the success of the first step), and data reference relationships defined by variable placeholders (such as ${expense_id}).

[0052] Step S3: The engine schedules the UI element recognition and interaction layer to perform an operation sequence on the specified heterogeneous application according to the logical order of the process object model; The engine's collaborative scheduling module, based on the process object model, begins execution of the first step (WEB-OA-SUBMIT). It identifies the target application type as WEB and, through adaptation scheduling logic, calls the Web application adapter in the UI element identification and interaction layer. The scheduling module encapsulates the operation sequence (e.g., input_text) and locators (e.g., {"type": "ID", "value": "username"}) into a standardized request and sends it to the adapter. Upon receiving the request, the Web application adapter (based on Selenium) automatically opens a browser, navigates to the OA login page, and sequentially executes operations such as entering username and password, clicking login, filling in the reimbursement amount, and clicking submit. Finally, it executes the get_text operation to extract the reimbursement form ID (e.g., EXP20240520001) from the page. This output data, expense_id, is immediately uploaded and stored in the engine's global data bus module.

[0053] Step S4: During the execution process, the UI element recognition and interaction layer uploads the output data defined in the operation result to the global data bus of the engine for storage; After the first step is executed successfully, the collaborative scheduling module verifies that the preconditions for the second step have been met. It retrieves the output data ${expense_id} stored in the previous step from the global data bus, and its value EXP20240520001 will be used as an input parameter. Next, the scheduling module automatically starts the C# financial approval tool and calls the C# / .NET desktop application adapter (based on pywin32 / UIAutomation). The adapter fills the expense report ID into the designated text box in the approval tool and clicks the "Approval Passed" button. This completes the seamless transfer of data from the web application to the C# desktop application.

[0054] Step S5: When the engine executes subsequent steps, it obtains the output data stored in the previous steps from the global data bus and passes it as an input parameter to the UI element recognition and interaction layer to realize cross-application data flow. Step S6: The engine feeds back the execution status of the entire process, data flow details and abnormal information to the unified control and task management platform in real time for visual monitoring and recording.

[0055] Throughout the entire process, the engine's status and exception management module maintains the status of each step in real time (e.g., "Running," "Successful"). Simultaneously, any operational details (e.g., "Successfully retrieved expense report ID: EXP20240520001") or exception information (e.g., "Element not found") are recorded. All this status data and log information is fed back to the unified control and task management platform in real time. The platform's task control module updates the progress bar in real time (e.g., "Executing step 2 of 2 steps"), while the log and report module dynamically displays hierarchical logs, allowing users to view task status or export detailed reports at any time.

[0056] Example 3: This embodiment proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements an automated operation and collaborative execution method for heterogeneous application software as described in any embodiment of the present invention.

[0057] Example 4: This embodiment proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements an automated operation and collaborative execution method for heterogeneous application software as described in any embodiment of the present invention.

[0058] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0059] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An automated operation and collaborative execution system for heterogeneous application software, characterized in that, include: The system comprises a UI element recognition and interaction layer, an intelligent business process orchestration and collaborative execution engine, and a unified control and task management platform; among which: The UI element recognition and interaction layer is used to provide standardized UI element positioning and interaction capabilities for different types of heterogeneous applications through the adapter pattern. Its operation results and status data are output to the intelligent business process orchestration and collaborative execution engine. The intelligent business process orchestration and collaborative execution engine is used to receive and parse structured process configurations from the unified control and task management platform, schedule UI element recognition and interaction layer to execute specific application operations according to process logic, and manage cross-application data streams and state streams generated during process execution. The unified control and task management platform provides users with a visual interface to receive process configuration instructions and task control instructions and send them to the intelligent business process orchestration and collaborative execution engine. It also displays the process execution status, data flow details and exception logs received from the engine in real time.

2. The automated operation and collaborative execution system for heterogeneous application software according to claim 1, characterized in that: The UI element recognition and interaction layer includes a unified abstract interface, multiple specialized adapters, and adaptation scheduling logic. A unified abstract interface is used to define a set of standard UI operation methods that are independent of the technology stack; Multiple specialized adapters, corresponding to different types of heterogeneous applications, are used to implement the unified abstract interface and convert standard operation methods into native interactive instructions for specific technology stacks; The adaptation scheduling logic is used to route requests to the corresponding special adapters for execution based on the application type identifier in the operation requests issued by the intelligent business process orchestration and collaborative execution engine.

3. The automated operation and collaborative execution system for heterogeneous application software according to claim 2, characterized in that: The specialized adapters include Web application adapters, Java desktop application adapters, and C# / .NET desktop application adapters; The Web application adapter integrates Selenium WebDriver for interacting with web browser applications; The Java desktop application adapter is based on the PyJab library and interfaces with the Java Accessibility API for interacting with Java desktop applications; The C# / .NET desktop application adapter uses pywin32 to call the Win32 API or the Windows UIAutomation framework to interact with the C# desktop application.

4. The automated operation and collaborative execution system for heterogeneous application software according to claim 1, characterized in that: The intelligent business process orchestration and collaborative execution engine includes a process parsing module, a collaborative scheduling module, a global data bus module, and a status and exception management module. The process parsing module is used to verify and parse the structured process configuration, and generate a process object model that includes the execution order of steps, dependencies, and data reference relationships defined by variable placeholders. The collaborative scheduling module is used to call the UI element recognition and interaction layer to execute each step operation in sequence according to the process object model, and to manage the temporal dependencies between steps. The global data bus module is used to centrally store and manage variable data generated during process execution, enabling data flow across steps and applications; The status and exception management module is used to maintain and update the execution status of processes and steps in real time, capture and handle exceptions during execution, and synchronize status and exception information to the unified control and task management platform.

5. The automated operation and collaborative execution system for heterogeneous application software according to claim 4, characterized in that: The structured process configuration defines the sequence of process steps, the target application type for each step, the operation sequence, input variables, and output variables.

6. The automated operation and collaborative execution system for heterogeneous application software according to claim 1, characterized in that: The unified control and task management platform includes a user authentication and permission control module, a process management module, a task control module, and a log and report module; The user authentication and access control module is used to implement user login verification and role-based access control. The process management module is used to provide process library management, visual process editor and process parameter configuration functions; The task control module is used to receive the user's task control instructions and send them to the engine, while displaying the task execution progress and status in real time. The log and report module is used to receive, hierarchically display and export system execution logs, and generate execution reports.

7. The automated operation and collaborative execution system for heterogeneous application software according to claim 1, characterized in that: The system adopts a multi-threaded architecture. The unified control and task management platform runs on the main thread and is responsible for interface response. The intelligent business process orchestration and collaborative execution engine and the task execution logic of the UI element recognition and interaction layer run on the slave thread. The main thread and slave thread communicate through the signal-slot mechanism to decouple interface operation from task execution.

8. A method for automated operation and collaborative execution of heterogeneous application software based on the system described in any one of claims 1 to 7, characterized in that, Includes the following steps: Step S1: Receive the user-configured structured process definition and task start command through the unified control and task management platform; Step S2: The intelligent business process orchestration and collaborative execution engine parses the structured process definition and generates an executable process object model; Step S3: The engine schedules the UI element recognition and interaction layer to perform an operation sequence on the specified heterogeneous application according to the logical order of the process object model; Step S4: During the execution process, the UI element recognition and interaction layer uploads the output data defined in the operation result to the global data bus of the engine for storage; Step S5: When the engine executes subsequent steps, it obtains the output data stored in the previous steps from the global data bus and passes it as an input parameter to the UI element recognition and interaction layer to realize cross-application data flow. Step S6: The engine feeds back the execution status of the entire process, data flow details and abnormal information to the unified control and task management platform in real time for visual monitoring and recording.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the automated operation and collaborative execution method for heterogeneous application software as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automated operation and collaborative execution method for heterogeneous application software as described in claim 8.

Images