Window element grabbing method for ai agent combining rpa, ai and llm and related product

Abstract

The present disclosure provides a window element grabbing method for realizing an AIAgent by combining RPA, AI and LLM and related products. The method is applied to an agent AIAgent, and the method comprises: judging whether a current process runs at least one target window based on a preset target framework; in response to determining that the current process runs at least one target window, identifying a target window element with a framework identifier in each target window according to a preset framework identifier corresponding to the target framework; and grabbing each target window element. This embodiment can obtain the element composition structure of each window when the process runs the window by driving the robot process automation PRA, and in the case that the artificial intelligence AI technology judges that there is a target window with an incomplete element composition structure, the target window element in the target window can be identified and grabbed, thereby effectively grabbing the unstructured target window element in the target window.

Description

Technical Field

[0001] The embodiments of this application relate to the field of information technology, and in particular to a method and related products for capturing window elements using an AIAgent by combining RPA, AI, and LLM. Background Technology

[0002] Robotic Process Automation (RPA) uses specific "robot software" to simulate human operations on a computer and automatically execute process tasks according to rules.

[0003] Artificial intelligence (AI) is a technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0004] Large Language Models (LLMs) are models trained on massive amounts of text that can recognize human language, perform language-related tasks, and have a large number of parameters.

[0005] Artificial Intelligence Agents (AI Agents) are capable of perceiving their environment, making decisions, and executing actions. Unlike traditional artificial intelligence, they possess the ability to think and act independently, and can utilize tools to achieve given goals. AI Agents are based on Large Language Models (LLMs) as their core computing engine, enabling them to engage in dialogue, perform tasks, reason, and exhibit a degree of autonomy. They possess the ability to autonomously understand, perceive, plan, remember, and use tools, and can automate complex tasks. Specifically, LLM-driven AI Agents, composed of various AI capabilities, can interact with employees using natural language, understand employee instructions and needs, and provide feedback and responses; they can acquire domain-specific knowledge relevant to the business to complete complex professional tasks; they can break down complex tasks into several executable tasks and use data and tools to complete them; they can also collaborate with employees, and AI Agents can collaborate with each other to complete complex tasks, enabling digital employees to leap from automation to intelligence, helping employees complete their work more efficiently, and fully realizing human-machine collaboration.

[0006] For processes executed by electronic devices, the process often runs multiple windows simultaneously. In some cases, the multiple windows running in the same process can be windows based on different frames. In this case, when capturing the various window elements in each window of different frames, an application for capturing window elements is usually used.

[0007] However, since each window is based on a different frame, the application used to capture window elements is not applicable to windows of all frames. For example, for windows based on some special frames, their window elements are not naturally exposed due to their rendering mechanism or other reasons.

[0008] Therefore, in this case, it is usually possible to effectively capture window elements within a portion of the window. When capturing window elements within another portion of the window, it is difficult to effectively capture these windows through the same application because the latter portion of the window is based on a special frame.

[0009] In this situation, the upper-level interface of the window is usually obtained first, and the interface elements corresponding to each window element in the upper-level interface are identified by Optical Character Recognition (OCR) to achieve crawling. However, the OCR method usually has a high rejection rate and false recognition rate, and it is also difficult to achieve automated crawling of each window element through RPA. Summary of the Invention

[0010] In view of this, embodiments of this disclosure propose a method and related products for window element capture of an AI Agent by combining RPA, AI, and LLM.

[0011] In a first aspect, embodiments of this disclosure provide a method for capturing window elements in an AI Agent by combining RPA, AI, and LLM, applied to an intelligent AI Agent. The method includes: Determine whether the current process is running at least one target window based on a preset target frame, where the target window of the target frame does not contain structured window elements; In response to determining that the current process is running at least one target window, the target window elements with frame identifiers are identified in each target window according to the preset frame identifiers corresponding to the target frames; Capture elements from each target window.

[0012] In some optional implementations, determining whether the current process is running at least one target window based on a preset target frame includes: The Robotic Process Automation (RPA) program calls a preset element-grabbing automation process program to obtain the element composition structure of each window currently running in the process. Determine if there is a window with an incomplete elemental structure; In response to the determination of existence, the window with an incomplete element composition structure is identified as the target window.

[0013] In some optional implementations, determining whether the current process is running at least one target window based on a preset target frame includes: Based on artificial intelligence (AI) technology, it can determine whether there are windows with preset target frame characteristics in each window of the currently running process; In response to the determination of existence, the window with the characteristics of the target frame is identified as the target window.

[0014] In some alternative implementations, target window elements with frame identifiers are identified in each target window, including: Invoke the preset process debugger to inject the preset binary script into the current process; In response to determining that the current process is running a binary script, the target window elements in the target window are identified through the preset target interface corresponding to the target frame.

[0015] In some alternative implementations, each target window element is extracted, including: In response to the determination of the binary script to identify each target window element in each target window, the target window elements are sent to the preset element viewing program using the preset interactive interface; In response to determining that the element composition structure of each target window element in any target window is complete, the element fetching automated process program uses a preset interactive interface to fetch the complete element composition structure of each target window from the structure reconstruction program.

[0016] Accordingly, the element composition structure includes a tree-like hierarchical structure; and The system uses a pre-defined interactive interface to send each target window element to a pre-defined element viewing program, including: The pre-defined Large Language Model (LLM) is invoked to assemble the elements of each target window into a tree-like hierarchical structure.

[0017] Accordingly, the automated element-grabbing process uses a pre-defined interactive interface to retrieve the complete elemental structure of each target window from the structure reconstruction program, including: The automated process of element capture is used to capture each element of each target window, which is represented by a tree hierarchy.

[0018] Secondly, embodiments of this disclosure provide a window element capture device that combines RPA, AI, and LLM to implement an AI Agent. The device includes: a target window determination module, a target window element recognition module, and a capture module. The target window determination module is configured to determine whether the current process is running at least one target window based on a preset target frame, and the target window of the target frame does not have structured window elements; The target window element recognition module is configured to, in response to determining that at least one target window is running in the current process, identify the target window elements with frame identifiers in each target window according to the preset frame identifiers corresponding to the target frames; The capture module is configured to capture elements from each target window.

[0019] In some alternative implementations, determining the target window module includes: The element-grabbing unit is configured to drive Robotic Process Automation (RPA) and call a preset element-grabbing automation process program to obtain the element composition structure of each window running in the current process. The judgment unit is configured to determine whether there is a window whose element structure is incomplete. The target window determination unit is configured to determine a window with an incomplete element composition structure as the target window in response to the determination of its existence.

[0020] In some optional implementations, the target window element recognition module is further configured as follows: Based on artificial intelligence (AI) technology, it can determine whether there are windows with preset target frame characteristics in each window of the currently running process; In response to the determination of existence, the window with the characteristics of the target frame is identified as the target window.

[0021] In some optional implementations, the target window element recognition module is further configured as follows: Invoke the preset process debugger to inject the preset binary script into the current process; In response to determining that the current process is running a binary script, the target window elements in the target window are identified through the preset target interface corresponding to the target frame.

[0022] In some alternative implementations, the capture module includes: The target window element sending unit is configured to send each target window element to a preset element viewing program in response to the determination of the binary script to identify each target window element in each target window using a preset interactive interface; The target window element capture unit is configured to, in response to determining that the element composition structure of each target window element in any target window is complete, use the element capture automated process program to capture the complete element composition structure of each target window from the structure reconstruction program through a preset interactive interface.

[0023] Accordingly, the element composition structure includes a tree-like hierarchical structure; and The system uses a pre-defined interactive interface to send each target window element to a pre-defined element viewing program, including: The pre-defined Large Language Model (LLM) is invoked to assemble the elements of each target window into a tree-like hierarchical structure.

[0024] The automated element-grabbing process uses a pre-defined interactive interface to extract the complete elemental structure of each target window from the structure reconstruction program, including: The automated process of element capture is used to capture each element of each target window, which is represented by a tree hierarchy.

[0025] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0026] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation of the first aspect.

[0027] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the implementations of the first aspect.

[0028] To address the problem of ineffective capture of non-naturally exposed window elements based on special structures, embodiments of this disclosure provide a window element capture method and related products that combine RPA, AI, and LLM to implement an AI Agent. By accurately identifying target windows based on preset target frames in the current process, and then recognizing and capturing target window elements based on frame identifiers, the logic of feature matching and identifier recognition of target frames can achieve targeted capture of window elements based on special frames. This ensures that when a process runs multiple windows with different frames, the element information of the target window can be obtained in a targeted manner, improving the comprehensiveness and applicability of window element capture and providing basic support for subsequent automated operations. Attached Figure Description

[0029] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure may be applied; Figure 2AA flowchart of an embodiment of a window element crawling method for implementing an AI Agent by combining RPA, AI, and LLM according to this disclosure; Figure 2B This is a flowchart illustrating an embodiment of the decomposition process 2010 according to one embodiment of step 201 of this disclosure; Figure 2C This is an exploded flowchart of an embodiment of the process 2020 according to one embodiment of step 202 of the present disclosure; Figure 2D This is an exploded flowchart of one embodiment of the process 2030 according to one embodiment of step 203 of the present disclosure; Figure 3 This is a schematic diagram of a window element grabbing device according to an embodiment of the present disclosure that combines RPA, AI, and LLM to implement an AI Agent; Figure 4 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0030] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0031] In the description of this disclosure, the term "multiple" means two or more.

[0032] In the description of this disclosure, the term "core" refers to the core engine. The browser kernel is the underlying component that enables the browser to perform core functions such as web page rendering, script execution, and resource loading. It is the core of the browser's operations such as processing web page data and building UI interfaces.

[0033] In the description of this disclosure, the term "user interface (UI)" refers to the set of visual interfaces and elements that a user sees and operates when interacting with software, applications, or electronic devices.

[0034] In the description of this disclosure, the term "Desktop Bus (DBUS)" refers to a message bus system in an operating system used for inter-process communication. Applications such as browsers can send UI element structures via DBUS, which are then encapsulated through an Assistive Technology Service Provider Interface (AT-SPI) interface. Accessibility testing tools receive and parse this data via DBUS to identify the application's interface structure.

[0035] In the description of this disclosure, the term "Assistive Technology Service Provider Interface (AT-SPI)" is a standard interface for interacting with applications. Applications send their UI element structures, such as buttons, dialog boxes, and text boxes, to the DBUS bus via AT-SPI, and accessibility testing tools then retrieve this structure information through the bus.

[0036] In the description of this disclosure, the term "Accessibility Tester (Accerciser)" is an open-source interactive accessibility exploration tool that relies on the AT-SPI interface and DBUS bus to read structured UI objects (UI elements) or window objects (window elements), but cannot modify the process or complete the structure of UI objects (UI elements) or window objects (window elements).

[0037] In the description of this disclosure, the term "element crawling automation process program (Creator)" refers to a class of RPA process development tools whose function is to visually build automated processes, thereby automatically executing repetitive computer operations according to the process. For example, it uses technical means to identify UI elements of target applications such as browsers, obtain information such as the position and attributes of the elements, and then completes element crawling. In this process, it relies on the AT-SPI interface to obtain the UI element structure of the browser from the DBUS bus.

[0038] In the description of this disclosure, the term "GIMP Drawing Kit (GDK)" refers to an open-source graphics library used for functions such as graphics rendering, window creation, and drawing user interface elements. For example, the open file dialog box used by a browser process to open local files is developed based on the GDK framework, rather than native browser code, which is the core reason why traditional tools cannot capture elements in the open file dialog box.

[0039] GIMP refers to the GNU Image Manipulation Program, an open-source, cross-platform professional image processing tool that provides image editing capabilities. GNU represents a type of operating system.

[0040] In this disclosure, the term "GNU Debugger (GDB)" refers to an open-source program debugging tool that supports operations such as setting breakpoints, viewing memory, and injecting code into running processes. It can be used to inject the UI element of the "Open File" dialog box from a browser process into the browser process, thereby reattaching the UI element of the "Open File" dialog box to the browser.

[0041] The implementation of the Digital Employee Platform (WEP) has gone through three stages. The first stage is automation: targeting RPA (Robotic Process Automation) with low business complexity, it utilizes software automation technology to automate rule-based, predefined procedural tasks. The second stage is intelligence: leveraging AI to extend the boundaries of RPA, such as processing unstructured documents and making data-driven decisions. The third stage is human-machine collaboration: utilizing the understanding, planning, and execution capabilities of large-scale models to automate complex tasks end-to-end.

[0042] In the human-machine collaboration phase, the digital employee platform serves as a bridge connecting workers and systems, workers and data, and systems and data. It is capable of: operating complex systems, processing various types of data, and interacting and collaborating with employees. The digital employee platform helps industries build large-scale, model-enabled digital employees (i.e., intelligent agents), achieving automation, intelligence, and human-machine collaboration in business processes.

[0043] The digital employee platform can seamlessly integrate multiple capabilities such as Agentic Process Automation (APA), Agentic Document Processing (ADP), and Agentic Business Insights (ABI). It has five major functions: "business understanding", "process creation", "run anywhere", "centralized management and control" and "human-machine collaboration". It enables enterprises to achieve end-to-end intelligent automation of business processes, replace manual operations, further improve business efficiency, and accelerate digital transformation.

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Figure 1 An exemplary system architecture 100 is shown, which can be applied to the window element capture method and related products of the present disclosure that combine RPA, AI, and LLM to implement an AI Agent.

[0046] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0047] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various artificial intelligence (AI) based models and services, such as LLM and RPA services, can be installed on terminal devices 101, 102, and 103. These AI-based models and services collectively form an intelligent agent (AIAgent) capable of performing predetermined tasks.

[0048] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays and process execution functions, used to run various windows in processes and display the corresponding interfaces of various windows through the display screen. Terminal devices 101, 102, and 103 include, but are not limited to, smartphones, tablets, e-book readers, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are hardware, each terminal device can independently carry a WEP, on which an AI Agent composed of the aforementioned AI-based models and services is deployed. Alternatively, each terminal device can carry one or more of the aforementioned AI-based models and / or services, and the AI ​​Agent can be composed of the AI-based models and services carried by each of terminal devices 101, 102, and 103.

[0049] When terminal devices 101, 102, and 103 are software, they can be installed on the aforementioned terminal devices and used to run various windows within a process, displaying the corresponding interfaces of these windows on a screen. They can be implemented as multiple software programs or software modules (e.g., used to host an AI Agent composed of the aforementioned models and services). That is, each terminal device as software can act as a WEP (Web Application Program) to deploy a complete AI Agent, or it can be implemented as a single software program or software module. In other words, each terminal device as software can act as a module composed of one or more AI-based models and / or services. No specific limitations are imposed here.

[0050] In some cases, the window element capture method for implementing an AI Agent by combining RPA, AI, and LLM provided in this disclosure can be executed individually by terminal devices 101, 102, and 103, or jointly by terminal devices 101, 102, and 103. Accordingly, the window element capture device for implementing an AI Agent by combining RPA, AI, and LLM can be set in terminal devices 101, 102, and 103. In this case, system architecture 100 may not include server 105.

[0051] In some cases, the window element capture method for implementing an AI Agent by combining RPA, AI, and LLM provided in this disclosure can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, the step of "determining whether the current process is running at least one target window based on a preset target frame" can be executed by terminal devices 101, 102, or 103, and the steps such as "capturing each target window element" can be executed by server 105. This disclosure does not limit this. Correspondingly, the window element capture device for implementing an AI Agent by combining RPA, AI, and LLM can also be respectively set in terminal devices 101, 102, and 103 and server 105.

[0052] In some cases, the window element capture method for AI Agent that combines RPA, AI, and LLM provided in this disclosure can be executed by server 105 alone, and the execution results of server 105 can be displayed through terminal devices 101, 102, and 103. In this case, server 105 carries WEP, on which AI Agent composed of the above-mentioned models and services is deployed, and various windows can run in the process, and the interface corresponding to each window can be displayed on the screen. Correspondingly, terminal devices 101, 102, and 103 can also be set in server 105. In this case, system architecture 100 may not include terminal devices 101, 102, and 103.

[0053] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers. Each server in the server cluster can run various windows in a process and display the corresponding interface of each window on a screen. Alternatively, it can be implemented as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (for example, to provide the aforementioned AI-based models and services, or to provide an AI Agent composed of the aforementioned models and services), or it can be implemented as a single software program or software module. No specific limitations are made here.

[0054] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0055] In some alternative implementations, refer to Figure 2A The document illustrates a flowchart 200 of an embodiment of the window element crawling method for AIAgent that combines RPA, AI, and LLM, which includes the following steps 201 to 203: Step 201: Determine whether the current process is running at least one target window based on a preset target frame.

[0056] A window is an independent visual container defined at the operating system level. It is the basic unit for user interaction with applications and can include multiple elements. For example, in the process of a browser program, the browser's webpage display window and the open file dialog box are both different windows of the browser program process.

[0057] Both the webpage display window and the open file dialog box can contain one or more window elements.

[0058] In this application, when each window is displayed to the user, the operating system-level window can be displayed as the corresponding interface in the display layer. For example, the operating system-level web page display window can be displayed as the corresponding web page display interface, the open file dialog window can be displayed as the corresponding open file dialog interface, and the elements in each window are also displayed as UI elements in the interface.

[0059] In this step, for the currently running process, since each process can run one or more windows at the same time, and since each window may be based on a different structure, the window elements in each window have different characteristics.

[0060] Specifically, some types of frames enable windows to form structured window elements. Windows based on such frames can have their structured window elements completely transmitted to the DBUS bus via AT-SPI, such as the web page display window of a browser developed based on Core.

[0061] Other types of windows lack structured window elements, which prevents these window elements from being directly sent to DBUS via AT-SPI. For example, the open file dialog window based on the GDK framework. In this case, lacking structured window elements can mean that the structure of the window elements is unrecognizable by the current operating system or the program currently used to capture the window elements.

[0062] Based on this, during the process of capturing window elements of each window, windows that do not have structured window elements, as well as windows whose window element structure cannot be recognized by the current operating system or the program currently used to capture window elements, can be used as target windows according to the above characteristics of each window, and the frame of the target window can be used as the corresponding target frame.

[0063] Therefore, by deploying the AI ​​Agent in WEP, it is possible to determine in real time whether one or more target windows are running in the current process, and then, if a target window exists, to capture the elements of the target window and the window elements of other windows in different ways.

[0064] Specifically, in some cases, target frame features can be pre-set for the target frame and used as the benchmark for judging the target window. The AI ​​Agent deployed in WEP can use AI technology to determine whether each window has the above-mentioned target frame features, thereby determining whether the current process has one or more target windows.

[0065] In a specific example, taking the browser process as an example, the browser process runs multiple windows simultaneously. These multiple windows can be, for example, a webpage display window running in the browser process, and an open file dialog box running in the browser process to open local files. Since the open file dialog box is a window based on the GDK framework, the structure of its window elements cannot be recognized by the program that captures window elements. Therefore, the target framework characteristics can be set in advance for the GDK framework.

[0066] Based on this, the AI ​​Agent deployed in WEP can use AI technology to determine the frame similarity between each window and the target frame features, i.e., the frame similarity of the GDK frame. If the frame similarity is greater than or equal to a preset frame similarity threshold, the window is considered to have the target frame features and is identified as the target window.

[0067] Step 202: In response to determining that the current process is running at least one target window, identify the target window elements with frame identifiers in each target window according to the preset frame identifiers corresponding to the target frames.

[0068] In this step, based on the target window determined in step 201 above, during the process of capturing target window elements in the target window, since the target window has a different frame from other windows, and the target window elements in the target window are often not attached to other windows, applications that are not usually used to capture target window elements can usually only recognize the existence of the target window, but cannot recognize the target window elements based on the target frame.

[0069] Therefore, a binary script for capturing target window elements can be pre-set and injected into the current process.

[0070] Based on this, once the target window is identified, this binary script can be used to identify each element of the target window.

[0071] In the process of identifying each target window element, since each target window element in the target window has the characteristics of a corresponding target frame, the characteristics of the target frame of each target window element can be used as frame identifiers. By running a binary script, the frame identifiers of each target window element in the target window can be identified, thereby identifying the target window elements with frame identifiers.

[0072] Step 203: Capture the elements of each target window.

[0073] In this step, based on the target window elements identified in step 202 above, the AI ​​Agent deployed in WEP can capture the identified target window elements by having the current process run a binary script.

[0074] In some cases, at least two windows can run in the current process, including one target window.

[0075] In this scenario, taking the webpage display window and the open file dialog box running in the browser process as examples, the AI ​​Agent deployed in WEP can call the application used to capture window elements to capture the window elements in each window. Since the open file dialog box is based on the GDK framework, when using the application to capture window elements, only the window elements in the structured webpage display window can be captured. However, for the open file dialog box, only the single window element representing the open file dialog box can be captured, and it is difficult to capture the various target window elements inside the open file dialog box.

[0076] Accordingly, the AI ​​Agent deployed in WEP can identify each target window element in the open file dialog box using binary scripts, and then capture each identified target window element.

[0077] In this case, after identifying each target window in the webpage display window and each target window element in the open file dialog box, each target window element can be classified into a sub-level of the single window element representing the open file dialog box, and together with other window elements, form the element composition structure of all window elements in the current process.

[0078] As can be seen, by accurately identifying the target window based on the preset target frame in the current process, and then identifying and capturing the target window elements based on the frame identifier, the logic of feature matching and identifier recognition of the target frame can realize the targeted capture of window elements based on special frames. This ensures that when multiple different frame windows are running in the process, the element information of the target window can be obtained in a targeted manner, improving the comprehensiveness and applicability of window element capture and providing basic support for subsequent automated operations.

[0079] In some alternative implementations, refer to Figure 2B This illustrates a decomposition process 2010 of one embodiment of step 201 of this disclosure. The decomposition process 2010 includes the following steps 2011 to 2013: Step 2011: Drive Robotic Process Automation (RPA) and call the preset element capture automation process program to obtain the element composition structure of each window running in the current process.

[0080] In determining whether a current process is running with a target window, in addition to using AI technology to identify whether each window has the characteristics of a target frame, it is also possible to determine whether a window is a target window by judging whether the element composition structure of each window is complete.

[0081] Specifically, the AI ​​Agent deployed in WEP can drive RPA, thereby automatically calling the Creator to fetch window elements of each window through RPA. The Creator is an automated process program for fetching elements, which is used to automatically fetch window elements of each window in the current process.

[0082] When calling Creator to capture window elements of various windows, for windows other than the target window, Creator can directly capture all structured window elements in that window, such as window elements that form a complete tree structure.

[0083] Taking the webpage display window and open file dialog box in the browser process as examples, Creator can directly capture all the elements of the complete tree structure in the webpage display window, and Creator can display all the window elements of the complete tree structure in its interface.

[0084] As for the Open File dialog box, since Creator cannot directly identify the individual window elements, it cannot extract the element composition structure of each window element in the Open File dialog box. In general, Creator can only identify a single window element that represents the Open File dialog box. That is, under this window element, there are no other window elements that are part of the Open File dialog box task.

[0085] Step 2012: Determine if there are windows with incomplete element composition structures.

[0086] Based on the results of using Creator to capture the window elements of each window in step 2011 above, it can be determined whether the element composition structure of each window element is complete.

[0087] It can be seen that for any window, if only a single window element can be obtained after capturing its window elements, or if no window elements can be captured, then the element composition structure of the captured window elements can be considered incomplete.

[0088] Taking the webpage display window and the open file dialog box in the browser process as an example, after Creator captures the window elements of the webpage display window and the open file dialog box respectively, since the window element of the open file dialog box is only a single window element, it can be considered that the captured element composition structure of the open file dialog box is incomplete.

[0089] Step 2013: In response to the determination of existence, the window with an incomplete element composition structure is identified as the target window.

[0090] Based on the judgment in step 2012 above, if any window has an incomplete element structure, for example, only a single window element exists, it can be considered that Creator cannot recognize and / or capture the window element in that window. Therefore, that window can be identified as the target window.

[0091] Taking the webpage display window and the open file dialog box in the browser process as an example, if Creator can only capture a single window element from the open file dialog box, then the open file dialog box can be identified as the target window.

[0092] Based on this, by leveraging RPA to drive an automated process for element crawling, the target window is judged based on the completeness of its element composition structure, achieving accurate filtering of the target window. Without the need for manual intervention, window type identification can be completed entirely through the automated process, ensuring the objectivity and consistency of the judgment results while saving the time cost of manual filtering. At the same time, using the completeness of the element composition structure as the criterion directly relates to the crawlability of window elements, providing a clear basis for subsequent targeted crawling operations and improving the coherence and reliability of the overall process.

[0093] In some alternative implementations, further reference is made. Figure 2C The diagram illustrates a flow 2020 of one embodiment of step 202 of this disclosure. This flow 2020 includes steps 2021 to 2022: Step 2021: Invoke the preset process debugger to inject the preset binary script into the current process.

[0094] Based on one or more target windows identified by Creator, during the process of identifying each target window element in the target window, a pre-set GDB can be invoked. By invoking GDB, GDB can further invoke a pre-set binary script and inject the binary script into the current process, so that the binary script can be run through the current process in subsequent steps.

[0095] Step 2022: In response to determining that the current process is running a binary script, identify each target window element in the target window through the preset target interface corresponding to the target frame.

[0096] Based on the binary script injected into the current process in step 2021 above, the binary script can be run in the current process.

[0097] Furthermore, during the execution of this binary script, the target interface can be determined, which is the interface corresponding to the target window under the target frame.

[0098] Based on this, by executing this binary script, the various target window elements in the target window can be identified through the target interface.

[0099] Taking the open file dialog box in the browser process as an example, based on the binary script running in the browser process, the GDK interface pointing to the open file dialog box of the GDK architecture can be used as the target structure.

[0100] Therefore, after identifying the GDK interface, it is possible to further identify various target window elements with GDK framework characteristics in the open file dialog box through the GDK interface.

[0101] Based on this, a binary script was injected into the current process using a process debugger, and then the target window elements were identified using the target interface corresponding to the target framework. This successfully solved the problem of the difficulty in identifying unstructured window elements. The injection of the binary script enabled targeted intervention in the current process, ensuring the effective execution of the identification logic. The targeted calls to the target interface made element identification more accurate, avoiding interference from irrelevant elements, and enabling each element in the target window to be accurately identified, laying a solid foundation for subsequent crawling operations.

[0102] In some alternative implementations, further reference is made. Figure 2D The process 2030 of one embodiment of step 203 of this disclosure includes the following steps 2031 to 2032: Step 2031: In response to the determination that the binary script identifies each target window element in each target window, the target window elements are sent to the preset element viewing program using the preset interactive interface.

[0103] Based on the target window elements identified by the binary script in the aforementioned steps, by running the binary script, each target window element can be captured and sent to the preset Accerciser.

[0104] In the process of sending each target window element to Accerciser, the binary script can complete the sending through a pre-set interactive interface, which can be an interface applicable to all windows for encapsulating and sending window elements.

[0105] Taking the webpage display window and the open file dialog box in the browser process as examples, both the webpage display window and the open file dialog box support the AT-SPI interface, that is, they can both transmit their respective window elements through the AT-SPI interface. Therefore, the AT-SPI interface can be used as an interaction interface.

[0106] Accordingly, after identifying each target window element in the open file dialog box, the binary script can send each target window element to Accerciser via the AT-SPI interface.

[0107] In the process of sending each target window element to Accerciser, since the open file dialog box may contain multiple different target window elements, and there may be a certain hierarchical relationship between different target window elements, for example, the open file dialog box itself can be the root node in a tree hierarchy, where "local storage" in the target window element is the next level node of the root node, and "first folder" in the target window element is the next level node of "local storage". Based on this, the target window elements identified in the preset LLM open file dialog box can be reconstructed into a tree hierarchy and used as the element composition structure of the open file dialog box.

[0108] During the process of rebuilding the tree hierarchy in LLM, LLM can identify the semantics of each target window element in the open file dialog box, and construct the structural logic of each target window element into a tree hierarchy based on the semantics.

[0109] In other cases, during the construction of a tree hierarchy, based on the identified target window elements, each target window element can also be encapsulated into a tree hierarchy using AT-SPI.

[0110] Step 2032: In response to determining that the element composition structure of each target window element in any target window is complete, the element capture automated process program captures the complete element composition structure of each target window from the structure reconstruction program through a preset interactive interface.

[0111] Based on the target window elements of the target window obtained by Accerciser in step 2031 above, Creator can identify each target window element in the tree hierarchy from Accerciser. Accordingly, Creator can effectively capture each target window element of the target window from Accerciser and display it in a tree hierarchy.

[0112] During the process of Creator fetching various target window elements, Creator can obtain various target window elements in a tree hierarchy from Accerciser through the preset DBUS.

[0113] Based on this, the identified target window elements are sent to Accerciser via AT-SPI. After confirming that the element composition structure is complete, Creator completes the capture, ensuring the integrity and reliability of the capture results. This process avoids capture failures due to missing element structures through Accerciser's intermediate verification. At the same time, efficient data transmission between different programs is achieved through the interactive interface, allowing the complete element composition structure to be accurately obtained, providing high-quality data support for further processing of the target window elements.

[0114] Further reference Figure 3 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a window element crawling device that combines RPA, AI, and LLM to implement an AIAgent. This device embodiment is similar to... Figure 2A Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0115] like Figure 3 As shown, the window element grabbing device 300 that combines RPA, AI, and LLM to implement AIAgent in this embodiment includes: a target window judgment module 301, a target window element recognition module 302, and a grabbing module 303; The target window determination module 301 is configured to determine whether the current process is running at least one target window based on a preset target frame, and the target window of the target frame does not have structured window elements; The target window element recognition module 302 is configured to, in response to determining that at least one target window is running in the current process, identify the target window elements with frame identifiers in each target window according to the preset frame identifiers corresponding to the target frames; The capture module 303 is configured to capture elements of each target window.

[0116] In this embodiment, the specific processing of the target window judgment module 301, target window element recognition module 302, and grasping module 303 of the window element grasping device 300 that combines RPA, AI, and LLM to implement AIAgent can be found in the following references. Figure 2A The relevant descriptions of steps 201, 202 and 203 in the corresponding embodiments will not be repeated here.

[0117] In some optional implementations, the target window determination module 301 includes: The element-grabbing unit is configured to drive Robotic Process Automation (RPA) and call a preset element-grabbing automation process program to obtain the element composition structure of each window running in the current process. The judgment unit is configured to determine whether there is a window whose element structure is incomplete. The target window determination unit is configured to determine a window with an incomplete element composition structure as the target window in response to the determination of its existence.

[0118] In some optional implementations, the target window element recognition module 302 is further configured as follows: Based on artificial intelligence (AI) technology, it can determine whether there are windows with preset target frame characteristics in each window of the currently running process; In response to the determination of existence, the window with the characteristics of the target frame is identified as the target window.

[0119] In some optional implementations, the target window element recognition module 302 is further configured as follows: Invoke the preset process debugger to inject the preset binary script into the current process; In response to determining that the current process is running a binary script, the target window elements in the target window are identified through the preset target interface corresponding to the target frame.

[0120] In some alternative implementations, the capture module 303 includes: The target window element sending unit is configured to send each target window element to a preset element viewing program in response to the determination of the binary script to identify each target window element in each target window using a preset interactive interface; The target window element capture unit is configured to, in response to determining that the element composition structure of each target window element in any target window is complete, use the element capture automated process program to capture the complete element composition structure of each target window from the structure reconstruction program through a preset interactive interface.

[0121] Accordingly, the element composition structure includes a tree-like hierarchical structure; and The system uses a pre-defined interactive interface to send each target window element to a pre-defined element viewing program, including: The pre-defined Large Language Model (LLM) is invoked to assemble the elements of each target window into a tree-like hierarchical structure.

[0122] The automated element-grabbing process uses a pre-defined interactive interface to extract the complete elemental structure of each target window from the structure reconstruction program, including: The automated process of element capture is used to capture each element of each target window, which is represented by a tree hierarchy.

[0123] It should be noted that the implementation details and technical effects of each module in the window element grabbing device that combines RPA, AI, and LLM to implement an AI Agent provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0124] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 4 The computer system 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0125] like Figure 4 As shown, the computer system 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the computer system 400. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0126] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows computer system 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4A computer system 400 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0127] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0128] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0130] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2A The illustrated embodiments and their alternative implementations demonstrate a method for capturing window elements using an AIAgent by combining RPA, AI, and LLM.

[0131] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the unit itself; for example, a target window determination module can also be described as "a module for determining whether the current process is running at least one target window based on a preset target frame".

[0134] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for window element crawling in an AI Agent that combines RPA, AI, and LLM, characterized in that, The method, applied to an intelligent agent (AI agent), includes: Determine whether the current process is running at least one target window based on a preset target frame, wherein the target window of the target frame does not have structured window elements; In response to determining that the current process is running at least one of the target windows, the target window elements with the frame identifiers in each target window are identified according to the preset frame identifiers corresponding to the target frames; The identification of target window elements bearing the frame identifier in each target window includes: Invoke the preset process debugger to inject the preset binary script into the current process; In response to determining that the current process is running the binary script, the target window elements in the target window are identified through the preset target interface corresponding to the target frame; Capture elements from each target window; The elemental structure includes a tree-like hierarchical structure; The process of capturing each target window element includes: In response to determining that the binary script has identified each target window element in each target window, a preset large language model (LLM) is invoked to form the tree-like hierarchical structure of each target window element corresponding to each target window. In response to determining that the element composition structure of each target window element in any target window is complete, the element-grabbing automated process program is used to grab each target window element of each target window represented by the tree hierarchy structure.

2. The method according to claim 1, characterized in that, The step of determining whether the current process is running at least one target window based on a preset target frame includes: The Robotic Process Automation (RPA) is driven to call a preset element-grabbing automated process program to obtain the element composition structure of each window running in the current process. Determine if there is a window with an incomplete elemental structure; In response to the determination of existence, the window with an incomplete element composition structure is identified as the target window.

3. The method according to claim 1, characterized in that, The step of determining whether the current process is running at least one target window based on a preset target frame includes: Based on artificial intelligence (AI) technology, it can determine whether there are windows with preset target frame characteristics in each window of the currently running process; In response to the determination of existence, a window possessing the target frame characteristics is identified as the target window.

4. A window element grabbing device that combines RPA, AI, and LLM to implement an AI Agent, characterized in that, include: The target window determination module is configured to determine whether the current process is running at least one target window based on a preset target frame, wherein the target window of the target frame does not have structured window elements; The target window element identification module is configured to, in response to determining that at least one target window is running in the current process, identify the target window elements in each target window that have the frame identifier according to the preset frame identifier corresponding to the target frame; The identification of target window elements bearing the frame identifier in each target window includes: Invoke the preset process debugger to inject the preset binary script into the current process; In response to determining that the current process is running the binary script, the target window elements in the target window are identified through the preset target interface corresponding to the target frame; The capture module is configured to capture elements from each target window. The elemental structure includes a tree-like hierarchical structure; The process of capturing each target window element includes: In response to determining that the binary script has identified each target window element in each target window, a preset large language model (LLM) is invoked to form the tree-like hierarchical structure of each target window element corresponding to each target window. In response to determining that the element composition structure of each target window element in any target window is complete, the element-grabbing automated process program is used to grab each target window element of each target window represented by the tree hierarchy structure.

5. The apparatus according to claim 4, characterized in that, The target window determination module includes: The element-grabbing unit is configured to drive Robotic Process Automation (RPA) and call a preset element-grabbing automation process program to obtain the element composition structure of each window running in the current process. The judgment unit is configured to determine whether there is a window whose element structure is incomplete. The target window determination unit is configured to determine a window with an incomplete element composition structure as the target window in response to the determination of its existence.

6. The apparatus according to claim 4, characterized in that, The crawling module includes: The target window element sending unit is configured to send each target window element to a preset element viewing program in response to the determination of the binary script to identify each target window element in each target window using a preset interactive interface; The target window element capture unit is configured to, in response to determining that the element composition structure of each target window element in any target window is complete, use the element capture automated process program to capture the complete element composition structure of each target window from the structure reconstruction program through a preset interactive interface.

7. A window element capture device that combines RPA, AI, and LLM to implement an AI Agent, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1-3.

9. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1-3.

Images