Process automation method and device and electronic equipment

Abstract

The invention discloses a process automation method and device and electronic equipment. The method comprises the following steps: acquiring a user interface image; target detection and text recognition are conducted on the user interface image through a visual recognition model, visual information corresponding to the user interface image is obtained, and the visual information comprises interface element information and text information in the user interface image; receiving user request information, and processing the user request information and the visual information through a semantic understanding model to generate an operation instruction sequence; and converting the operation instruction sequence into an operation instruction, and automatically simulating a user operation behavior according to the operation instruction. According to the method and the device, the technical problems of relatively poor interface adaptability, relatively weak user intention understanding ability and insufficient robustness in a complex scene due to the fact that an RPA system in related technologies is generally based on fixed interface coordinates or control features are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a process automation method, device, and electronic equipment. Background Art

[0002] As an innovative solution, RPA (Robotic Process Automation) technology has been widely adopted in numerous business areas, including enterprise office automation, financial accounting, and human resources management. By mimicking human interaction with software interfaces, it automates routine tasks, significantly improving work efficiency and accuracy. Related RPA technologies typically record and replay interfaces based on scripts or predefined rules, which can reduce labor costs and improve work efficiency to a certain extent. However, this approach, which relies on fixed interface coordinates or control features, has the following significant limitations:

[0003] Poor interface adaptability: When the interface layout, theme style, or resolution of the target application changes, the automated script may become ineffective and require frequent manual adjustments.

[0004] Limited command comprehension capabilities: Most RPA systems can only recognize and execute simple pre-set commands. They lack the ability to understand natural language instructions and perform contextual reasoning, making it difficult to handle complex or ambiguous user needs.

[0005] Lack of robustness in complex scenarios: Faced with multi-window, multi-language environments or dynamically generated interface content, the recognition accuracy and execution stability of RPA systems in related technologies are often difficult to meet the requirements, which can easily lead to task execution interruptions or errors.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0007] The embodiments of the present application provide a process automation method, device, and electronic device to at least solve the technical problems that the RPA system in the related art is usually based on fixed interface coordinates or control features, resulting in poor interface adaptability, weak ability to understand user intent, and insufficient robustness in complex scenarios.

[0008] According to one aspect of an embodiment of the present application, a process automation method is provided, including: acquiring a user interface image; performing target detection and text recognition on the user interface image through a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image; receiving user request information, and processing the user request information and visual information through a semantic understanding model to generate an operation instruction sequence; converting the operation instruction sequence into an operation instruction, and simulating user operation behavior based on the operation instruction.

[0009] Optionally, target detection is performed on the user interface image through a visual recognition model, including: performing target detection on the user interface image through a target detection algorithm in the visual recognition model to obtain initial element information, wherein the initial element information includes at least one of the following: element bounding box coordinate information, element confidence and element category of the user interface image; determining the intersection-and-union ratio of all element bounding boxes in the initial element information, and filtering the initial element information according to a preset intersection-and-union ratio and a preset confidence to obtain interface element information.

[0010] Optionally, text recognition is performed on the user interface image through a visual recognition model, including: determining a text candidate area from the interface element information, wherein the text candidate area is a bounding box containing the element category; denoising and binarizing the text candidate area, and identifying the text candidate area through an optical character recognition algorithm in the visual recognition model to obtain text information; and determining the coordinate mapping relationship between the text information and the corresponding bounding box.

[0011] Optionally, the method also includes: processing the user interface image through an edge detection algorithm to obtain edge information; when there is an irregular bounding box in the interface element information, adjusting the irregular bounding box based on the edge information; when there is fuzzy edge text in the text information, restoring the fuzzy edge text based on the edge information.

[0012] Optionally, the user request information and visual information are processed through a semantic understanding model to generate an operation instruction sequence, including: obtaining an execution instruction in the user request information; determining a corresponding prompt word template based on the execution instruction and visual information, wherein the prompt word template is used to specify the output format and constraints of the execution instruction; analyzing the execution instruction, visual information and prompt word template through a semantic understanding model to generate an atomic operation sequence, wherein the atomic operation sequence is used to reflect the logical reasoning process of the user's operation behavior; and determining an operation instruction sequence based on the atomic operation sequence.

[0013] Optionally, converting the operation instruction sequence into an operation instruction includes: obtaining execution parameters in the operation instruction sequence; mapping the execution parameters to corresponding execution functions according to a preset rule base to obtain a parameter mapping result, wherein the preset rule base is used to store a mapping relationship between the execution parameters and the execution functions corresponding to the system operation; generating an executable code snippet based on the parameter mapping result, and generating an operation instruction based on the code snippet.

[0014] Optionally, the method also includes: obtaining a target interface image after the operation instruction is executed; performing a similarity comparison between the target interface image and the user interface image to obtain a similarity difference value, and determining a target change area when the similarity difference value exceeds a preset threshold, wherein the target change area is the area where the user interface image changes after the operation instruction is executed; determining target visual information corresponding to the target change area through a visual recognition model, wherein the target visual information includes updated interface element information and updated text information; processing the user request information and target visual information through a semantic understanding model to obtain an updated operation instruction sequence.

[0015] According to another aspect of an embodiment of the present application, a process automation device is also provided, including: an acquisition module for acquiring a user interface image; an identification module for performing target detection and text recognition on the user interface image through a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image; a generation module for receiving user request information, and processing the user request information and visual information through a semantic understanding model to generate an operation instruction sequence; an operation module for converting the operation instruction sequence into an operation instruction, and automatically simulating user operation behavior based on the operation instruction.

[0016] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned process automation method.

[0017] According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned process automation method by running the computer program.

[0018] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned process automation method when executed by a processor.

[0019] In an embodiment of the present application, a user interface image is obtained; target detection and text recognition are performed on the user interface image through a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image; user request information is received, and the user request information and visual information are processed through a semantic understanding model to generate an operation instruction sequence; the operation instruction sequence is converted into operation instructions, and the user operation behavior is automatically simulated based on the operation instructions, thereby achieving the purpose of intelligently parsing the association between user natural language instructions and dynamic interface elements, thereby realizing the technical effect of high robustness, flexibility and intelligence of automated operations, and further solving the technical problems that the RPA system in the related technology is usually based on fixed interface coordinates or control features, and has poor interface adaptability, weak user intention understanding ability and insufficient robustness in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a hardware structure diagram of a computer terminal for implementing a process automation method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of a process automation method according to an embodiment of the present application;

[0023] Figure 3 This is a structural diagram of a process automation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the process of explaining the embodiments of this application are subject to the following explanations:

[0027] RPA (Robotic Process Automation): aims to automate rule-based business processes through software robots that can simulate human operations on computer systems, such as data entry, record query, document generation, etc., thereby improving efficiency and reducing errors.

[0028] LLM (Large Language Model): An AI model based on deep learning, with a particular emphasis on its large scale, such as a huge number of parameters (up to billions to trillions), and pre-trained on a wide range of text data, giving it powerful natural language understanding and generation capabilities.

[0029] Computer Vision (CV): Computer vision studies how machines can "see" the world. It involves processing, analyzing, and understanding images and videos. The goal is to extract meaningful information from visual data, such as object detection, face recognition, and environmental understanding.

[0030] YOLO algorithm: A real-time object detection algorithm that can simultaneously predict the location and category of an object in a single image inference. It is widely used in the field of computer vision and is highly favored for its balance between speed and accuracy.

[0031] Optical Character Recognition (OCR): A technology that converts text in an image (whether printed, handwritten, or screenshot) into a machine-readable, editable text format.

[0032] Instruction Tuning: A specific type of fine-tuning performed on large language models that enables the model to better understand and follow human instructions, especially in the context of a specific task.

[0033] COT (Chain of Thought): A method that forces a model to reveal its internal reasoning process, making the model's decision logic more transparent, thereby improving its accuracy and coherence in solving complex problems.

[0034] In order to solve the problem of poor process automation efficiency in related technologies, the present invention provides a process automation method that can be run on Figure 1 Among the computer terminals shown, the computer terminal will be described below.

[0035] The process automation method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a process automation method. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0036] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the process automation method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned process automation method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0040] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0041] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a process automation method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Figure 2 is a flow chart of a process automation method according to an embodiment of the present application, such as Figure 2As shown, the method includes the following steps:

[0043] Step S202: Acquire a user interface image.

[0044] Step S204 , performing target detection and text recognition on the user interface image through a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image.

[0045] Step S206: receiving user request information, and processing the user request information and visual information through a semantic understanding model to generate an operation instruction sequence.

[0046] Step S208: convert the operation instruction sequence into an operation instruction, and automatically simulate the user operation behavior according to the operation instruction.

[0047] Through steps S202 to S208 above, the goal of intelligently parsing the association between user natural language commands and dynamic interface elements is achieved, thereby achieving the technical effect of highly robust, flexible, and intelligent automated operations. This further addresses the technical issues of RPA systems in related technologies, which are typically based on fixed interface coordinates or control features, resulting in poor interface adaptability, weak understanding of user intent, and insufficient robustness in complex scenarios. This is explained in detail below.

[0048] In the above step S202, the main task is to capture the interface image that the user is currently interacting with in real time, which serves as the basic input for subsequent visual analysis and semantic understanding. This involves two steps: data acquisition and image preprocessing. The specific analysis is as follows:

[0049] (1) Data Acquisition

[0050] Real-time screenshot: Use system APIs or plug-ins to periodically capture the entire screen, specified window, or browser page to obtain user interface images.

[0051] Video stream input: In scenarios where continuous monitoring of interface changes (such as pop-up windows and dynamic animations) is required, screen video frames can be continuously acquired and key frames (as user interface images) can be extracted for analysis to ensure that the system can capture all important interface events.

[0052] (2) Image preprocessing

[0053] Sizing and formatting: Considering the characteristics of different target interfaces and the input requirements of the visual recognition model, the acquired images need to be resized while keeping the aspect ratio unchanged to avoid image deformation and affect the accuracy of subsequent detection.

[0054] Denoising and contrast enhancement: When the interface background is complex, the color theme is dark, or the elements are close to the background color, you can apply denoising algorithms and techniques such as histogram equalization and brightness and contrast adjustment to improve image clarity, thereby increasing the accuracy of visual element detection and text recognition.

[0055] Caching mechanism: To avoid repeated analysis of the same interface, an image caching mechanism is also equipped. When the same interface appears repeatedly, the cached analysis results will be directly used to reduce unnecessary calculations and improve overall processing efficiency.

[0056] For example, after the RPA system is started, it monitors the user's screen. When a user issues a request, such as "Please help me fill out the application form. My name is Zhang xx, I'm 28 years old, my email address is 12345xxxxx, and my address is City A, and then submit it," the RPA uses the system API to capture a complete screenshot of the current screen, save it as a PNG file, and obtain a user interface image. The resolution of the obtained user interface image is 1920x1080. Since the optimal input size of the YOLO V8 model (the YOLO V8 model is used as the target detection model in the visual recognition model in the embodiment of this application) is 640x640, the user interface image is scaled down to 640x360 (maintaining the aspect ratio) and the Lanczos interpolation algorithm is used to reduce scaling distortion. To enhance the contrast of the form interface, a contrast-limited adaptive histogram equalization algorithm is also applied to improve the clarity of text and UI elements.

[0057] In step S204, the user interface image acquired in step S202 is deeply analyzed using computer vision models (e.g., YOLO for object detection and OCR for text recognition) to identify UI elements (such as buttons, text boxes, drop-down lists, etc.) and the text within them. This step is the core of "visual context awareness," ensuring the system understands the real-time state of the interface and maintains accurate and stable recognition even when the position of interface elements changes or new elements appear.

[0058] Optionally, target detection is performed on the user interface image through a visual recognition model, including: performing target detection on the user interface image through a target detection algorithm in the visual recognition model to obtain initial element information, wherein the initial element information includes at least one of the following: element bounding box coordinate information, element confidence and element category of the user interface image; determining the intersection-and-union ratio of all element bounding boxes in the initial element information, and filtering the initial element information according to a preset intersection-and-union ratio and a preset confidence to obtain interface element information.

[0059] In an embodiment of the present application, the YOLO V8 model is used as the target detection model in the visual recognition model, and its main task is to detect and classify different UI elements from the user interface image.

[0060] Specifically, when processing an image, the YOLO model outputs a series of bounding boxes, namely Each detection box contains the following key information:

[0061] That is, the element bounding box coordinate information, which represents the center coordinates, width, and height of the detection box, and is used to locate the specific position of the UI element on the screen.

[0062] Element confidence, that is, the model's estimate of the confidence level of the identified element, usually ranges from 0 to 1. The higher the confidence, the more reliable the detection result.

[0063] Element category, used to indicate the type of UI element recognized by the model, such as buttons, text boxes, drop-down lists, etc., is the basis for generating subsequent operation instructions.

[0064] Optionally, the YOLO V8 model is trained by: obtaining historical user interface images and corresponding element annotation data; training the initial object detection model based on the historical user interface images and element annotation data to obtain the YOLO V8 model. The specific analysis is as follows:

[0065] During the model training phase, we need to collect a large number of sample interface images of different resolutions, themes, and language environments, and annotate the bounding boxes of elements such as buttons, text boxes, and menus to obtain element annotation data. For example, we can collect a large number of screenshots of various online forms and use tools such as LabelImg or Roboflow to manually annotate UI elements such as "textbox," "button," "dropdown," and "label" in the form, draw bounding boxes, and finally save the resulting element annotation data as an XML or JSON file in Pascal VOC or COCO format.

[0066] At the same time, for various fonts and languages ​​that may appear in the form, screenshots containing text are collected, and the text content is annotated using OCR tools or manually, such as using the Tesseract OCR engine combined with manual proofreading to obtain text annotation data.

[0067] Subsequently, the initial object detection model was fine-tuned using historical user interface images and element annotation data. The model output layer was modified to adapt to the number of UI element categories (such as text boxes, buttons, drop-down lists, etc.), resulting in a fine-tuned YOLO model. The loss function of the YOLO V8s model is as follows:

[0068]

[0069] Where, It is used to measure the positioning deviation of the detection frame; Used to measure target confidence and whether there is misjudgment; Used to measure the accuracy of category recognition; coord ,λ obj ,λ cls is the corresponding hyperparameter.

[0070] Furthermore, the trained YOLO model is used to perform target detection on the real-time user interface image to obtain initial element information. For example:

[0071]

[0072]

[0073] To improve the accuracy of target detection results and the system's real-time response capabilities, and to better serve the automation needs of RPA, the initial element information can be post-processed and optimized, including but not limited to:

[0074] NMS (Non-Maximum Suppression): For multiple overlapping detection frames of the same target element, duplicates are removed based on confidence and IoU. Specifically, in the initial element information output by the YOLO model, the same target may be recognized multiple times, resulting in multiple detection frames. This not only increases the computational burden but also introduces uncertainty. Therefore, the NMS algorithm can be used to compare the IoU (Intersection over Union) and confidence of the detection frames, eliminating those with severe overlap and low confidence, leaving only the most confident and relatively independent detection frames, thereby improving the accuracy and robustness of the detection results.

[0075] Dynamic threshold optimization: In different RPA application scenarios, factors such as interface background complexity, lighting conditions, and UI element density can vary significantly. To adapt to these variations, the confidence threshold (tau) can be dynamically adjusted during the detection phase. In scenarios with complex backgrounds, lowering the tau value can capture more candidate detection frames and avoid missing important elements due to background interference. However, a lower threshold may introduce more noise. Therefore, these additional candidate frames can be further verified using text recognition or Canny edge detection to ensure the accuracy and applicability of the final detection results.

[0076] Model pruning and quantization: To maintain real-time performance, the YOLO model can be pruned or quantized to improve inference speed when GPU or CPU resources are limited.

[0077] For example, for the initial element information, non-maximum suppression is applied, the IoU threshold is set to 0.5, and the confidence threshold is set to 0.6 to remove overlapping detection boxes and filter low-confidence detection results to obtain the final interface element information.

[0078] Optionally, text recognition is performed on the user interface image through a visual recognition model, including: determining a text candidate area from the interface element information, wherein the text candidate area is a bounding box containing the element category; denoising and binarizing the text candidate area, and identifying the text candidate area through an optical character recognition algorithm in the visual recognition model to obtain text information; and determining the coordinate mapping relationship between the text information and the corresponding bounding box.

[0079] In an embodiment of the present application, OCR text recognition technology is used to perform text recognition in the visual recognition model to ensure that the system can understand the text information displayed on the interface.

[0080] Specifically, during the object detection phase, the YOLO model has already detected various interface elements in the image, including text boxes, buttons, and labels. Subsequently, the system prioritizes those elements marked as likely to contain text based on the YOLO model's recognition results, such as text boxes, button labels, and menu items, as candidate text regions for OCR processing.

[0081] It should be noted that if no text candidate area is detected, but the user request information contains text requirements (such as "search for buttons containing the word 'Submit'"), the entire user interface image can be scanned by OCR in text detection mode.

[0082] Secondly, the resulting text candidate regions are binarized, for example using the Otsu or adaptive thresholding methods, and inverted for dark themes or low-contrast interfaces to enhance text visibility.

[0083] Next, the binarized text candidate regions are denoised using morphological opening or closing operations to remove noise and scattered pixels. For example, a 3x3 median filter is used to remove salt and pepper noise.

[0084] The denoised text candidate regions are then segmented and recognized. The Tesseract OCR engine performs character segmentation on the binarized text candidate regions, then matches each character against the language model dictionary, outputting the recognized text information. For example, the text in the "button" region is identified as "Submit."

[0085] Finally, the recognized text and its bounding box are mapped back to the original user interface image coordinates for subsequent fusion with the YOLO detection results. For example, the OCR-recognized text "Submit" is mapped to the bounding box of the "button" [450, 500, 80, 30].

[0086] It should be noted that the system can select the corresponding language pack based on the language characteristics of the user interface image, such as the "chi_sim" Chinese language pack. For rare symbols or custom fonts, Tesseract can be further fine-tuned or assisted by a deep CNN model for image-to-character matching.

[0087] Optionally, the method also includes: processing the user interface image through an edge detection algorithm to obtain edge information; when there is an irregular bounding box in the interface element information, adjusting the irregular bounding box based on the edge information; when there is fuzzy edge text in the text information, restoring the fuzzy edge text based on the edge information.

[0088] In the embodiments of this application, edge and contour detection is considered an important complementary component of computer vision technology, particularly when integrated with object detection (YOLO) and text recognition (OCR). Through edge detection, particularly the Canny algorithm, RPA systems can obtain more detailed information about the outlines of interface elements, which is crucial for processing UI elements with irregular bounding boxes and recovering text with blurred edges.

[0089] Specifically, the Canny algorithm processes the image through the following steps to identify clear edge information, including:

[0090] 1. Gaussian filtering: Smoothes the user interface image, reduces noise, and provides cleaner input for subsequent edge detection.

[0091] 2. Gradient calculation: Use the Sobel operator to calculate the horizontal and vertical gradients of the user interface image to determine the direction and intensity of the image edge.

[0092] 3. Non-maximum suppression: exclude gradient values ​​that are not local maxima to ensure the continuity and clarity of the edge lines of the user interface image.

[0093] 4. Dual threshold detection: Apply high and low thresholds to detect strong edges and potential weak edges respectively. By connecting weak edge and strong edge areas, a complete edge map is constructed.

[0094] In an embodiment of the present application, when the YOLO model encounters irregular boundaries or low confidence when detecting certain UI elements, edge detection can serve as a second line of defense to assist in correcting boundary information and adjusting irregular bounding boxes. For example, for rectangular buttons with rounded corners, the Canny algorithm can detect these subtle edges, helping YOLO refine and correct the bounding box and improve detection accuracy. For another example, through Canny's edge information, the system can more accurately identify and locate those irregularly shaped UI elements, ensuring that RPA operations can correctly point to these elements.

[0095] Similarly, edge detection can also help with blurry or edged text encountered during OCR recognition. For example, the Canny algorithm can identify the edges around text, helping to determine the precise boundaries of the text. This guides OCR for more accurate text segmentation and recognition, reducing misjudgments caused by blurry boundaries.

[0096] In the embodiment of the present application, adaptive threshold optimization processing is also introduced. Specifically, under different UI themes, such as dark mode and light mode, the effect of edge detection will be different, and it is necessary to adaptively adjust the threshold to optimize the detection results. The specific method includes:

[0097] 1. Standard element sampling: By analyzing standard elements on the interface (such as window borders and fixed-size buttons), the system can derive initial high and low thresholds suitable for the current theme.

[0098] 2. Dynamic threshold adjustment: During the actual detection process, the system automatically adjusts the dual threshold parameters based on the characteristics of different image regions to obtain the best edge detection effect. This adaptive capability is particularly critical when dealing with the diversity of UI elements.

[0099] Through deep integration with visual recognition models, edge and contour detection algorithms not only improve the RPA system's ability to recognize complex interface elements, but also enhance the processing effect of blurred text, thereby improving the overall robustness and accuracy of the automated process.

[0100] For example, we perform Canny edge detection on the preprocessed user interface image. The Gaussian filter kernel size is 5x5, the high threshold is set to 100, and the low threshold is set to 50. Subsequently, we use the edge information detected by Canny to assist in verifying the YOLO detection results. For example, if the boundary of a "button" detected by YOLO is irregular, we use the edge contours detected by Canny to correct the boundary so that the bounding box more closely matches the actual shape of the button.

[0101] In step S206, the user's request or instruction information in natural language is received and then semantically parsed using a large language understanding model (such as an LLM model). The LLM model combines the visual information extracted in step S204 to understand the user's intent. Using instruction fine-tuning and thought chaining techniques, it performs multi-step reasoning and plans a series of precise operation instruction sequences (such as click, input, drag, etc.). These instruction sequences constitute the specific guidelines for RPA execution.

[0102] Optionally, the user request information and visual information are processed through a semantic understanding model to generate an operation instruction sequence, including: obtaining an execution instruction in the user request information; determining a corresponding prompt word template based on the execution instruction and visual information, wherein the prompt word template is used to specify the output format and constraints of the execution instruction; analyzing the execution instruction, visual information and prompt word template through a semantic understanding model to generate an atomic operation sequence, wherein the atomic operation sequence is used to reflect the logical reasoning process of the user's operation behavior; and determining an operation instruction sequence based on the atomic operation sequence.

[0103] In the embodiment of the present application, the semantic understanding model can combine the user's request information (natural language instructions or action descriptions) with visual information (interface element data parsed from the image), perform logical reasoning based on the current interface state, and generate a series of operation instruction sequences. The specific steps can be as follows:

[0104] S1: Obtain the execution instruction in the user request information.

[0105] First, parse the user request information, extract key operations (such as click, input) and target elements (such as login button, username box), obtain execution instructions, and convert them into text vectors, such as using the Sentence-BERT model for conversion.

[0106] Secondly, the visual information obtained by the visual recognition model (including interface element information obtained by the YOLO model and text information obtained by OCR technology) is organized into structured text as follows:

[0107]

[0108] In the embodiment of the present application, in order to more intelligently understand and respond to user request information, the above visual information and execution instructions can be processed through instruction tuning technology. The instruction tuning is mainly reflected in the following points:

[0109] A bridge from language to action: For user requests, such as "Press the red button, then enter 'test' in the text box that appears and press Enter," the Instruction Tuning LLM model can be broken down into specific steps: 1) Find the UI element ID corresponding to the "red button" in the UI context; 2) Generate a CLICK instruction; 3) Generate a TYPE instruction and process the text "test." Without Instruction Tuning, even large models with strong language capabilities would struggle to generate commands directly in the format required for RPA execution.

[0110] Coping with diverse instruction expressions: Different users may describe the same operation in different ways or in a mixture of languages, such as a combination of Chinese and English, such as "Please click 'Submit' and enter 'Name = John'." Instruction Tuning allows the LLM model to better generalize and adapt to various expressions, typos, or colloquialisms, making the system more compatible with natural language.

[0111] Reduced hardcoding and script maintenance: Conventional RPA often requires writing specific scripts for each interface element, which then need to be updated when the interface changes. With Instruction Tuning, the LLM model automatically infers operational instructions based on visual context and natural language commands (user request information), significantly reducing the need for hardcoding and making the RPA system more flexible and easier to maintain.

[0112] Multi-round dialogue and dynamic updates: When encountering interface changes or abnormal situations (such as element not found, click failure, etc.), the LLM model after instruction tuning can automatically generate repair or alternative operations based on real-time visual information and historical operation sequences, improving the system's fault tolerance and dynamic adaptability.

[0113] Alternatively, the instruction fine-tuning process of the LLM model (such as Qwnen2) can be as follows:

[0114] 1. Construction of instruction fine-tuning dataset.

[0115] First, we build a dataset containing a large number of "user request information - visual information - operation instruction sequence" triplets. For example, for the "filling out a form" scenario, data samples may include:

[0116]

[0117]

[0118] It should be noted that this dataset includes historical user request information and historical visual information. The historical user request information can contain multilingual text, colloquial expressions, etc. The historical visual information is the historical detection results obtained by the visual recognition model.

[0119] 2. Use the "triplet alignment" strategy to train the model, including:

[0120] 1) Use "UIElementID" as the core identifier;

[0121] 2) Using bounding box position coordinate information and text label data as auxiliary information;

[0122] 3) The execution instructions appearing in the user request information are explicitly associated with the element label data through multi-head attention.

[0123] Ultimately, the expected output of each sample is a set of multi-step operation instructions (such as CLICK, TYPE, DRAG, SCROLL, etc.), presented in the form of text tokens.

[0124] It should be noted that in the LLM model, there are different hierarchical task designs, as follows:

[0125] Basic layer: basic operations such as clicking buttons, entering text boxes, dragging, scrolling, and right-clicking, such as atomic operations such as "clicking a button" and "entering content in a text box";

[0126] Composite layer: Continuous operation, multi-window, multi-page scenarios, such as multi-step composite tasks such as "first log in to the system, then fill out the form, and then download the report after submitting it";

[0127] Exception layer: pop-up confirmation, permission verification, etc., such as fallback or retry logic in the case of "button does not exist", "login failed", and "network timeout".

[0128] By constructing the dataset in layers, the model can first master atomic operations during learning, and then gradually learn combined operations and abnormal scenarios, thus having strong progressive learning and scenario adaptability.

[0129] In addition, the LLM model also has multi-language and multi-expression coverage capabilities, as follows:

[0130] Multi-language support: The same UI element may have multiple language descriptions such as "OK", "OK", and "OK (OK)". Model training data must reflect various language scenarios.

[0131] Supports multiple expressions: such as "click the blue button", "click that blue button", and "press the blue button", all of which correspond to the same operation, allowing the model to learn to merge synonymous instructions.

[0132] The resulting high-dimensional data set of "user language-interface elements-operation instructions" is no longer limited to a single language or fixed expression, significantly improving its adaptability to spoken and cross-language scenarios.

[0133] 3. Fine-tune alignment.

[0134] Specifically, we used the constructed dataset to perform supervised fine-tuning on the Qwen2 model. For example, we used cross-entropy loss as the loss function, AdamW as the optimizer, and a learning rate of 1e-5. We trained for 10 epochs, monitoring performance on the validation set and using an early stopping strategy to prevent overfitting.

[0135] It should be noted that in order for the model to trace back the interface element information and text information when generating the operation instruction sequence, the YOLO / OCR output needs to be marked in the prompt or the internal structure of the model, such as:

[0136] UIElementID=Button_OK, label_text="OK", position={"x":100,"y":200,"width":150,"height":30}.

[0137] At the same time, during fine-tuning, the mapping between the action type and the UI element is repeatedly demonstrated. When the user mentions "OK button" in the command, the model will learn to align "OK" with UIElementID=Button_OK.

[0138] The model training strategy adopts layered and iterative training, as follows:

[0139] Layered training: First, fine-tune the model on atomic operation data for several epochs, then gradually add complex and abnormal scenarios.

[0140] Validation set and early stopping: Monitor the "Step-Level Match" or "Executable Success Rate" metrics in the validation set; perform early stopping or lower the learning rate when overfitting or stagnation occurs.

[0141] 4. Two-stage verification and online update.

[0142] Through the above process, a preliminary usable language understanding model can be obtained. During the test environment verification, if any mismatch cases are found, they can be collected and added to the "corrected dataset". The details are as follows:

[0143] Feasibility verification and incremental fine-tuning: mark and correct error cases (generated mismatched UIElementID, incorrect input text, etc.); then perform incremental fine-tuning on the model to narrow the gap with the real scene; this "trial and error-correction" loop can be repeated multiple times to gradually improve the ability to adapt to complex interfaces.

[0144] Online learning: After the system goes online, it regularly collects error samples of new interfaces and new commands; manually or automatically corrects them and then makes incremental adjustments; achieves adaptive evolution, and continuously updates the command mapping for new scenarios.

[0145] 5. Output after fine-tuning the command.

[0146] Once the model has completed instruction fine-tuning, it can "learn" how to convert the input "user request information + visual information" into an operation sequence with a fixed format, as follows:

[0147]

[0148] S2: Determine the prompt word template.

[0149] In this embodiment, to ensure that the LLM model can accurately and reliably convert user request information and current interface visual information into an executable RPA operation quality sequence, a prompt word engineering optimization strategy is introduced. Specifically, a corresponding prompt word template is determined based on the execution instructions in the user request information and the visual information of the user interface image to specify the structure and format of the execution instructions.

[0150] Specifically, the prompt word template clearly specifies the output format and constraints for executing instructions, including but not limited to:

[0151] Enforced sequentiality: Requires that the generated steps be arranged strictly in a logical execution order.

[0152] Unique numbering: Each step must be assigned a unique serial number or ID for easy tracking and reference.

[0153] Standardized structure: Each step should contain predefined key information fields, such as <action type (action_type)>, <operation object (UIElementID)>, and optional <parameters (OptionalText / Parameters)>.

[0154] The sample prompt word template is as follows:

[0155] Based on the following user request information and the current interface visual information, please generate the RPA operation step sequence in strict order. Requirements:

[0156] 1. Steps must be numbered consecutively starting from 1 and cannot be repeated.

[0157] 2. Each step must contain a clear 'action_type' and 'UIElementID'.

[0158] 3. If the action involves inputting text, the 'OptionalText' field must be included.

[0159] 4.UIElementID must be an ID from the provided UI element list.

[0160] S3: Generate atomic operation sequence.

[0161] In an embodiment of the present application, for user instructions involving multiple steps, conditional judgments, reference resolution, or the need for complex reasoning in combination with the context, relying solely on structured templates may not be sufficient to ensure the accuracy of the logic. To this end, a chain of thought (CoT) method is also introduced and strengthened. By guiding the LLM model to explicitly output its internal analysis and reasoning steps before generating the final operation sequence, it helps the LLM model to decompose macro user request information (such as "fill in the complete form and submit") into a series of specific atomic operation sequences corresponding to UI elements. For example:

[0162] Suppose the user's request is: "Find the window titled 'Invoice Details', enter 500 in the 'Amount' input box, and if the 'Review Status' is 'Pending Review', click the 'Approve' button."

[0163] The intermediate reasoning steps output by the LLM model under the guidance of CoT, that is, the atomic operation sequence is as follows:

[0164] 1. Identify the core intent: The user wants to perform an action in a specific window ('Invoice Details'): fill in the amount and decide whether to click the 'Approve' button based on specific conditions ('Review Status' is 'Pending Review').

[0165] 2. Locate the target window / area: You need to find the window or area with the title "Invoice Details" on the current screen or in the window list. Assume that Visual Analysis provides `Window_InvoiceDetail_ID` corresponding to this window.

[0166] 3. Locate the 'Amount' input box: Find the input box with the text label "Amount" in the Window_InvoiceDetail_ID window. Assume that Visual Analysis finds the TextBox_Amount_ID, which has the label "Amount".

[0167] 4. Plan the filling operation: Generate a `TYPE` operation, the target is `TextBox_Amount_ID`, and the input text is "500".

[0168] 5. Locate the 'Audit Status' element: Find the element displaying 'Audit Status' within the Window_InvoiceDetail_ID window. Assuming that the visual analysis finds 'Label_AuditStatus_ID', its OCR text content needs to be checked.

[0169] 6. Conditional judgment: Check whether the text content of `Label_AuditStatus_ID` is "Pending Review".

[0170] IF TRUE: Continue to the next step.

[0171] IF FALSE: Skip clicking the "Pass" button.

[0172] 7. Locate the 'Approve' button: (Execute only if the condition is true) Find a button with the text label "Approve" within the Window_InvoiceDetail_ID window. Assume that Visual Analysis finds Button_Approve_ID.

[0173] 8. Plan click action: (Execute only when the condition is true) Generate `CLICK` action, the target is `Button_Approve_ID`.

[0174] S4: Generate an operation instruction execution sequence.

[0175] Based on the above atomic operation sequence, the final operation instruction sequence can be generated by combining them in order.

[0176] Based on the above user request information: "Please help me fill out the application form, my name is Zhang XX, age is 28, email address is 123456xxxxx, address is City A, and then submit", the following JSON format operation instruction sequence is finally generated:

[0177]

[0178] By combining explicit structured templates and CoT reasoning enhancement, the LLM model can not only generate operation instruction sequences in the correct format, but also ensure that these operation instruction sequences logically and accurately reflect the user's complex intentions and the current interface status, thereby significantly improving the intelligence level of RPA and its reliability in changing environments.

[0179] In step S208, the sequence of operation instructions generated in step S206 is converted into specific, executable automated operation instructions, and these operation instructions are executed by simulating user operation behaviors. During the execution process, the system will also dynamically adjust according to the real-time changes in the interface to ensure the consistency and effectiveness of the operation instructions with the current interface state.

[0180] Optionally, converting the operation instruction sequence into an operation instruction includes: obtaining execution parameters in the operation instruction sequence; mapping the execution parameters to corresponding execution functions according to a preset rule base to obtain a parameter mapping result, wherein the preset rule base is used to store a mapping relationship between the execution parameters and the execution functions corresponding to the system operation; generating an executable code snippet based on the parameter mapping result, and generating an operation instruction based on the code snippet.

[0181] In the embodiments of this application, converting an operation instruction sequence into an operation instruction is a key step in achieving automated operation in the RPA system. This is accomplished by mapping the abstract operation instruction sequence generated by the LLM into a specific system operation function, thereby generating a directly executable code snippet. A specific analysis can be as follows:

[0182] 1. Get the execution parameters in the operation instruction sequence.

[0183] The execution parameters of each operation instruction are extracted from the operation instruction sequence generated by LLM, including but not limited to the action type (such as click, input text, etc.), the ID of the target UI element, and the necessary operation details (such as the text content that needs to be input).

[0184] 2. Map the execution parameters to the corresponding execution functions.

[0185] In the RPA system, a preset rule base is maintained, which contains the mapping relationship between operation instructions and specific system APIs or library functions (such as PyAutoGUI). The specific mapping relationship is as follows:

[0186] 1) Mouse operation

[0187] Move the mouse: call pyautogui.moveTo(x,y) to move the mouse cursor to the specified coordinates.

[0188] Click: Call pyautogui.click() to perform a left click at the current location.

[0189] The movement speed can be controlled by the duration parameter, such as moveTo(x,y,duration=0.5), to simulate user operation.

[0190] If you need to right-click or double-click, use rightClick() or doubleClick().

[0191] 2) Keyboard input

[0192] Click on the text box: Use pyautogui.moveTo(x,y) and click() to activate the text box focus.

[0193] Enter text: call pyautogui.typewrite("text content") to simulate keyboard input, or use press() to press keys one by one.

[0194] Hotkey combination: For example, use the pyautogui.hotkey() function to simulate the Ctrl+Shift+Esc key combination.

[0195] 3) Screen scrolling

[0196] Scrolling the mouse wheel PyAutoGUI provides a scroll() function that can simulate mouse wheel scrolling.

[0197] Subsequently, the parameters extracted from the operation instruction sequence are converted into the format required to execute the function. The specific mapping strategy can be as follows:

[0198] 1) Get the mapping relationship between the action_type of the storage instruction and the PyAutoGUI function from the rule base constructed above. For example:

[0199] action_type="CLICK" maps to the pyautogui.click() function.

[0200] action_type="TYPE" maps to the pyautogui.typewrite() function.

[0201] 2) Parameter mapping and conversion: Map the execution parameters extracted from the instruction sequence to the parameters of the PyAutoGUI function. For example, the instruction {'step_id':'S1','action_type':'CLICK','UIElementID':'TextBox_Name_1','position':{"x":100,"y":200,"width":150,"height":30}}, the `x` and `y` in the instruction are directly mapped to the `x` and `y` parameters of the pyautogui.click(x=100,y=200) function.

[0202] 3. Generate executable code snippets.

[0203] In the embodiment of the present application, a predefined PyAutoGUI code template is used to fill in the parameters to generate an executable Python code snippet. The algorithm flow mainly includes the following steps:

[0204] 1) Code template construction: Predefine the PyAutoGUI code template, which contains the basic structure of the PyAutoGUI function call and reserves parameter placeholders. For example, for the pyautogui.click() function, the code template is: pyautogui.click(x={x},y={y}).

[0205] 2) Parameter filling and code assembly: Based on the mapping results output by the instruction parsing and mapping module, fill the parameters into the placeholder positions in the code template. For example, if the instruction is {'step_id':'S1','action_type':'CLICK','UIElementID':'TextBox_Name_1','postion':{"x":100,"y":200,"width":150,"height":30}}, then replace {x} with 100 and {y} with 200 in the code template pyautogui.click(x={x},y={y}), and the final PyAutoGUI code is generated: pyautogui.click(x=100,y=200).

[0206] 3) Code output: The generated PyAutoGUI code can be a code snippet in the form of a string or a complete Python script file. The code can be directly executed by the Python interpreter to achieve the automated operation described by the user instruction sequence.

[0207] 4. Execute the operation instructions.

[0208] In this embodiment of the application, the RPA execution engine will execute the generated Python code snippets in sequence, using the PyAutoGUI library to simulate mouse clicks and keyboard input operations. For example:

[0209]

[0210] It's important to note that RPA automation requires that each step be completed before proceeding to the next. Therefore, after executing an action, you can use interface detection to confirm that the corresponding status has changed (such as jumping to a new page or filling in a text box). Also, for actions that require a long network or system response time (such as pop-up windows or data loading), wait appropriately to prevent click failures due to interface delays.

[0211] The process automation approach also optimizes the efficiency and stability of RPA operations, including two key strategies: parallel processing and exception retry. Parallel processing allows multiple independent tasks to be executed simultaneously, reducing total execution time, but requires careful attention to avoid conflicts between mouse and keyboard operations. Exception retry, by setting a maximum number of retries, ensures automatic recovery or prompts the user after an operation fails, enhancing the system's fault tolerance.

[0212] Optionally, the above method also includes: obtaining a target interface image after the operation instruction is executed; performing a similarity comparison between the target interface image and the user interface image to obtain a similarity difference value, and determining a target change area when the similarity difference value exceeds a preset threshold, wherein the target change area is the area where the user interface image changes after the operation instruction is executed; determining target visual information corresponding to the target change area through a visual recognition model, wherein the target visual information includes updated interface element information and updated text information; processing the user request information and target visual information through a semantic understanding model to obtain an updated operation instruction sequence.

[0213] Specifically, during RPA execution, interface elements may change due to user actions, background data updates, asynchronous loading, or animation effects. These changes may cause elements (such as pop-ups, loaded result lists, and dynamically generated form fields) that were not present during the initial screenshot (user interface image) analysis to appear, while existing elements may disappear or change state. If the RPA system cannot perceive and adapt to these changes, subsequent operations may fail based on outdated interface information.

[0214] Therefore, the embodiment of this application also introduces a set of dynamic interface adaptation and operation instruction re-planning mechanisms, which are key links to ensure that the RPA system can continue to operate effectively in an environment with changing interfaces. This mechanism allows the system to monitor interface changes in real time. When changes are detected in interface elements or text information, it can re-analyze the target interface, update the visual context information, and regenerate the operation instruction sequence through the LLM model to ensure the consistency and accuracy of the automation process. The specific process can be as follows:

[0215] 1. Trigger change detection.

[0216] In the embodiment of the present application, change detection can be triggered at the following times:

[0217] 1) Periodicity: A screenshot of the screen or target window is taken at a preset time interval (such as 0.5 seconds or 1 second) and compared with the initial user interface image.

[0218] 2) Event-driven: After executing an operation that may cause interface changes (such as CLICK, SUBMIT, SCROLL), a change detection is triggered immediately.

[0219] 3) Instruction specific: The instructions generated by the LLM model can be attached with an "expected change" flag, indicating that an interface refresh and analysis must be forced after execution.

[0220] 2. Perform change detection.

[0221] Compare the current screenshot (e.g., the target interface image after executing an operation) with the initial screenshot (user interface image) before the previous operation. Methods such as pixel-level comparison, structural similarity index (SSIM), or feature point matching can be used to calculate the difference between the target interface image and the initial user interface image, resulting in a similarity difference value. When the difference value exceeds a preset threshold (e.g., 0.95), the user interface is considered to have changed significantly, and the target changed area is located for subsequent targeted analysis.

[0222] It should be noted that when an interface change is detected, it is not necessary to perform a complete YOLO+OCR analysis on the entire screen. Instead, the analysis focuses on the target changed area to improve efficiency. The specific steps include:

[0223] 1) New element recognition: Identify new interface elements that appear in the target change area and extract their category, location, text label and other information.

[0224] 2) Element status update: For existing elements in the target change area, check whether their status has changed (such as a button changes from gray to enabled, a text box content is filled, etc.), and update their properties.

[0225] 3) Element disappearance processing: Compare the new and old element lists and mark the elements that disappear in the changed area.

[0226] 4) Assign temporary ID: Assign a unique temporary UIElementID to all newly identified elements.

[0227] 3. Update visual information.

[0228] In an embodiment of the present application, when a target change area is detected and the change area may affect operation instructions that have not yet been executed (for example, the button to be clicked next disappears, or a new, higher priority element such as an error pop-up window appears), the LLM model needs to be triggered to perform instruction replanning.

[0229] Specifically, the visual recognition model (such as YOLO, OCR, and edge detection) must first be re-run on the target changed area to obtain updated interface element information and text information, including newly appeared elements and changes in element status, to obtain the target visual information. Secondly, the updated target visual information is merged with the initial visual information. Elements with high overlap in position and content are deduplicated or updated. The integrated visual information (including all currently visible and interactive elements and their attributes) is formatted and used as input for the next LLM model call.

[0230] 4. Re-evaluation and planning of instruction sequences.

[0231] Based on the latest interface state, the remaining tasks are re-evaluated. That is, the original user request information and the updated visual information are submitted as new input context to the instruction-fine-tuned LLM model, requiring the model to regenerate the operation instruction sequence based on the current interface state.

[0232] If the originally planned action is still feasible, the LLM model generates the same subsequent instructions as before. If the originally planned action is blocked due to interface changes (such as the target element not existing or the state not matching), the LLM model uses its reasoning capabilities (possibly with the help of CoT) to generate alternative solutions (such as finding an alternative button, handling pop-up windows, scrolling the page to find the element, etc.), and outputs an updated action instruction sequence or an error report.

[0233] Finally, the RPA execution engine will switch to the operation instructions corresponding to the updated operation instruction sequence.

[0234] In the embodiment of the present application, an execution recording and playback function is also introduced.

[0235] Specifically, during execution, the system records the instructions, execution time, coordinate location, and corresponding screenshots for each operation, and stores this information in log files or databases. This provides a detailed basis for post-audits, allowing for screenshots of key operation nodes (such as when the submit button is clicked), enabling quick verification of process correctness or analysis of anomalies when needed. The playback function allows users to intuitively view the execution of the entire automated process, which is crucial for process optimization, troubleshooting, and compliance checks.

[0236] In the embodiment of the present application, an anomaly detection and processing mechanism is also introduced.

[0237] Specifically, during the execution process, the system will monitor in real time possible visual anomalies (such as disappearance of interface elements, low confidence or OCR recognition failure) and execution anomalies (such as operations not achieving the expected results, unexpected window closure, insufficient permissions, etc.).

[0238] For these anomalies, various strategies are employed: First, automatic retries are attempted to resolve potentially temporary issues. If the problem persists, rollbacks are performed, undoing some completed operations or executing alternative instructions to restore a stable state. When the automated process cannot continue, the system issues prompts or warnings to the user, requesting manual intervention. The implementation of these strategies not only improves the system's ability to handle complex situations but also reduces process interruptions caused by anomalies, enhancing the overall robustness of the RPA system and the user experience.

[0239] In the embodiment of the present application, a feedback and adaptive optimization mechanism is also introduced.

[0240] Specifically, by collecting abnormal situations that occur during the execution process (such as OCR misrecognition, operation failures, etc.) as well as the corresponding screenshots and annotation information, this data is re-incorporated into the training data of the YOLO model or OCR model, thereby continuously optimizing the performance of these models and improving their recognition accuracy of interface elements and text. At the same time, the system also collects error cases that occur when the LLM model parses user request information, such as generating incorrect operation sequences, and uses these cases to update the instruction fine-tuning dataset. In this way, LLM can continuously learn during daily use, enhancing its ability to understand and execute complex instructions, thereby improving the overall robustness and adaptability of the system. This feedback mechanism not only enables the system to learn from its errors, but also provides data support for continuous improvement, ensuring that the RPA system remains efficient and reliable in an ever-changing environment.

[0241] In summary, in the embodiments of this application, a visual recognition model is used to perform real-time target detection and text recognition on the user interface. In combination with a semantic understanding model, user instructions are deeply analyzed to generate a sequence of operation instructions and perform automated operations. This emphasizes the adaptive capabilities of dynamic interfaces. By continuously monitoring interface changes, performing incremental visual analysis, and replanning instructions, the system is able to adaptively adjust operation instructions when the interface dynamically changes, thereby significantly improving the intelligence and robustness of the RPA system. Furthermore, a feedback mechanism is used to collect abnormal data for continuous model optimization, further enhancing the system's adaptive capabilities and reliability.

[0242] According to an embodiment of the present application, a process automation system is provided, which can be integrated into a personal computer, server, or cloud platform. The hardware foundation of this system generally includes a processor (CPU), a graphics processing unit (GPU, used to accelerate CV and LLM model inference), memory (RAM), storage devices (hard disk or SSD), and input / output devices (keyboard, mouse, display) for user interaction. Its core software architecture includes the following main modules:

[0243] (1) User interaction module

[0244] The user interaction module serves as a bridge for two-way communication between the RPA system and users. It receives user requests, whether via text input or voice-to-text conversion, effectively capturing user intent. This module also clearly presents the RPA system's execution status, task results, and process issues to users, and requests user instructions. To meet diverse user needs, the user interaction module offers a variety of user interface options, including an intuitive graphical user interface (GUI), a command-line interface (CLI) for technicians, and an application programming interface (API) for easy integration with other systems. These diverse interface designs ensure that users can interact with the RPA system in the manner that best suits them, improving overall user experience and efficiency.

[0245] (2) Input acquisition and preprocessing module

[0246] The Input Acquisition and Preprocessing module is the cornerstone of the RPA system's "visual perception," focusing on acquiring and preparing the image data required for system operation. This module's primary task is to efficiently capture user interface images or, as needed, acquire live video streams of specified windows or applications to capture dynamic changes in the user interface. To ensure the accuracy and efficiency of subsequent visual analysis, the module incorporates a sophisticated image preprocessing pipeline. This includes intelligently scaling user interface images based on model requirements, removing image noise to improve clarity, and enhancing image contrast to highlight interface elements and text. To optimize performance, the module also features image cache management, avoiding repeated processing of the same interface images and significantly improving the responsiveness of the RPA system. Technically, this module flexibly calls operating system APIs, leverages mature screen recording libraries such as MSSS and OBS, and integrates powerful image processing libraries such as OpenCV, thereby building a stable, efficient, and customizable image data acquisition and preprocessing pipeline.

[0247] (3) Visual perception module

[0248] The visual perception module is the core of the RPA system's ability to understand screens. It analyzes the image data provided by the input acquisition module, simulates the user's visual system, identifies and accurately locates various elements on the user interface, and extracts key textual information. This module consists of multiple submodules that work together to complete complex visual understanding tasks, as follows:

[0249] Object Detection Submodule: This submodule is the foundation of visual perception and uses advanced deep learning object detection models, such as the efficient and accurate YOLO V8 series, to detect various UI elements on the user interface. Its capabilities cover a variety of common elements, including buttons, text boxes, icons, drop-down lists, checkboxes, and more. It not only accurately identifies the element's category but also precisely locates its position on the screen, providing precise bounding box coordinate information for subsequent operations.

[0250] Text Recognition Submodule: This submodule focuses on extracting text from images and integrates a high-performance optical character recognition (OCR) engine, such as the widely used Tesseract OCR. This submodule can recognize text labels within UI elements and any text appearing anywhere on the screen, converting text in images into machine-readable text data, providing important semantic information for the language understanding and instruction planning modules.

[0251] Edge Detection Submodule: To further improve the accuracy and robustness of UI element positioning, especially when element boundaries are blurred or blend into the background, this submodule introduces edge detection algorithms, such as the Canny edge detection algorithm. By detecting edge information in the image, it can assist in locating the precise boundaries of UI elements and help distinguish adjacent or overlapping elements, improving the level of visual perception.

[0252] Information Fusion Submodule: Serving as the "brain" of the visual perception module, the Information Fusion Submodule is responsible for integrating the output results from various submodules, including object detection, text recognition, and edge detection. It correlates and fuses information from different sources to construct a structured visual context for the interface. This context includes key attributes such as the unique ID of each identified UI element, the element type (such as a button or text box), its precise on-screen location coordinates, and the associated text label. This context serves as an important basis for the subsequent language understanding and planning modules to understand the interface and generate operational instructions.

[0253] (4) Language comprehension and planning module

[0254] The Language Understanding and Planning module is the "intelligent decision-making center" of the RPA system. It is primarily responsible for receiving instructions from the User Interaction Module and visual information provided by the Visual Perception Module, leveraging powerful language understanding models (such as the LLM) to perform in-depth intent understanding and intelligent operation planning. At the core of the module is an LLM specifically fine-tuned for RPA task instructions (Instruction Tuning), such as the open-source and high-performance Qwen2 series models. Through instruction tuning, the LLM is trained to accurately understand users' natural language instructions for RPA tasks. Based on the current complex interface state, it can infer and generate a series of executable operation instructions, guiding the RPA system to complete the automated task. Technically, the module offers the flexibility of invoking remote LLM API services or deploying LLM models locally to meet varying performance and privacy requirements. To fully leverage the potential of the LLM, the module also leverages Prompt Engineering technology, including advanced prompt techniques such as Chain of Thought (CoT), to guide the LLM in deeper reasoning and more accurate instruction generation. At the same time, in order to process the structured information from the visual perception module, this module has multimodal input processing capabilities, effectively integrating text instructions and visual context, constructing comprehensive contextual information, and providing a solid foundation for LLM's intelligent decision-making.

[0255] (5) Execution Control Module

[0256] The execution control module is the key execution unit in the RPA system, transforming intelligent decisions into actual operations. Its core responsibility is to accurately parse the ordered sequence of operational instructions generated by the language understanding and planning module and effectively map these high-level instructions into low-level mouse and keyboard commands. The module then calls system APIs provided by the operating system or mature automation libraries (such as the cross-platform PyAutoGUI and the process automation-focused RPA Framework) to simulate user manual operations, such as mouse movements, clicks, drags, and keyboard input. To ensure the stability and smoothness of the automated process, the execution control module also manages the entire execution process in a precise manner. For example, it inserts appropriate delays between necessary steps to ensure sufficient time for the system to respond and for interface elements to load, and synchronizes operations to avoid errors caused by overly rapid or slow execution. Through these sophisticated control mechanisms, the execution control module ensures that the RPA system can interact with the application like a real user, completing automated tasks efficiently and reliably.

[0257] (6) State management and dynamic adaptation module

[0258] The state management and dynamic adaptation module enables the RPA system to maintain stable operation in complex and dynamic environments. Its core function is to maintain the running status of the current RPA task, track the task execution progress, and provide real-time UI context information. More importantly, the module monitors dynamic changes in the user interface. Using mechanisms such as Demographic Interactions (DEMIA), it can detect changes in the position, status, and content of interface elements in real time. When an interface changes, the module immediately triggers the visual perception module to reanalyze the interface and update the visual information. Subsequently, the module collaborates with the language understanding and planning module to replan instructions based on the latest interface state, ensuring that subsequent instructions remain valid and meet the user's ultimate intent. Technically, the module employs image comparison algorithms, such as pixel difference analysis or Structural Similarity Index (SSIM) comparison, to efficiently detect interface changes. It also leverages the operating system's event monitoring mechanism to capture window or application state changes in real time, enabling sensitive response and intelligent adaptation to dynamic interfaces.

[0259] (7) Recording and exception handling module

[0260] The logging and exception handling module is crucial for ensuring the reliability and maintainability of RPA systems. Its core function is to comprehensively record various information during the RPA system's execution, including step-by-step instructions, system status, and screenshots of key moments. This detailed log information is not only used for subsequent audits and process replay, facilitating user tracking of task execution, but also provides valuable data for troubleshooting and system optimization. Furthermore, the module boasts powerful anomaly detection capabilities, enabling real-time monitoring of the RPA system's operational status and detecting various anomalies that may arise during execution, such as visual anomalies (disappearing UI elements, recognition failures), execution anomalies (operations not executing as expected, system errors), and even logical anomalies (process logic errors). Once an anomaly is detected, the module intelligently handles it according to pre-set policies, such as automatically retrying failed steps, reverting to a previous stable state, or reporting the anomaly to the user and requesting manual intervention. In terms of technical implementation, the module can use mature logging libraries (such as Python's logging module) for detailed logging, adopt exception capture mechanisms (such as try-except statements) to capture runtime errors, and combine them with rule engines or simple conditional judgment logic to implement flexible and configurable exception handling strategies to ensure that the RPA system can respond promptly and effectively when encountering problems, ensuring the stable completion of automated tasks.

[0261] (8) Feedback and Learning Module

[0262] The feedback and learning module empowers the RPA system to continuously evolve, enabling it to learn from errors and user feedback, continuously improving its intelligence and performance. The module's core function is to systematically collect various error cases generated during RPA execution, including visual recognition errors, command misunderstanding deviations, execution failures, and other errors, as well as direct user feedback or manual corrections. The module organizes and analyzes this valuable data and uses it to drive the retraining or fine-tuning of visual models (such as YOLO and OCR models) or LLM models. Through this continuous feedback learning mechanism, the RPA system can continuously optimize its visual perception capabilities, language understanding capabilities, and operation execution accuracy, thereby achieving self-improvement and performance enhancement over the long term. Technically, the module requires the design of an effective data collection interface to facilitate the collection of various types of feedback data. It also provides a model training / fine-tuning script interface to automatically iterate and update the model, enabling continuous optimization and intelligent evolution of the RPA system.

[0263] According to an embodiment of the present application, a process automation device is provided. It should be noted that the process automation device of the embodiment of the present application can be used to execute the process automation method provided in the embodiment of the present application. The process automation device provided in the embodiment of the present application is introduced below.

[0264] Figure 3 This is a structural diagram of a process automation device provided according to an embodiment of the present application. Figure 3 As shown, the device includes:

[0265] An acquisition module 30 is used to acquire a user interface image;

[0266] A recognition module 32 is configured to perform target detection and text recognition on the user interface image using a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image;

[0267] A generation module 34 is configured to receive user request information, process the user request information and visual information through a semantic understanding model, and generate an operation instruction sequence;

[0268] The operation module 36 is used to convert the operation instruction sequence into an operation instruction and automatically simulate the user's operation behavior according to the operation instruction.

[0269] Through the acquisition module, recognition module, generation module and operation module in the above-mentioned process automation device, the purpose of intelligently parsing the association between user natural language instructions and dynamic interface elements is achieved, thereby realizing the technical effect of high robustness, flexibility and intelligence of automated operations, and further solving the technical problems that the RPA system in related technologies is usually based on fixed interface coordinates or control features, resulting in poor interface adaptability, weak ability to understand user intentions and insufficient robustness in complex scenarios.

[0270] In the process automation device provided in the embodiment of the present application, the recognition module is also used to perform target detection on the user interface image through the target detection algorithm in the visual recognition model to obtain initial element information, wherein the initial element information includes at least one of the following: element bounding box coordinate information, element confidence and element category of the user interface image; determine the intersection-and-union ratio of all element bounding boxes in the initial element information, and filter the initial element information according to the preset intersection-and-union ratio and preset confidence to obtain interface element information.

[0271] In the process automation device provided in the embodiment of the present application, the recognition module is also used to determine a text candidate area from the interface element information, wherein the text candidate area is a bounding box containing the element category; the text candidate area is denoised and binarized, and the text candidate area is identified by the optical character recognition algorithm in the visual recognition model to obtain text information; and the coordinate mapping relationship between the text information and the corresponding bounding box is determined.

[0272] In the process automation device provided in the embodiment of the present application, the recognition module is also used to process the user interface image through an edge detection algorithm to obtain edge information; when there is an irregular bounding box in the interface element information, the irregular bounding box is adjusted according to the edge information; when there is fuzzy edge text in the text information, the fuzzy edge text is restored according to the edge information.

[0273] In the process automation device provided in the embodiment of the present application, the generation module is also used to obtain the execution instructions in the user request information; determine the corresponding prompt word template based on the execution instructions and visual information, wherein the prompt word template is used to specify the output format and constraints of the execution instructions; analyze the execution instructions, visual information and prompt word template through the semantic understanding model to generate an atomic operation sequence, wherein the atomic operation sequence is used to reflect the logical reasoning process of the user's operation behavior; and determine the operation instruction sequence based on the atomic operation sequence.

[0274] In the process automation device provided in the embodiment of the present application, the operation module is also used to obtain the execution parameters in the operation instruction sequence; map the execution parameters to the corresponding execution functions according to the preset rule base to obtain the parameter mapping results, wherein the preset rule base is used to store the mapping relationship between the execution parameters and the execution functions corresponding to the system operations; generate an executable code snippet based on the parameter mapping result, and generate an operation instruction based on the code snippet.

[0275] The process automation device provided in the embodiment of the present application also includes an update module 38, which is used to obtain the target interface image after the operation instruction is executed; compare the target interface image and the user interface image for similarity to obtain a similarity difference value, and determine the target change area when the similarity difference value exceeds a preset threshold, wherein the target change area is the area where the user interface image changes after the operation instruction is executed; determine the target visual information corresponding to the target change area through a visual recognition model, wherein the target visual information includes updated interface element information and updated text information; process the user request information and the target visual information through a semantic understanding model to obtain an updated operation instruction sequence.

[0276] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned process automation method.

[0277] It should be noted that the above electronic equipment is used to perform Figure 2 The process automation method shown, therefore the relevant explanations in the above process automation method are also applicable to the electronic device and will not be repeated here.

[0278] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned process automation method by running the computer program.

[0279] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The process automation method shown, therefore the relevant explanations in the above process automation method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0280] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned process automation method when executed by a processor.

[0281] It should be noted that the above-mentioned computer program product is used to execute Figure 2The process automation method shown, therefore the relevant explanations in the above process automation method are also applicable to the computer program product and will not be repeated here.

[0282] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0283] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0285] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0286] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0287] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0288] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A process automation method, characterized in that: include: Get the user interface image; Performing target detection and text recognition on the user interface image through a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image; Receive user request information, and process the user request information and the visual information through a semantic understanding model to generate an operation instruction sequence; The operation instruction sequence is converted into an operation instruction, and user operation behavior is simulated according to the operation instruction.

2. The method according to claim 1, characterized in that Performing target detection on the user interface image using a visual recognition model includes: Performing target detection on the user interface image using a target detection algorithm in the visual recognition model to obtain initial element information, wherein the initial element information includes at least one of the following: element bounding box coordinate information, element confidence, and element category of the user interface image; The intersection-over-union ratio of all element bounding boxes in the initial element information is determined, and the initial element information is screened according to a preset intersection-over-union ratio and a preset confidence level to obtain the interface element information.

3. The method according to claim 2, characterized in that Performing text recognition on the user interface image using a visual recognition model includes: Determining a text candidate region from the interface element information, wherein the text candidate region is a bounding box containing the element category; Performing denoising and binarization processing on the text candidate area, and identifying the text candidate area using an optical character recognition algorithm in the visual recognition model to obtain the text information; Determine a coordinate mapping relationship between the text information and the corresponding bounding box.

4. The method according to claim 3, characterized in that The method further comprises: Processing the user interface image using an edge detection algorithm to obtain edge information; In the case where the interface element information has an irregular bounding box, adjusting the irregular bounding box according to the edge information; In the case where there are fuzzy edge characters in the text information, the fuzzy edge characters are restored according to the edge information.

5. The method according to claim 1, wherein The user request information and the visual information are processed by a semantic understanding model to generate an operation instruction sequence, including: Obtaining the execution instruction in the user request information; Determining a corresponding prompt word template according to the execution instruction and the visual information, wherein the prompt word template is used to specify an output format and constraint conditions of the execution instruction; Analyzing the execution instruction, the visual information, and the prompt word template through the semantic understanding model to generate an atomic operation sequence, wherein the atomic operation sequence is used to reflect the logical reasoning process of the user's operation behavior; The operation instruction sequence is determined according to the atomic operation sequence.

6. The method according to claim 1, characterized in that Converting the operation instruction sequence into an operation instruction comprises: Obtaining execution parameters in the operation instruction sequence; Mapping the execution parameters to corresponding execution functions according to a preset rule base to obtain a parameter mapping result, wherein the preset rule base is used to store a mapping relationship between the execution parameters and the execution functions corresponding to the system operations; An executable code segment is generated according to the parameter mapping result, and the operation instruction is generated according to the code segment.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a target interface image after executing the operation instruction; performing a similarity comparison between the target interface image and the user interface image to obtain a similarity difference value, and determining a target change region if the similarity difference value exceeds a preset threshold, wherein the target change region is a region of the user interface image that changes after the operation instruction is executed; Determining target visual information corresponding to the target change area through the visual recognition model, wherein the target visual information includes updated interface element information and updated text information; The user request information and the target visual information are processed by the semantic understanding model to obtain an updated operation instruction sequence.

8. A process automation device, characterized in that: include: An acquisition module, used to acquire a user interface image; a recognition module, configured to perform target detection and text recognition on the user interface image using a visual recognition model to obtain visual information corresponding to the user interface image, wherein the visual information includes interface element information and text information in the user interface image; A generation module is used to receive user request information, and process the user request information and the visual information through a semantic understanding model to generate an operation instruction sequence; The operation module is used to convert the operation instruction sequence into an operation instruction and simulate user operation behavior according to the operation instruction.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the process automation method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the process automation method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the process automation method according to any one of claims 1 to 7 is implemented.

Images