RPA instruction generation method based on user operation video

Abstract

The invention discloses an RPA instruction generation method based on a user operation video, and the method comprises the following steps: 1, recognizing an interface element in a webpage in the user operation video based on a target detection algorithm, and obtaining a timestamp of the change of the interface element; 2, analyzing the user operation recording video based on the video understanding model, obtaining a single-frame image and a text feature, injecting a timestamp into the single-frame image to form a visual feature, and fusing the visual feature and the text feature to obtain a multi-modal feature; and step 3, decoding the multi-modal features to obtain a mapping relation between the interface elements and the user operation behaviors, and then packaging the mapping relation into an RPA instruction. According to the method, deep understanding of user operation behaviors and automatic generation of the RPA instruction are realized, so that the accuracy and integrity of an automatic process are improved.

Description

Technical Field

[0001] The present invention relates to the field of robotic process automation technology, and specifically to a method for generating RPA instructions based on user operation videos. Background Art

[0002] In the current RPA (Robotic Process Automation) technology, the writing of automated processes often relies on manual programming or is achieved by simulating manual operations. This traditional method often has certain problems of complexity and poor flexibility, especially when automating the design of complex business processes. Developers need to program the operation process in detail, or rely on users to manually record through mouse clicks, keyboard input, etc. in the recording stage. In the existing technology, although there is a certain degree of recording function that can record user operations on the interface, these methods still have limitations. For example, they may only be able to record simple operation behaviors. For dynamically changing interface elements or complex user operation processes, the accuracy and completeness of the records are often difficult to guarantee. In addition, the recording process can often only record surface behaviors and lacks a deep understanding of the logic behind user behavior. Summary of the invention

[0003] The purpose of the present invention is to provide a method for generating RPA instructions based on user operation videos. The present invention realizes in-depth understanding of user operation behaviors and automatic generation of RPA instructions, thereby improving the accuracy and completeness of the automation process.

[0004] The technical solution of the present invention: A method for generating RPA instructions based on user operation videos comprises the following steps: Step 1: Identify the interface elements in the web page in the user operation video based on the target detection algorithm, and obtain the timestamp of the interface element change; Step 2: Analyze the user operation recording video based on the video understanding model, obtain single-frame image and text features, inject the timestamp into the single-frame image to form visual features, and then fuse the visual features and text features to obtain multimodal features; Step 3: Decode the multimodal features to obtain the mapping relationship between interface elements and user operation behaviors, and then encapsulate the mapping relationship into RPA instructions.

[0005] In the aforementioned RPA instruction generation method based on user operation video, in step 1, before identifying the interface elements, the target detection algorithm first collects screenshots of the user operation interface, and then annotates the collected screenshots of the user operation interface to obtain an annotated data set, and then builds a target detection model, and inputs the annotated data set into the target detection model for model training; based on the trained target detection model, the interface elements in the web page in the user operation video are identified, and the timestamp of the interface element change is obtained.

[0006] In the aforementioned RPA instruction generation method based on user operation video, the interface elements include a date selection panel, a drop-down box, a text input box, a button, a verification code panel, a slider and a scroll bar; the data annotation includes the position annotation and category annotation of the interface elements in each user operation interface screenshot.

[0007] In the aforementioned RPA instruction generation method based on user operation video, the model used by the target detection algorithm is the YOLOv8 model. The training process of the YOLOv8 model is based on the features of the interface element category annotations in the interface screenshot learning interface based on the labeled data set, and the weights are optimized by calculating the loss function.

[0008] In the aforementioned RPA instruction generation method based on user operation video, in step 2, the process of forming visual features is to extract and preprocess the user operation video to obtain a preprocessed single-frame image; encode the single-frame image through the ViT encoder of the video understanding model, and then use the target detection model trained in step 1 to identify the interface elements in the single-frame image and inject the timestamp of the interface element change, and finally locate the operation behavior by timestamp to obtain the visual feature.

[0009] In the aforementioned RPA instruction generation method based on user operation video, in step 2, the process of obtaining text features is to input the operation description of the user operation recording video, convert the operation description into a high-dimensional text vector and encode it to obtain text features.

[0010] In the aforementioned RPA instruction generation method based on user operation video, the process of obtaining multimodal features is to align visual features with text features using a self-attention mechanism and a fully connected network, and to fuse visual features with text features through a cross-modal attention mechanism to obtain multimodal features.

[0011] In the aforementioned RPA instruction generation method based on user operation video, in step 3, the process of multimodal feature decoding is to use a cross-modal Transformer decoder to decode the multimodal features, combine the user operation behavior in the video, and automatically establish a mapping relationship between the user operation behavior and the interface elements through intelligent analysis of the changes in the interface elements. Finally, the mapping relationship is encapsulated into an RPA instruction, and time information is associated with each user operation behavior.

[0012] In the aforementioned RPA instruction generation method based on user operation video, the cross-modal Transformer decoder includes a self-attention layer, a cross-modal attention layer, a feedforward neural network layer, and residual connections and layer normalization; the self-attention layer is used to process the current context of the cross-modal Transformer decoder; the cross-modal attention layer extracts relevant information from visual features and text features; each layer of cross-modal attention in the feedforward neural network layer is followed by a feedforward network for further feature mapping and enhancing the expressiveness of the output; the residual connection and layer normalization ensure smooth information flow and stable gradients, helping the video understanding model to retain the original features in the deep layer and prevent gradient disappearance.

[0013] In the aforementioned RPA instruction generation method based on user operation video, the video understanding model is fine-tuned for instructions and time positioning to enhance the video understanding model's ability to understand the user operation video and the specific time point of the user operation; The instruction fine-tuning is performed using the RPA instruction dataset generated by GPT. The data source of the dataset is the screenshots of the RPA at runtime and the corresponding operation instructions. The specific samples include video clip descriptions and generated RPA code snippets. 330k user operation video samples are used for fine-tuning, and the learning rate is 4e-6. The time positioning fine-tuning is to use the screenshot of the RPA application interface to generate an automated video, automatically determine the appearance and disappearance of interface elements through the target detection algorithm, generate corresponding timestamp annotation data, divide the long sequence video into multiple short sequences, and store the timestamp and operation description in JSON format. The learning rate of time positioning fine-tuning is set to 1e-6.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention, through the combination of target detection algorithm and video understanding model, can more accurately identify changes in user operation behavior and interface elements, establish a mapping relationship between user operation behavior and interface elements, so as to better understand the logic behind user behavior, thereby improving the accuracy and completeness of the automation process. The present invention encapsulates the mapping relationship into RPA instructions, supports operation behaviors in different scenarios, enables the system to adapt to different operation scenarios, and improves the flexibility and versatility of automated operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram of the VideoVLM model architecture of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0017] Embodiment: A method for generating RPA instructions based on a user operation video comprises the following steps: Step 1: Identify the interface elements in the web page in the user operation video based on the target detection algorithm, and obtain the timestamp of the interface element change; Step 2: Analyze the user operation recording video based on the video understanding model to obtain visual features and text features, inject the timestamp into the visual features and then integrate the text features to obtain multimodal features; Step 3: Decode the multimodal features to obtain the mapping relationship between interface elements and user operation behaviors, and then encapsulate the mapping relationship into RPA instructions.

[0018] The present invention includes a key control analysis module, a video understanding module, an intelligent instruction calling module, and a cross-platform support and operation adaptation module; In this embodiment, the key control analysis module is used to automatically analyze and identify interface elements in the user operation interface using a target detection algorithm. First, a large number of user operation interface screenshots are collected. These screenshots cover different applications, operating systems, and various common interface elements. Interface elements include date selection panels, drop-down boxes, text input boxes, buttons, verification code panels, sliders, and scroll bars. Then, data annotation is performed on each collected user operation interface screenshot, and each interface element key panel and its attributes are manually annotated to obtain an annotation data set. The data annotation includes the position annotation and category annotation of the key panel of the interface element in each interface screenshot.

[0019] Specifically, the position annotation uses a rectangular box to mark the position information of each interface element. Each annotation box contains the coordinates of the upper left corner and the lower right corner of the element, defining the position of the key panel. The category annotation assigns a category label to each annotated rectangular box, indicating that it is a date picker, a drop-down box, a verification code panel, etc. Each category should have a unique identifier. For example, assume that each category has about 1,000 data samples annotated to form a complete annotation dataset containing categories, locations, and attributes. The format of the annotation data usually adopts a common target detection annotation format (such as the YOLO format). Then build a target detection model and input the annotation dataset into the YOLOv8 target detection model for model training. The goal of the training is to allow the YOLOv8 target detection model to learn the characteristics of each interface element key panel category through a large number of interface screenshots, and to accurately identify these interface element key panels in unknown user interface screenshots. The YOLOv8 target detection model is the latest version of the YOLO series model, with significant improvements and optimizations. The YOLOv8 target detection model has been comprehensively improved in terms of speed, accuracy, and ease of use. It draws on the design advantages of models such as YOLOv5, YOLOv6, and YoloX, while maintaining the engineering simplicity and ease of use of YOLOv5. During the training process, the YOLOv8 target detection model will gradually optimize the weights of each detection box by calculating the loss function, thereby improving the detection accuracy.

[0020] The loss function calculation formula of the YOLOv8 target detection model is as follows: ; in, It is the position loss, usually calculated using IoU-related loss. is the confidence loss, calculated using binary cross entropy or other suitable loss functions, It is the category loss, and the cross entropy loss is usually used to measure the difference between the classification result and the actual label.

[0021] The specific process of optimizing weights with the loss function is as follows: Calculate the loss value of each data sample, calculate the gradient by backpropagation, determine the update direction of each parameter, use an optimization algorithm (such as Adam) to update the weights, gradually reduce the loss function value, and repeat the above process until the loss function of the YOLOv8 target detection model converges and the performance on the test set meets expectations.

[0022] Furthermore, during the training process, the loss functions (such as cross entropy loss, IoU loss, etc.) are optimized so that the YOLOv8 object detection model is as accurate as possible in predicting the key panel positions and categories of interface elements.

[0023] In this embodiment, the video understanding module is used to identify the user's operating behavior from the user operation video, and in combination with the detection of changes in interface elements, establish a mapping relationship between the operating behavior and the interface elements, and finally form an operating instruction. The video understanding module aims to automatically generate timestamp annotations of user behavior and precise operating instructions through in-depth analysis of user operation videos. The video understanding module uses a video recording tool to capture the video of the user's operating behavior, and uses time series analysis technology to segment it according to changes in interface elements and user operating behaviors (such as clicks, inputs, etc.). Through the deep learning behavior recognition model, each slice is labeled and a timestamp is generated to ensure that the specific time period of the operation can be accurately located.

[0024] Specifically, the video understanding module is based on the VideoVLM model, which is a large visual language model that combines visual and textual information and demonstrates outstanding capabilities in understanding and generating content involving images and text. VideoVLM fine-tunes the open source model CogVLM2-Video and uses the extracted video key frames and timestamp information to achieve intelligent analysis and time positioning of user operations, thereby enhancing the model's understanding of user interface videos and timestamp positioning capabilities. CogVLM2-Video is a model that has made significant progress in the field of video understanding. CogVLM2-Video solves the limitations of existing video understanding models in the temporal and spatial dimensions by introducing multi-frame video images and timestamps as encoder inputs. In the implementation of the video understanding module, ViideoVLM adopts a fusion architecture that combines video frame features with text features. The specific details of the model architecture are as follows: Figure 1 Specifically, 1. Input: image or video sequence (1) The input of the model is a video sequence or a single frame image, which is used to capture the user's operating behavior in the interface.

[0025] 2. Frame extraction and preprocessing module (1) Frame extraction: Extract key frames from the user operation video to obtain a single-frame image to reduce the amount of subsequent calculations and ensure that the key operation points are captured. Usually 24 frames per second are extracted.

[0026] (2) Preprocessing: including normalizing and resizing single-frame images (e.g., 224x224 pixels).

[0027] 3. Visual Encoding Module (ViT Encoder) Use Vision Transformer (ViT) to encode the preprocessed single-frame image and extract high-level visual features. The Vision Transformer (ViT) encoder is a visual model based on the Transformer architecture. The ViT encoder uses a self-attention mechanism to capture global and local image dependencies, and obtains a feature matrix representing the features of a single-frame image by encoding the image.

[0028] 4. Object detection and feature enhancement module (1) Object detection: Use deep learning-based object detection algorithms (such as YOLOv8) to identify interface elements (such as buttons, input boxes, verification code panels, etc.) in a single frame image.

[0029] (2) Injecting timestamp and spatial information: The object detection model not only identifies components, but also injects their timestamp and spatial features such as location, type, and time of appearance and disappearance into subsequent modules, and timestamps the operation behavior to enhance the contextual nature of feature understanding.

[0030] The process of injecting timestamps refers to combining time information (such as timestamps in videos) with the output of the target detection model. In this process, after the target detection model recognizes the position and category of each interface element, the system will record the timestamp of the frame image and bind the timestamp as additional information to the prediction result of each interface element. The specific operation is as follows: In the detection result of each interface element, in addition to the bounding box coordinates and category probability, timestamp information will be added to indicate the moment when the element appears in the video. The system uses the timestamp to match the interface element with the specific time point in the video. The method of injecting timestamps can be to directly use the timestamp as an additional field, or to embed the timestamp information into the output feature vector of the target detection model, so that the recognition result of each element contains the timestamp.

[0031] Timestamp location of operation behavior: Using the processed visual features and timestamp information, the system can perform timestamp location of operation behavior, helping the VideoVLM model understand what specific operations (such as clicking, inputting, scrolling, etc.) the user performed at different time points in the video, and determine the sequence and specific time of these user operation behaviors.

[0032] Through the above steps, we can finally obtain visual features including interface elements, timestamps, category labels, and user operation behaviors. These features will serve as input for the subsequent RPA instruction generation process.

[0033] 5. Text encoding module (1) The input operation description is converted into a high-dimensional text vector representation through Word Embedding. Word Embedding is an important technology in the field of natural language processing (NLP). It represents words as vectors, enabling computers to understand and process natural language.

[0034] (2) The text encoding part uses embedding technology to semanticize the input description to facilitate subsequent multimodal information fusion.

[0035] 6.Adapter feature alignment module (1) The main task of the Adapter module is to align visual features to the text feature space so that multimodal information can be processed uniformly in subsequent modules. The Adapter feature alignment module is a technology used in deep learning models to solve the problem of aligning features of data from different sources or different modalities.

[0036] (2) The Adapter module uses a self-attention mechanism and a fully connected network to ensure that the features obtained from the visual encoding module and the object detection module have the same vector representation format and dimension as the text features.

[0037] 7. Multimodal Fusion Module (1) Fusion mechanism: Visual features and text features are combined in the multimodal fusion module. The cross-modal attention mechanism is used to ensure that visual information and text information interact with each other in the same feature space. The cross-modal attention mechanism is a technology that allows the model to extract information from one modality and associate it with information in another modality. The cross-modal attention mechanism is built on the basis of the Transformer neural network architecture, which uses self-attention layers and encoder-decoder structures to process information from different modalities.

[0038] (2) Cross-modal interaction: The multimodal fusion module learns weights to focus on the parts of the visual features that are most relevant to the operation description, such as the position and state of a specific button or date picker. This step is accomplished through a multi-layer attention mechanism to capture the association between the visual features and the operation description.

[0039] 8. Cross-modal Transformer Decoder (1) Self-attention layer: used to process the current context of the decoder to ensure that the generated language description is coherent and semantically complete.

[0040] (2) Cross-modal attention layer: Extract relevant information from the visual features and text features input by the multimodal fusion module. When generating each output symbol, the VideoVLM model uses cross-modal attention to focus on the most relevant part of the visual features.

[0041] (3) Feedforward neural network layer: Each layer of cross-modal attention is followed by a feedforward network for further feature mapping and enhancing the expressiveness of the output.

[0042] (4) Residual connections and layer normalization: ensure smooth information flow and stable gradients. Residual connections are used to help the model retain original features in deep layers and prevent gradient disappearance.

[0043] 9. Output: language description or operation instructions (1) The final output is a language description or operation instruction for the user's operation behavior, such as "click a button", "select a date", "enter text", etc.

[0044] (2) These instructions can be used directly to guide RPA tools to perform specific tasks.

[0045] Specifically, the VideoVLM model has been improved and enhanced in the following aspects: 1. Through the extraction of video key frames and the combination of time series information, the VideoVLM model can achieve detailed understanding and time annotation of user operation behaviors.

[0046] 2. Support high-resolution input (1344×1344 pixels) to ensure accurate recognition of complex interface elements in the video.

[0047] 3. Through multi-stage fine-tuning (command fine-tuning and time positioning fine-tuning), the accuracy of the VideoVLM model on user operation behavior is improved.

[0048] Furthermore, the goal of the instruction fine-tuning is to enhance the model's ability to understand user operation videos, especially in terms of identifying interface changes and user specific operation behaviors. The specific steps of the instruction fine-tuning process are as follows: Dataset construction: Use a custom RPA instruction dataset for fine-tuning. This dataset is generated by GPT-4. The data source is the interface screenshots and corresponding operation instructions of RPA during runtime. Specific samples include: 1. Video clip description: For example, "The user selected option A in the form"; 2. Target output RPA: generated code snippet; 3. Data volume and learning rate: A total of 330k user operation video samples were used for fine-tuning, and the learning rate was 4e-6 to ensure that the VideoVLM model has strong generalization ability in video understanding.

[0049] Furthermore, the goal of the time positioning fine-tuning is to train the VideoVLM model to understand the specific timestamps of user operation behaviors and to accurately mark the times in the user operation videos. The specific process is as follows: 1. Dataset construction: Use screenshots of the RPA application interface to generate automated videos, and train a target detection model based on YOLOv8 to detect various interface elements; then, generate dynamic videos by adding random stops and noise to these interface element screenshots. Through target detection, automatically determine the appearance and disappearance of the panel, and generate corresponding time annotation data; split the long sequence video into multiple short sequences, and store the time and operation behavior description in JSON format.

[0050] 2. Learning rate: The learning rate for temporal localization fine-tuning is set to 1e-6 to accommodate the complexity of timestamp information and ensure precise understanding of time series operations.

[0051] The VideoVLM model, which is fine-tuned in two stages, has the following key capabilities: 1. Behavior and object mapping: The VideoVLM model can automatically establish a mapping relationship between behavior and interface objects by combining the changes in interface elements with the operation behaviors in the video.

[0052] 2. Operation instruction generation: By decoding visual features and text features through the ViT decoder, the system can generate a detailed operation sequence and associate time information with each operation behavior. These operation sequence instructions are further used to automatically generate RPA execution instructions.

[0053] In addition, the panel detection of the object detection module will input the signals of the appearance and disappearance of important interface elements to the video understanding module, and the video understanding model will perform feature fusion to better understand the context of user operations. This fusion enables the VideoVLM model to accurately locate the state changes of interface elements, thereby generating more accurate automation processes.

[0054] In this embodiment, based on the key control analysis module to identify and analyze the user operation interface, the intelligent instruction calling module is used to automatically execute the basic operation call operation instructions, and the operation instruction encapsulation for the intelligent component operation is performed to realize dynamic operation behaviors in different scenarios. The intelligent instruction calling module is one of the core modules used in the present invention to realize the RPA system to process user interface operations more efficiently and intelligently. The task of this module is to call the appropriate operation instructions for automatic execution after identifying the user operation behavior, especially in processing intelligent components and complex dynamic interface elements. The encapsulated high-level operation instructions greatly simplify the user's automated operation logic.

[0055] Specifically, the intelligent instruction calling module includes basic operation instruction processing, packaging and processing of intelligent components, and adaptation and configuration of dynamic elements; Furthermore, the basic operation instruction processing generates basic operation instructions based on the interface elements and operation behaviors recognized by the video understanding module. In automated operations, the most common instructions include: 1. Click: click on buttons, links, drop-down menus, etc.; 2. Input: Enter text in the text box; 3. Scroll: Scroll the page content.

[0056] These basic operations usually correspond to direct user interactions in the interface, such as clicking a button or entering text. The intelligent command calling module generates basic operation instructions based on the interface elements and user operation behaviors identified by the video understanding module.

[0057] Specifically, the encapsulation and processing of the intelligent component encapsulates the interactive logic of the interface elements based on the analysis results of the key control analysis module, obtains the corresponding operation instruction encapsulation, and passes in variables according to the dynamic elements in the interface elements to realize dynamic operation behaviors in different scenarios. The dynamic elements refer to certain values ​​or options that change dynamically according to the context during the operation. Intelligent component operation instructions are different from basic operation instructions. They involve more complex interface logic, usually require multiple steps, or require processing of dynamic elements on the interface elements. In the present invention, the core feature of the intelligent component operation instructions is the encapsulation of complex logic, making the operation simpler and more universal.

[0058] Furthermore, the operation logic of the smart component operation instructions first depends on the detection results of the key panel analysis module. For example, the date picker, drop-down box, verification code panel, etc. are all objects identified by the key panel analysis module, and these panels have specific interaction logic in operation: 1. Date selector: You need to select a specific date based on the user's input; 2. Drop-down box: You need to open the drop-down list and select a specific option; 3. Captcha panel: may involve image recognition or external decoding steps.

[0059] In order to make these operations reusable in different scenarios, smart components are designed to encapsulate their specific interaction logic. For example, in the encapsulation of the date selection component, the process of the user selecting a date is encapsulated as an instruction, the date is passed in as a variable, and the operation content is dynamically generated; in the encapsulation of the drop-down box selection component, the logic of opening the drop-down box, scrolling, and selecting a specific option is encapsulated as an instruction.

[0060] Furthermore, in the automated operation of RPA, dynamic elements need to be processed in many cases, that is, some values ​​or options change dynamically according to the context during the operation. In order to support this flexibility, smart components allow variables to be passed in so that they can be reused in different scenarios. For example, in date selection, the dynamic element date can change dynamically according to user input or specific context; when processing a drop-down box, the user's selection of a specific option can be a value passed in by external logic according to the actual scenario. In this way, smart components can adapt to different operation scenarios, improving the flexibility and versatility of automated operations.

[0061] Specifically, the adaptation and configuration of the dynamic elements is based on the analysis results of the key control analysis module to identify the dynamic elements and provide corresponding configuration options, and finally select the corresponding operation instruction package for execution through the configuration options. The processing of dynamic elements is a relatively complex part of the RPA system, especially when processing verification codes, dynamically generated options, etc., the traditional recording method is often difficult to ensure the correctness of the operation. In order to meet this challenge, this paper Ming uses a ready-made dynamic element identification and configuration selection method. In the intelligent instruction calling module, dynamic elements are classified into a group of similar elements through the existing identification method (calling the existing RPA method: identifying similar elements). The problem of dynamic elements becomes selecting any one from a group of similar elements. For example, the specific date element in the date panel is usually regarded as an input area that needs to be processed dynamically; the dynamic options in the drop-down list can be identified as a group of selection elements with similar attributes.

[0062] The present invention determines the intention of the user's operation through intelligent analysis of the user's operation behavior. In addition to basic operation instructions (such as click, input), the module also supports the call of intelligent component operation instructions (complex scenarios such as date, drop-down box, verification code, chart, etc.). For example, for the date selection panel, the system will identify the date selected by the user and call the corresponding encapsulation instruction to complete the date selection operation. For dynamic elements such as verification codes and drop-down boxes, the present invention encapsulates the processing method through a pre-trained YOLOv8 target detection model and a custom script, automatically identifies these interface elements during recording, generates corresponding operation instructions, and avoids manual adjustments by users in subsequent processes.

[0063] In this embodiment, in order to make the intelligent instruction calling module adapt to different platforms and applications, this module is based on operating system level simulation and combines the encapsulation of intelligent components to achieve cross-platform adaptation capabilities, specifically including: 1. Operating system event monitoring and simulation: Through WiinAPI or macOS Accessibility API, the intelligent command calling module can monitor operating system events and simulate user input to ensure that the system can work normally on various platforms and achieve seamless operation of different software and platforms.

[0064] 2. Cross-platform component operation encapsulation: Using frameworks such as Electron or Qt, the intelligent instruction calling module encapsulates the operation logic for systems such as Windows, macOS, and Linux, thereby ensuring that these intelligent components can be used normally on different systems and maximizing the adaptability of the system.

[0065] The present invention, through the combination of the video understanding module and the key control analysis module, can more accurately identify the changes in user operation behaviors and interface elements, establish a mapping relationship between operation behaviors and interface elements, so as to better understand the logic behind user behavior, thereby improving the accuracy and completeness of the automation process. In addition, the intelligent instruction calling module of the present invention can encapsulate instructions for basic operations and intelligent component operations, support dynamic operation behaviors in different scenarios, enable the system to adapt to different operation scenarios, and improve the flexibility and versatility of automated operations. The cross-platform support and operation adaptation module of the present invention can monitor operating system events and simulate user input, ensure that the system can work normally on various platforms, and achieve seamless operation of different software and platforms, with strong adaptability.

[0066] In summary, the present invention achieves a deep understanding of user operation behaviors and automatic generation of RPA instructions, thereby improving the accuracy and completeness of the automation process.

Claims

1. A method for generating RPA instructions based on user operation videos, characterized in that: The steps include: Step 1: Identify the interface elements in the web page in the user operation video based on the target detection algorithm, and obtain the timestamp of the interface element change; Step 2: Analyze the user operation recording video based on the video understanding model, obtain single-frame image and text features, inject the timestamp into the single-frame image to form visual features, and then fuse the visual features and text features to obtain multimodal features; Step 3: Decode the multimodal features to obtain the mapping relationship between interface elements and user operation behaviors, and then encapsulate the mapping relationship into RPA instructions.

2. The RPA instruction generation method based on user operation video according to claim 1 is characterized in that: In step 1, before identifying interface elements, the target detection algorithm first collects screenshots of the user operation interface, then annotates the collected screenshots of the user operation interface to obtain an annotated data set, and then builds a target detection model, and inputs the annotated data set into the target detection model for model training; based on the trained target detection model, the interface elements in the web page in the user operation video are identified, and the timestamp of the interface element change is obtained.

3. The RPA instruction generation method based on user operation video according to claim 2 is characterized in that: The interface elements include a date selection panel, a drop-down box, a text input box, a button, a verification code panel, a slider and a scroll bar; the data annotation includes a position annotation and a category annotation of the interface elements in each user operation interface screenshot.

4. The RPA instruction generation method based on user operation video according to claim 2 is characterized in that: The model used by the target detection algorithm is the YOLOv8 model. The training process of the YOLOv8 model is based on the features of the interface element category annotations in the interface screenshots of the labeled data set learning interface, and the weights are optimized by calculating the loss function.

5. The RPA instruction generation method based on user operation video according to claim 2 is characterized in that: In step 2, the process of forming visual features is to extract and preprocess the frames of the user operation video to obtain a preprocessed single-frame image; encode the single-frame image through the ViT encoder of the video understanding model, and then use the target detection model trained in step 1 to identify the interface elements in the single-frame image and inject the timestamp of the interface element change, and finally locate the timestamp of the operation behavior to obtain the visual features.

6. The RPA instruction generation method based on user operation video according to claim 4 is characterized in that: In step 2, the process of obtaining text features is to record the user operation video input operation description, convert the operation description into a high-dimensional text vector and encode it to obtain text features.

7. The RPA instruction generation method based on user operation video according to claim 4 is characterized in that: The process of obtaining the multimodal features is to align the visual features with the text features using a self-attention mechanism and a fully connected network, and to fuse the visual features with the text features through a cross-modal attention mechanism to obtain the multimodal features.

8. The RPA instruction generation method based on user operation video according to claim 1 is characterized in that: In step 3, the multimodal feature decoding process is to use a cross-modal Transformer decoder to decode the multimodal features, combine the user operation behavior in the video, and automatically establish a mapping relationship between the user operation behavior and the interface elements through intelligent analysis of the changes in the interface elements. Finally, the mapping relationship is encapsulated into RPA instructions, and time information is associated with each user operation behavior.

9. The RPA instruction generation method based on user operation video according to claim 8 is characterized in that: The cross-modal Transformer decoder includes a self-attention layer, a cross-modal attention layer, a feedforward neural network layer, and residual connections and layer normalization; the self-attention layer is used to process the current context of the cross-modal Transformer decoder; the cross-modal attention layer extracts relevant information from visual features and text features; each layer of cross-modal attention in the feedforward neural network layer is followed by a feedforward network for further feature mapping to enhance the expressiveness of the output; the residual connections and layer normalization ensure smooth information flow and stable gradients, helping the video understanding model to retain original features in deep layers and prevent gradient disappearance.

10. The RPA instruction generation method based on user operation video according to claim 1 is characterized in that: Fine-tune the video understanding model for instructions and time positioning to enhance the model's ability to understand user operation videos and specific timestamps of user operation behaviors. The instruction fine-tuning is performed using the RPA instruction dataset generated by GPT. The data source of the dataset is the screenshots of the RPA at runtime and the corresponding operation instructions. The specific samples include video clip descriptions and generated RPA code snippets. 330k user operation video samples are used for fine-tuning, and the learning rate is 4e-6. The time positioning fine-tuning is to use the screenshot of the RPA application interface to generate an automated video, automatically determine the appearance and disappearance of interface elements through the target detection algorithm, generate corresponding timestamp annotation data, divide the long sequence video into multiple short sequences, and store the timestamp and operation description in JSON format. The learning rate of time positioning fine-tuning is set to 1e-6.

Images