RPA code generation method based on HTML and image

Abstract

The invention discloses an RPA code generation method based on an HTML and an image, and the method comprises the following steps: S1, receiving multi-modal input data provided by a user, the multi-modal input data comprising an HTML structure file of a target business system, an interface screenshot image and a natural language operation demand description; s2, performing standardized analysis on the HTML structure file, extracting DOM tree structure information, and performing visual feature enhancement processing on the interface screenshot image; and S3, inputting the processed HTML structure information, the interface screenshot image and the natural language operation requirement into a pre-trained visual language model, and outputting a target XPath path and a Python code sequence through the visual language model. According to the method, comprehensive understanding of a complex service system is realized by combining HTML structure information and an interface image, so that efficient and accurate RPA script codes capable of being executed on multiple platforms are automatically generated.

Description

Technical Field

[0001] The field of the present invention is the field of robotic process automation technology, and specifically relates to an RPA code generation method based on HTML and images. Background Art

[0002] With the rapid development of information technology, enterprise business systems have become increasingly complex, involving multiple platforms such as Web, Windows, Mac, and mobile terminals, and also including diverse user interface components such as date pickers, dropdown boxes, etc. In such a context, traditional RPA systems rely on manually writing script codes, which not only consumes a large amount of time but is also prone to errors. Due to the frequent changes in business requirements, the method of manually writing codes is difficult to adapt quickly, greatly affecting the business efficiency of enterprises.

[0003] Existing RPA code generation methods are mainly based on natural language descriptions. Although they improve the efficiency of code generation to a certain extent, there are many deficiencies. First of all, relying solely on natural language descriptions, it is difficult to comprehensively understand the interface context of the current business system, resulting in the generated codes being unable to accurately match the actual business processes. Secondly, for business systems with complex logic processes, simple natural language descriptions cannot cover all interaction details, affecting the accuracy and executability of the generated RPA scripts. In addition, existing methods are usually optimized for a single platform (such as Web) and lack automation support for multiple terminals (Web, Windows, Mac, mobile phones, etc.), restricting the extensiveness of RPA applications. Summary of the Invention

[0004] The purpose of the present invention is to provide an RPA code generation method based on HTML and images. By combining HTML structure information and interface images, the present invention realizes a comprehensive understanding of complex business systems, thereby automatically generating efficient, accurate, and RPA script codes that can be executed on multiple platforms.

[0005] The technical solution of the present invention: An RPA code generation method based on HTML and images includes the following steps:

[0006] S1. Receive multimodal input data provided by the user, where the multimodal input data includes an HTML structure file of the target business system, an interface screenshot image, and a natural language operation requirement description;

[0007] S2. Perform standardized parsing on the HTML structure file, extract the HTML tag sequence, and perform visual feature enhancement processing on the interface screenshot image;

[0008] S3. Input the HTML tag sequence, the interface screenshot image, and the natural language operation requirements into a pre-trained vision-language model, and output the target XPath path and the Python code sequence through the vision-language model.

[0009] In the above HTML and image-based RPA code generation method, in step S3, the vision-language model includes the following components:

[0010] Text encoder: The text encoder encodes the HTML tag sequence and the natural language operation requirements based on the Transformer architecture and outputs a text feature vector.

[0011] Vision encoder: The vision encoder extracts features from the interface screenshot image based on the VIT model and outputs a vision feature vector.

[0012] Multimodal fusion module: The multimodal fusion module fuses the text feature vector and the vision feature vector through a multi-head cross-attention mechanism to generate a multimodal feature vector.

[0013] Task decoder: The task decoder decodes the multimodal feature vector into the target XPath path and the Python code sequence based on the Transformer architecture.

[0014] In the aforementioned HTML and image-based RPA code generation method, the text encoder receives the HTML tag sequence and the natural language operation requirements as text inputs, uses a tokenizer to convert the text into a token sequence, then performs an Embedding embedding operation on the token sequence to map each token sequence to a high-dimensional vector space, and then adds position encoding to preserve the order information of the text sequence, thereby generating a text feature vector.

[0015] In the aforementioned HTML and image-based RPA code generation method, the vision encoder receives the interface screenshot as an image input, divides the image into multiple pixel blocks, then maps the pixel blocks to a high-dimensional vector space, and then adds position encoding to preserve the spatial information of the image pixel blocks, thereby generating a vision feature vector.

[0016] In the aforementioned HTML and image-based RPA code generation method, the workflow of the multimodal fusion module includes:

[0017] Perform self-attention calculation on the text feature vector to construct the dependency relationship between HTML tags.

[0018] Perform spatial attention calculation on the vision feature vector to locate the interactive component area in the interface screenshot.

[0019] Establish the mapping relationship between HTML tags and image regions through the cross-attention mechanism to generate multi-modal feature vectors with spatial semantic associations.

[0020] For the aforementioned RPA code generation method based on HTML and images, the task decoder extracts the elements to be operated from the DOM tree structure according to the received multi-modal feature vectors, generates their XPath paths in the form of sequence annotation, analyzes the logical order of the business process at the same time to determine the execution order of each operation element, and finally automatically generates or invokes Python code in the database based on the extracted elements, XPath paths and logical parsing results.

[0021] For the aforementioned RPA code generation method based on HTML and images, the training method of the vision-language model includes:

[0022] Construct a training dataset: collect HTML structure files, interface screenshot images, manually annotated XPath paths and Python code to form a dataset;

[0023] Adopt a two-stage training strategy: freeze the vision encoder in the first stage and only train the text encoder and the task decoder; jointly train all modules in the second stage;

[0024] The optimization objective is a multi-task loss function:

[0025] L total = λL element +(1 - λ)L code ;

[0026] In the formula: represents the element extraction loss;

[0027] represents the code generation loss;

[0028] where N and M are the number of training samples of the corresponding loss functions respectively, T is the maximum length of the XPath path sequence, y ij is the one-hot encoding of the j-th token of the i-th sample in the true XPath path, is the probability distribution of the j-th token of the i-th sample predicted by the decoder; S is the maximum length of the Python code sequence; z kl is the one-hot encoding of the l-th token of the k-th sample in the true Python code, is the probability distribution of the l-th token of the k-th sample predicted by the decoder; λ represents the weight.

[0029] Compared with the prior art, the multi-modal input of the present invention combines the HTML structure and the interface image, enhancing the system's ability to understand the context, enabling a more accurate grasp of the overall picture of the business system, and generating code that better meets the actual requirements. By specifically constructing a visual language model, the present invention can accurately analyze the structure and functions of complex UI components, providing a guarantee for generating efficient RPA code. The present invention directly generates executable Python code, simplifies the user operation process, reduces the technical threshold of RPA development, and enables more non-professional developers to use RPA technology to achieve business automation. In addition, the element extraction function of the present invention eliminates the cumbersome steps of manually parsing HTML, improves the intelligence level of the system, and reduces the workload and error probability of manual operations. The present invention can automatically parse the XPath path of interface elements to ensure that the generated RPA script can stably and reliably locate and operate target components. The code generated by the present invention can be executed in a multi-terminal environment, meeting the automation requirements of different business systems and expanding the application scope of RPA. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It shows a schematic flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] The present invention will be further described below in conjunction with the drawings and embodiments, but it shall not be used as a basis for limiting the present invention.

[0032] Embodiment: An RPA code generation method based on HTML and images, as Figure 1 shown, includes the following steps:

[0033] S1. Receive multi-modal input data provided by the user. The multi-modal input data includes an HTML structure file of the target business system, an interface screenshot image, and a natural language operation requirement description. In this step, for example, the HTML structure file is an HTML file provided by the user for an e-commerce website login page, which contains the structure information of the login form. The interface screenshot image is, for example, a screenshot of the login page, showing the actual visual effect of the page. The natural language operation requirement description is, for example, "Enter the username and password on the login page, and then click the login button to complete the login operation", and at the same time, provide the outerHTML source code and XPath selection information of the relevant target elements.

[0034] S2. Standardize and parse the HTML structure file, extract the HTML tag sequence, and perform visual feature enhancement processing on the interface screenshot image, including image denoising, resolution adjustment, and key area cropping. In this step, standardizing the HTML file is to extract its DOM tree structure, analyze the hierarchical relationships of elements such as the login form, username input box, password input box, and login button, and these hierarchical relationships represent the HTML tag sequence. Perform image denoising enhancement on the screenshot to highlight the key elements in the page, such as the outlines of input boxes and buttons. Perform operations such as random cropping, scaling, and color perturbation on the interface screenshot to increase the diversity of the image.

[0035] S3. Input the processed HTML structure information, interface screenshot image, and natural language operation requirements into a pre-trained vision-language model, and output the target XPath path and Python code sequence through the vision-language model.

[0036] In this step, the vision-language model includes the following components:

[0037] Text encoder: The text encoder encodes the HTML tag sequence and natural language operation requirements based on the GPT-based Transformer architecture and outputs text feature vectors. The text encoder receives the HTML tag sequence and natural language operation requirements as text inputs, uses a tokenizer to convert the text into a token sequence, then performs an Embedding embedding operation on the token sequence to map each token sequence to a high-dimensional vector space, and then generates text feature vectors by adding positional encoding to retain the order information of the text sequence.

[0038] Vision encoder: The vision encoder extracts features from the interface screenshot image based on the VIT model (Vision Transformer) and outputs visual feature vectors. The vision encoder receives the interface screenshot as an image input, divides the image into multiple pixel blocks (16×16), then maps the pixel blocks to a high-dimensional vector space, and then generates visual feature vectors by adding positional encoding to retain the spatial information of the image pixel blocks.

[0039] Multimodal fusion module: The multimodal fusion module fuses the text feature vectors and visual feature vectors through a multi-head cross-attention mechanism to generate multimodal feature vectors. The workflow of the multimodal fusion module includes:

[0040] Perform self-attention calculation on the text feature vector to construct the dependency relationship between HTML tags; that is, allow interactions between features within the same modality, such as between different words in the text or between different patches in the image

[0041] Perform spatial attention calculation on the visual feature vector to locate the interactive component area in the interface screenshot; allow interactions between features of different modalities, such as which parts of the image the text features focus on and vice versa

[0042] Establish the mapping relationship between HTML tags and image regions through the cross-attention mechanism to generate a multimodal feature vector with spatial semantic associations, which contains text and visual information

[0043] Task decoder: The task decoder decodes the multimodal feature vector into a target XPath path and a Python code sequence based on the Transformer architecture. The task decoder extracts the elements to be operated on from the DOM tree structure according to the received multimodal feature vector, generates its XPath path in the form of sequence annotation, analyzes the logical order of the business process, determines the execution order of each operation element, and finally automatically generates or invokes the Python code in the database based on the extracted elements, XPath path, and logical parsing results. In this step, the logical order of the business process is based on the human operation order and is reflected in the dataset. The vision-language model learns this order relationship through supervised fine-tuning (SFT). For example, in the operation of a dropdown box, the model first learns to click on the dropdown box, then expand the dropdown panel, and finally select the target item. This order is data-driven, and the model autonomously learns and generates RPA code that conforms to human operation logic

[0044] Among them, the training method of the vision-language model includes:

[0045] Construct a training dataset: Collect HTML files, interface screenshot images, and manually annotated XPath paths and Python codes to form a dataset; this dataset contains a series of samples, and each sample includes: HTML source code, the corresponding interface screenshot, user operation requirement description, a list of XPath paths of the target elements, a list of outerHTML source codes of the target elements, and the corresponding Python code (target output). An example of the HTML source code in the training data of this embodiment is as follows

[0046] [CLS] <title>Login< / title> <form action＝" login"method＝"POST"> <div class＝"form-group"> <label for＝"username">Username< / label> <input type＝"text"id＝"username"name＝"username"placeholder＝"输入用户名"required> <label for＝"password">Password< / label> <input type＝"password"id＝"password"name＝"password"placeholder＝"输入密码"required> <div class＝"form-group"> <label for＝"remember">Remember Password< / label> <input type＝"checkbox"id＝"remember"name＝"remember"checked> <div class＝"form-group"> <label for＝"agree"> <input type＝"checkbox"id＝"agree"name＝"agree"> Agree to the user agreement< / label> <button type＝"submit"class＝"login-btn">Login< / button> < / form> [SEP]

[0047] An example of the user operation requirement description is as follows: [CLS] Complete the login operation by entering the username username and password password within the login panel [SEP];

[0048] Examples of element XPath paths are as follows: / / form[@action=' / login'] / input[@id='username'];

[0049] Examples of Python code are as follows:

[0050]

[0051]

[0052]

[0053] Subsequently, data augmentation is performed on the dataset: the input data is augmented to improve the generalization ability of the model, specifically including: randomly cropping, scaling, and color perturbing the interface screenshots to increase the diversity of the images; slightly randomly changing the HTML structure, such as randomly deleting or adding some unimportant attributes while retaining the key structure, to enhance the model's robustness to noise.

[0054] A two-stage training strategy is adopted: in the first stage, the visual encoder is frozen, and only the text encoder and task decoder are trained; in the second stage, all modules are jointly trained; in this step, the training objectives include: element extraction, code generation, and joint training; training parameters: use the Adam optimizer, with an initial learning rate of 3e-5, a batch size of 16, and 50 training epochs.

[0055] Among them, for element extraction, the decoder generates XPath paths. An XPath path can be regarded as a sequence, and each token represents a part of the XPath. The cross-entropy loss function (Cross-Entropy Loss) is used to calculate the loss for each token generated by the decoder: the formula is as follows:

[0056]

[0057] In the formula: N is the number of training samples corresponding to the loss function, T is the maximum length of the XPath path sequence, y ij is the one-hot encoding of the j-th token of the i-th sample in the true XPath path, is the probability distribution of the j-th token of the i-th sample predicted by the decoder;

[0058] For code generation, the decoder generates Python code. Python code can also be regarded as a sequence. The cross-entropy loss function (Cross-Entropy Loss) is used to calculate the loss for each token generated by the decoder:

[0059]

[0060] Where M is the number of training samples corresponding to the loss function, and S is the maximum length of the Python code sequence; z kl is the one-hot encoding of the l-th token of the k-th sample in the true Python code, and is the probability distribution of the l-th token of the k-th sample predicted by the decoder;

[0061] Joint training:

[0062] Train element extraction and code generation simultaneously so that the model can better understand the relationship between page elements and code generation. In this step, the element extraction loss and the code generation loss are weighted and summed:

[0063] L total = λL element + (1 - λ)L code

[0064] Where λ represents the weight used to balance the importance of element extraction and code generation. It can usually be set to 0.5, indicating equal importance.

[0065] Based on the above rpa code generation method, this embodiment provides a practical application of e-commerce order export automation.

[0066] Among them, the input data includes:

[0067] HTML file: containing the DOM structure of the order table;

[0068] Interface screenshot: showing the pagination button and the table scroll bar;

[0069] Natural language instruction: "Loop to export the order data of the first 10 pages".

[0070] Model processing:

[0071] Identify that the XPath of the pagination button is:

[0072] / / button[contains(@class,'pagination')];

[0073] Detect the visual features of the table scroll area and generate the code page.evaluate("window.scrollBy(0,500)").

[0074] Output code:

[0075]

[0076] In the above e-commerce operation scenario, the daily average processing volume of e-commerce order export has been significantly improved.

[0077] In addition, the present invention has undergone cross-platform testing, and UI compatibility testing is performed in a Windows / Android hybrid environment, with an error rate lower than 1.5%.

[0078] In summary, the multi-modal input of the present invention combines the HTML structure and interface images, enhancing the system's ability to understand the context, enabling a more accurate grasp of the overall picture of the business system, and generating code that better meets the actual needs. By specifically constructing a visual language model, the present invention can accurately analyze the structure and functions of complex UI components, providing a guarantee for generating efficient RPA code. The present invention directly generates executable Python code, simplifies the user operation process, lowers the technical threshold of RPA development, and enables more non-professional developers to use RPA technology to automate business processes. In addition, the element extraction function of the present invention eliminates the cumbersome steps of manual HTML parsing, improves the intelligence level of the system, reduces the workload and error probability of manual operations. The present invention can automatically analyze the XPath path of interface elements, ensuring that the generated RPA script can stably and reliably locate and operate target components. The code generated by the present invention can be executed in a multi-terminal environment, meeting the automation requirements of different business systems and expanding the application scope of RPA.

Claims

1. A method for generating RPA code based on HTML and images, characterized in that, Including the following steps: S1. Receive the multi-modal input data provided by the user, where the multi-modal input data includes the HTML structure file of the target business system, the interface screenshot image, and the natural language operation requirement description; S2. Perform standardized parsing on the HTML structure file to extract the HTML tag sequence, and perform visual feature enhancement processing on the interface screenshot image; S3. Input the HTML tag sequence, the interface screenshot image, and the natural language operation requirement into the pre-trained vision-language model, and output the target XPath path and the Python code sequence through the vision-language model.

2. The RPA code generation method based on HTML and images according to claim 1, characterized in that, In step S3, the vision-language model includes the following components: Text encoder: The text encoder encodes the HTML tag sequence and the natural language operation requirement based on the Transformer architecture, and outputs the text feature vector; Vision encoder: The vision encoder extracts features from the interface screenshot image based on the VIT model, and outputs the visual feature vector; Multi-modal fusion module: The multi-modal fusion module fuses the text feature vector and the visual feature vector through the multi-head cross-attention mechanism to generate the multi-modal feature vector; Task decoder: The task decoder decodes the multi-modal feature vector into the target XPath path and the Python code sequence based on the Transformer architecture.

3. The method for generating RPA code based on HTML and images according to claim 2, wherein: The text encoder receives the HTML tag sequence and the natural language operation requirement as text input, uses the tokenizer to convert the text into a token sequence, then performs Embedding embedding operation on the token sequence to map each token sequence to a high-dimensional vector space, and then adds position encoding to retain the order information of the text sequence, thereby generating the text feature vector.

4. The method for generating RPA code based on HTML and images according to claim 2, wherein: The vision encoder receives the interface screenshot as image input, divides the image into multiple pixel blocks, then maps the pixel blocks to a high-dimensional vector space, and then adds position encoding to retain the spatial information of the image pixel blocks, thereby generating the visual feature vector.

5. The method for generating RPA code based on HTML and images according to claim 2, wherein: The working process of the multi-modal fusion module includes: Perform self-attention calculation on the text feature vector to construct the dependency relationship between HTML tags; Perform spatial attention calculation on the visual feature vector to locate the interactive component area in the interface screenshot; Establish the mapping relationship between the HTML tag and the image area through the cross-attention mechanism to generate the multi-modal feature vector with spatial semantic association.

6. The method for generating RPA code based on HTML and images according to claim 2, wherein: The task decoder extracts the elements to be operated from the DOM tree structure according to the received multi-modal feature vector, generates its XPath path in the form of sequence annotation, analyzes the logical order of the business process at the same time, determines the execution order of each operation element, and finally automatically generates or calls the Python code in the database based on the extracted elements, XPath path, and logical parsing result.

7. The method for generating RPA code based on HTML and images according to claim 2, wherein The training method of the vision-language model includes: Construct a training data set: Collect the HTML structure file, the interface screenshot image, and the manually annotated XPath path and Python code to form a data set; Adopt a two-stage training strategy: In the first stage, freeze the visual encoder and only train the text encoder and the task decoder; in the second stage, jointly train all modules; The optimization objective is a multi-task loss function: Ltotal = λLelement+(1-λ)Lcode; In the formula: represents the element extraction loss; Indicates the code generation loss; where N and M are the number of training samples corresponding to the loss function, T is the maximum length of the XPath path sequence, y ij is the one-hot encoding of the j-th token of the i-th sample in the true XPath path, is the probability distribution of the j-th token of the i-th sample predicted by the decoder; S is the maximum length of the Python code sequence; zkl is the one-hot encoding of the l-th token of the k-th sample in the true Python code, z kl is the probability distribution of the l-th token of the k-th sample predicted by the decoder; λ represents the weight.

Images