RPA process script generation method and system based on large model and screen recording scene, and storage medium

Abstract

The application provides a kind of RPA process script generation method and system based on large model and screen recording scene and storage medium, it is related to computer technology field.The method includes the following steps: screen video of user operation is recorded by RPA software, and different device screen sources are covered;The recorded video is preprocessed, including noise reduction, adjusting frame number and filtering invalid action;Video is analyzed using a pre-trained video large model, and operation action and target object are identified;Action and target object templates are generated according to the identification results, and operation behavior and target object information are described;The template is mapped to RPA script, including component arrangement, parameter setting and process logic;The generated script is executed on the RPA platform to realize operation automation.The application automatically identifies the operation steps, interface elements and interaction logic in the screen recording video through deep learning and video processing technology, without manual intervention to generate the corresponding RPA script, greatly improving the script generation efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an RPA process script generation method, system, and storage medium based on large models and screen recording scenarios. Background Technology

[0002] In this era of information overload, screen recording, as an intuitive and easy-to-understand method of information dissemination, is widely used in education, training, technical support, and software demonstrations. Especially in the wave of digital transformation, enterprises urgently need to improve operational efficiency and reduce labor costs through automation. RPA technology has emerged to meet this need, enabling software robots to simulate human operations and automate highly repetitive and well-defined business processes. However, writing traditional RPA scripts relies on the programming skills of professionals, which is not only time-consuming and costly but also difficult to adapt to rapidly changing business needs.

[0003] Furthermore, with the rapid development of video processing technology and artificial intelligence, the ability to analyze, recognize, and understand video content has significantly improved. This makes it possible to automatically extract key information from screen recordings and convert it into executable scripts. In particular, the application of deep learning technology to large video models enables models to learn and understand complex scenes, actions, and processes in videos, laying a solid foundation for the automated conversion of screen recordings into RPA scripts. Summary of the Invention

[0004] Purpose of the invention: To propose an RPA process script generation method, system and storage medium based on large models and screen recording scenarios, aiming to fundamentally solve the problems of low efficiency, error susceptibility and difficulty in adapting to complex processes in the traditional RPA script generation process.

[0005] In a first aspect, the present invention proposes an RPA process script generation method based on a large model and screen recording scenario, comprising the following steps:

[0006] Record screen videos of user operations using RPA software, covering screen sources from different devices;

[0007] Preprocess the recorded video, including noise reduction, frame rate adjustment, and filtering out invalid actions;

[0008] Analyze videos using pre-trained large video models to identify operational actions and target objects;

[0009] Based on the recognition results, action and target templates are generated to describe the operation behavior and target information;

[0010] The template is mapped to an RPA script, which includes component arrangement, parameter settings and process logic;

[0011] The generated scripts are executed on the RPA platform to automate operations.

[0012] In a further embodiment of the first aspect, the training method of the video large model includes: optimizing model parameters through supervised learning based on historical screen recording videos and corresponding RPA script templates, so that the model can accurately identify actions and target objects;

[0013] The loss function for training large video models is:

[0014]

[0015] In the formula, N is the number of training samples; y i The true label for the i-th sample includes the action and the type of the target object; θ represents the model prediction probability; θ is the model parameter vector; and λ is the L2 regularization coefficient.

[0016] In a further embodiment of the first aspect, the preprocessing step employs an adaptive noise reduction algorithm to dynamically adjust noise reduction parameters according to the video frame rate while preserving key operational details.

[0017] The adaptive noise reduction algorithm uses the following filtering model:

[0018]

[0019] In the formula, V t ′ ω is the denoised video frame at time t; i The time window weighting coefficient satisfies Let be the mean vector of the neighboring frames at time t; represents the dynamic noise variance; ||*|| represents the L2 norm of the frame difference.

[0020] In a further embodiment of the first aspect, the video large model analysis video introduces a probabilistic graphical model for action recognition, mathematically expressed as:

[0021]

[0022] In the formula, This indicates that at time t, based on the previous t frames of video... Action A was identified. t and target object Q t The posterior probability; Indicates action A t and target object Q t Generate video sequences The probability of P(A) t Q t Action A tand target object Q t The conditional probability of P(Q) represents the logical correlation of operations; t ) is the target object Q t The prior probability is obtained from statistical analysis of historical screen recording data; A ′ Represents any action in the action set, and is related to A. t Belonging to the same action type space, including click, input, and drag; O ′ Represents any target object in the target object set, and Q t The same target object type space includes buttons, text boxes, and images.

[0023] In a further embodiment of the first aspect, the actions include mouse clicks, drags, inputs, and swipes, and the target objects include web page elements, images, buttons, and input boxes;

[0024] The logical expressions for the action and target templates are:

[0025] T = {(A h Q j C j )|j=1,2,…,M}

[0026] In the formula, T is the template set; A j Q represents the j-th action, belonging to the action set A = {click, input, drag, ...}; j Represents the j-th target object, including coordinate and type attributes; C j These are conditional constraints.

[0027] In a further embodiment of the first aspect, in the step of generating action and target template, intermediate description pseudocode is used to abstractly represent the operation steps and logical flow, and the pseudocode includes action type, target coordinates, operation parameters and time sequence.

[0028] In a further embodiment of the first aspect, in the step of mapping the template to an RPA script, the action and target information in the pseudocode are converted into script components supported by the RPA platform through a preset mapping rule. The mapping rule includes an action-component mapping table and a parameter conversion algorithm.

[0029] By dynamically adjusting the mapping rules through reinforcement learning, the state transition equation is:

[0030] s t+1 =f(s) t ,a t ,r t )

[0031] The reward function is defined as:

[0032]

[0033] In the formula, s t The mapping state at time t includes template features and script parameters; a t The mapping action at time t includes component selection and parameter adjustment; r t Let δ be the reward value at time t; exec A binary variable represents the success of the execution, with a value of 1 for success and 0 for failure; T script T represents the execution time of the RPA script. human For manual operation time; KL(P) target ||P generated ) represents the KL divergence between the target probability distribution and the generated distribution; α, β, and γ are weighting coefficients, satisfying α + β + γ = 1.

[0034] In a further embodiment of the first aspect, the RPA process script generation method further includes a script optimization step: during script execution, recording execution exceptions and efficiency data, updating the video big model and mapping rules through reinforcement learning, and optimizing the quality of subsequent script generation;

[0035] In the script optimization step, an operation behavior knowledge base is established, and high-frequency operation patterns are extracted into reusable script component templates, supporting user-defined component library expansion.

[0036] A second aspect of the present invention provides an RPA process script generation system for automatically executing the RPA process script generation method based on a large model and screen recording scenario described in the first aspect. The RPA process script generation system includes:

[0037] The recording module is used to record screen videos of user operations via RPA software;

[0038] The processing module is used to perform noise reduction, frame rate adjustment, and motion filtering on the recorded video.

[0039] The analysis module contains pre-trained large video models for identifying actions and objects in the video;

[0040] The template generation module is used to generate action and target templates based on the recognition results;

[0041] The script conversion module is used to map templates to RPA scripts;

[0042] An execution module is used to execute the generated RPA script on the RPA platform.

[0043] A third aspect of the present invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on an electronic device, causes the electronic device to perform the RPA process script generation method based on a large model and screen recording scenario as described in the first aspect.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) Automated generation of RPA scripts: Through deep learning and video processing technology, the operation steps, interface elements and interaction logic in the screen recording video are automatically identified, and the corresponding RPA scripts can be generated without manual intervention, which greatly improves the efficiency of script generation.

[0046] (2) Improve accuracy: Utilize the powerful feature extraction and pattern recognition capabilities of large models to accurately capture every key detail in the screen recording, ensuring that the generated RPA script is accurate and reducing errors caused by human factors.

[0047] (3) Adapting to complex processes: The system can handle complex operation processes that include multiple steps, conditional branches and loops, flexibly respond to various business scenarios, and meet the automation needs of different enterprises. Attached Figure Description

[0048] Figure 1 This is a flowchart of the RPA process script generation method in the embodiment.

[0049] Figure 2 A flowchart detailing the process of automating the video-to-RPA conversion workflow. Detailed Implementation

[0050] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0051] Before describing the following embodiments, the terms used below will be explained.

[0052] LLM (Large Language Model): A deep learning model trained on large amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0053] RPA (Robotic Process Automation) is an application that provides an alternative way to automate manual processes by mimicking the way end users operate on a computer.

[0054] Action and target template: An intermediate pseudocode used to describe actions in a video, which can abstractly represent the operation steps and logical flow in the video.

[0055] Target object: The specific object or operation target identified and extracted from the video. Examples include web page elements, images, buttons, and input boxes.

[0056] Actions: Mouse actions recognized in the video, such as clicking, double-clicking, dragging, etc.

[0057] RPA script: Robotic Process Automation script, used to automate the execution of tasks.

[0058] Example 1:

[0059] This embodiment discloses an RPA process script generation method based on a large model and screen recording scenario. The method records the automation scenario to be generated by screen recording, uses the large model's ability to understand images to annotate the screen recording, and finally uses semantic understanding to transform it into a specific RPA process script, thereby realizing the rapid understanding and orchestration of processes in a custom scenario.

[0060] Figure 1 To generate RPA process scripts based on large models and screen recording scenarios, this invention achieves this through the following steps:

[0061] Step S1: RPA video recording.

[0062] Record user actions via RPA screen recording. Supports recording from various sources (such as computer screens and mobile device screens), ensuring coverage of a wide range of user scenarios and device types.

[0063] Step S2: Video processing.

[0064] Preprocessing operations such as filtering, noise reduction, and video frame analysis are performed on RPA screen recording videos to improve the accuracy and efficiency of subsequent large-scale video model analysis.

[0065] Step S3: Large-scale video model analysis.

[0066] The pre-trained video model is used to perform deep analysis on the pre-processed video to identify the operation actions (such as clicking, dragging, inputting, etc.) and objects (such as buttons, text boxes, images, etc.) in the video.

[0067] Step S4: Generate action and target templates.

[0068] Based on the recognition results of the video big data model, corresponding action and target templates are automatically generated, and each operation and its target information are described in detail.

[0069] Step S5: RPA script generation.

[0070] The program converts and maps action and target template information into RPA scripts, including the arrangement of script components, parameter settings, and process logic, to ensure that the RPA scripts can accurately simulate the operation behavior in the screen recording video.

[0071] Step S6: Execute the RPA script.

[0072] The script generated by the mapping is executed on the RPA platform to automate the simulation and execution of operations in RPA screen recording videos.

[0073] Example 2:

[0074] Based on the aforementioned Embodiment 1, this embodiment discloses details of some steps.

[0075] Step S1: RPA video recording.

[0076] Record user actions via RPA screen recording. Supports recording from various sources (such as computer screens and mobile device screens), ensuring coverage of a wide range of user scenarios and device types.

[0077] Step S2: Video processing.

[0078] Preprocessing operations such as filtering, noise reduction, and video frame analysis are performed on RPA screen recording videos to improve the accuracy and efficiency of subsequent large-scale video model analysis.

[0079] In the preprocessing step, an adaptive noise reduction algorithm is used to dynamically adjust the noise reduction parameters according to the video frame rate, while preserving key operational details;

[0080] The adaptive noise reduction algorithm uses the following filtering model, as shown in equation (1):

[0081]

[0082] In the formula, V t ′ ω is the denoised video frame at time t; i The time window weighting coefficient satisfies Let be the mean vector of the neighboring frames at time t; represents the dynamic noise variance; ||*|| represents the L2 norm of the frame difference.

[0083] Step S3: Large-scale video model analysis.

[0084] The pre-trained video model is used to perform deep analysis on the pre-processed video to identify the operation actions (such as clicking, dragging, inputting, etc.) and objects (such as buttons, text boxes, images, etc.) in the video.

[0085] The logical expressions for the action and target templates are shown in equation (2):

[0086] T = {(A j Q j C j (2) |j=1,2,…,M}

[0087] In the formula, T is the template set; A j Q represents the j-th action, belonging to the action set A = {click, input, drag, ...}; j Represents the j-th target object, including coordinate and type attributes; C j These are conditional constraints.

[0088] The training methods for large video models include: optimizing model parameters through supervised learning based on historical screen recording videos and corresponding RPA script templates, so that the model can accurately identify actions and targets;

[0089] The loss function for training large video models is shown in equation (3):

[0090]

[0091] In the formula, N is the number of training samples; y i The true label for the i-th sample includes the action and the type of the target object; θ represents the model prediction probability; θ is the model parameter vector; and λ is the L2 regularization coefficient.

[0092] Step S4: Generate action and target templates.

[0093] Based on the recognition results of the video big data model, corresponding action and target templates are automatically generated, and each operation and its target information are described in detail.

[0094] In video big data analysis, a probabilistic graphical model is introduced for action recognition, and its mathematical expression is shown in equation (4):

[0095]

[0096] In the formula, This indicates that at time t, based on the previous t frames of video... Action A was identified. t and target object Q t The posterior probability; Indicates action A t and target object Q t Generate video sequences The probability of P(A) t Q t Action A t and target object Q t The conditional probability of P(Q) represents the logical correlation of operations; t ) is the target object Q t The prior probability is obtained from statistical analysis of historical screen recording data; A ′ Represents any action in the action set, and is related to A. t Belonging to the same action type space, including click, input, and drag; O ′ Represents any target object in the target object set, and Q t The same target object type space includes buttons, text boxes, and images.

[0097] Step S5: RPA script generation.

[0098] The program converts and maps action and target template information into RPA scripts, including the arrangement of script components, parameter settings, and process logic, to ensure that the RPA scripts can accurately simulate the operation behavior in the screen recording video.

[0099] Step S6: Execute the RPA script.

[0100] The script generated by the mapping is executed on the RPA platform to automate the simulation and execution of operations in RPA screen recording videos.

[0101] Example 3:

[0102] The following operations are all performed in the Chrome browser. The specific steps are as follows: Open the Chrome browser, enter the website address in the address bar, press Enter to access the webpage, then click "Login" in the upper right corner of the homepage, enter your username and password, then click "Read and Accept Baidu User Agreement," and finally click "Login." This is used as an example. The present invention uses a trained video parsing model and result verification to generate action and target object templates, and then converts and maps them into RPA scripts through a program. The specific implementation steps are as follows:

[0103] Step S1: Install RPA software and record video: The user installs RPA software on their computer and uses the video recording function to record the process of opening a browser, accessing Baidu, clicking login and entering the username and password, checking the user agreement, and finally clicking login.

[0104] Step S2: Video processing in Step 1: The recorded video is preprocessed using RPA software to remove invalid actions, noise, and video frames, and finally outputs a high-fidelity video.

[0105] Step S3: The video output from step S2 is sent to the video analysis big model. The big model can identify each operation step of the user in the video and its corresponding target object, and then output an action and target object template.

[0106] Step S4: Based on the template content output in step S3, the action and target templates are converted into RPA component scripts using the RPA conversion program.

[0107] Step S4-1: Open your browser

[0108] Step S4-2: Enter the URL

[0109] Step S4-3: Click the "Login" button

[0110] Step S4-4: Enter account

[0111] Step S4-5: Enter password

[0112] Step S4-6: Check the user agreement

[0113] Step S4-7: Click the "Login" button

[0114] Step S5: Script execution. The generated RPA script is run in the RPA execution machine to automate the Baidu login process.

[0115] The logical ideas behind the methods disclosed in the above embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs.

[0116] When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium may be a solid-state drive.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for generating RPA process scripts based on large models and screen recording scenarios, characterized in that, Includes the following steps: Record screen videos of user operations using RPA software, covering screen sources from different devices; Preprocess the recorded video, including noise reduction, frame rate adjustment, and filtering out invalid actions; A pre-trained video big data model is used to analyze videos and identify operational actions and target objects. In the video big data model analysis, a probabilistic graphical model is introduced for action recognition, mathematically expressed as: In the formula, Video sequence representing time t Action recognition and target The posterior probability; Indicates action and target Generate video sequences The probability of; For action and target The conditional probability represents the logical correlation of operations; For target object The prior probability is obtained from historical screen recording data statistics; Represents any action in the action set, and The same action type space includes click, input, and drag; Represents any target object in the target object set, and The same target object type space includes buttons, text boxes, and images; Based on the recognition results, action and target templates are generated to describe the operation behavior and target information; The template is mapped to an RPA script, which includes component arrangement, parameter settings and process logic; The generated scripts are executed on the RPA platform to automate operations.

2. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, The training method for the large video model includes: optimizing model parameters through supervised learning based on historical screen recording videos and corresponding RPA script templates, so that the model can accurately identify actions and targets; The loss function for training large video models is: In the formula, N is the number of training samples; To determine whether the i-th sample belongs to the true label, the true label includes the action and the type of the target object; Predict probabilities for the model; This is the model parameter vector; is the L2 regularization coefficient.

3. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, In the preprocessing step, an adaptive noise reduction algorithm is used to dynamically adjust the noise reduction parameters according to the video frame rate while preserving key operational details. The adaptive noise reduction algorithm uses the following filtering model: In the formula, The video frame at time t is after noise reduction; The time window weighting coefficient satisfies ; Let be the mean vector of the neighboring frames at time t; For dynamic noise variance; The L2 norm of the frame difference is represented.

4. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, The actions include mouse clicks, drags, inputs, and swipes, and the targets include web page elements, images, buttons, and input boxes; The logical expressions for the action and target templates are: In the formula, T is the template set; This indicates that the j-th action belongs to the action set. ; This represents the j-th target object, including coordinate and type attributes; These are conditional constraints.

5. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, In the steps of generating action and target templates, intermediate description pseudocode is used to abstractly represent the operation steps and logical flow. The pseudocode includes action type, target coordinates, operation parameters and time sequence.

6. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, In the step of mapping the template to an RPA script, the action and target information in the pseudocode are transformed into script components supported by the RPA platform through preset mapping rules. The mapping rules include an action-component mapping table and a parameter conversion algorithm. By dynamically adjusting the mapping rules through reinforcement learning, the state transition equation is: The reward function is defined as: In the formula, The mapping state at time t includes template features and script parameters; The mapping action at time t includes component selection and parameter adjustment; Let t be the reward value at time t; A binary variable represents the success of the execution; it is 1 for success and 0 for failure. For RPA script execution time; This refers to the time required for manual operation. Let KL divergence be the target probability distribution and the generated distribution; , , For the weighting coefficients, satisfying .

7. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, It also includes script optimization steps: during script execution, execution exceptions and efficiency data are recorded, and the video big model and mapping rules are updated through reinforcement learning to optimize the quality of subsequent script generation; In the script optimization step, an operation behavior knowledge base is established, and high-frequency operation patterns are extracted into reusable script component templates, supporting user-defined component library expansion.

8. An RPA process script generation system, used to automatically execute the RPA process script generation method based on a large model and screen recording scenario as described in any one of claims 1 to 7, characterized in that, The RPA process script generation system includes: The recording module is used to record screen videos of user operations via RPA software; The processing module is used to perform noise reduction, frame rate adjustment, and motion filtering on the recorded video. The analysis module contains pre-trained large video models for identifying actions and objects in the video; The template generation module is used to generate action and target templates based on the recognition results; The script conversion module is used to map templates to RPA scripts; An execution module is used to execute the generated RPA script on the RPA platform.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on an electronic device, causes the electronic device to execute the RPA process script generation method based on a large model and screen recording scenario as described in any one of claims 1 to 7.

Images