RPA process script generation method and system based on large model and screen recording scene, and storage medium
By generating RPA process scripts based on screen recording scenarios using large models, the problem of long time consumption and high cost in traditional RPA script generation is solved, realizing automated and accurate script generation to adapt to complex business needs.
Patent Information
- Application Number
- CN202510916975.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-03
- Publication Date
- 2025-11-04
AI Technical Summary
Traditional RPA script generation relies on professional programmers, which is time-consuming, costly, and difficult to adapt to rapidly changing business needs. Existing technologies struggle to achieve efficient and accurate automated script generation.
An RPA process script generation method based on large models and screen recording scenarios is adopted. By recording user operation videos, a pre-trained video large model is used to identify operation actions and targets, generate action and target templates, and map them into RPA scripts. Reinforcement learning is combined to optimize the mapping rules to achieve automated script generation.
It enables efficient and accurate generation of RPA scripts without manual intervention, can handle complex processes, adapt to various business scenarios, reduce generation costs and time, and improve the accuracy and flexibility of generated scripts.
Smart Images

Figure CN120892122A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an RPA process script generation method, system, and storage medium based on large models and screen recording scenarios. Background Technology
[0002] In this era of information overload, screen recording, as an intuitive and easy-to-understand method of information dissemination, is widely used in education, training, technical support, and software demonstrations. Especially in the wave of digital transformation, enterprises urgently need to improve operational efficiency and reduce labor costs through automation. RPA technology has emerged to meet this need, enabling software robots to simulate human operations and automate highly repetitive and well-defined business processes. However, writing traditional RPA scripts relies on the programming skills of professionals, which is not only time-consuming and costly but also difficult to adapt to rapidly changing business needs.
[0003] Furthermore, with the rapid development of video processing technology and artificial intelligence, the ability to analyze, recognize, and understand video content has significantly improved. This makes it possible to automatically extract key information from screen recordings and convert it into executable scripts. In particular, the application of deep learning technology to large video models enables models to learn and understand complex scenes, actions, and processes in videos, laying a solid foundation for the automated conversion of screen recordings into RPA scripts. Summary of the Invention
[0004] Purpose of the invention: To propose an RPA process script generation method, system and storage medium based on large models and screen recording scenarios, aiming to fundamentally solve the problems of low efficiency, error susceptibility and difficulty in adapting to complex processes in the traditional RPA script generation process.
[0005] In a first aspect, the present invention proposes an RPA process script generation method based on a large model and screen recording scenario, comprising the following steps:
[0006] Record screen videos of user operations using RPA software, covering screen sources from different devices;
[0007] Preprocess the recorded video, including noise reduction, frame rate adjustment, and filtering out invalid actions;
[0008] Analyze videos using pre-trained large video models to identify operational actions and target objects;
[0009] Based on the recognition results, action and target templates are generated to describe the operation behavior and target information;
[0010] The template is mapped to an RPA script, which includes component arrangement, parameter settings and process logic;
[0011] The generated script is executed on the RPA platform to realize operation automation.
[0012] In further embodiments of the first aspect, the method for training the video large model comprises: based on historical recording videos and corresponding RPA script templates, optimizing model parameters through supervised learning to enable the model to accurately identify actions and target objects;
[0013] The loss function for training the video large model is:
[0014]
[0015] In the formula, N is the number of training samples; y i is the true label of the i-th sample, including the action and the target object type; is the model prediction probability; θ is the model parameter vector; λ is the L2 regularization coefficient.
[0016] In further embodiments of the first aspect, in the preprocessing step, an adaptive noise reduction algorithm is used to dynamically adjust the noise reduction parameters according to the video frame rate, and key operation details are preserved.
[0017] The adaptive noise reduction algorithm uses the following filter model:
[0018]
[0019] In the formula, V t ′ is the video frame at time t after noise reduction; ω i is the time window weight coefficient, satisfying is the mean vector of the neighborhood frames at time t; is the dynamic noise variance; ||*|| represents the L2 norm of the frame difference.
[0020] In further embodiments of the first aspect, the video large model analyzes the video, and a probabilistic graph model is introduced for action recognition, which is mathematically expressed as:
[0021]
[0022] In the formula, represents the posterior probability of identifying action A and target object Q t at time t based on the previous t video frames t ; represents the probability of action A t and target object Q t producing video sequence ; P(A t , Q t ) is the action A tand target Q t , characterizing the operation logic relevance; P(Q t ) is the prior probability of target Q t , obtained by statistical history recording data; A ′ represents any action in the action set, which belongs to the action type space with A t , including clicking, inputting, dragging; O ′ represents any target in the target set, which belongs to the target type space with Q t , including buttons, text boxes, pictures.
[0023] In further embodiments of the first aspect, the actions include mouse clicks, dragging, inputting and sliding, and the targets include web elements, pictures, buttons and input boxes.
[0024] The logical expression of the action and target template is:
[0025] T={(A h ,Q j ,C j )|j=1,2,…,M}
[0026] In the formula, T is the template set; A j represents the jth action, which belongs to the action set A={click, input, drag, …}; Q j represents the jth target, which contains coordinate attributes and type attributes; C j is the conditional constraint.
[0027] In further embodiments of the first aspect, in the step of generating action and target templates, intermediate description pseudo code is used to abstractly represent the operation steps and logic flow, and the pseudo code contains action type, target coordinate, operation parameter and time sequence.
[0028] In further embodiments of the first aspect, in the step of mapping the template into an RPA script, the action and target information in the pseudo code is converted into script components supported by the RPA platform through a preset mapping rule, and the mapping rule includes an action-component mapping table and a parameter conversion algorithm.
[0029] The mapping rule is dynamically adjusted through reinforcement learning, and the state transition equation is:
[0030] s t+1 =f(s t ,a t ,r t )
[0031] The reward function is defined as:
[0032]
[0033] In the formula, s t The mapping state at time t includes template features and script parameters; a t The mapping action at time t includes component selection and parameter adjustment; r t Let δ be the reward value at time t; exec A binary variable represents the success of the execution, with a value of 1 for success and 0 for failure; T script T represents the execution time of the RPA script. human For manual operation time; KL(P) target ||P generated ) represents the KL divergence between the target probability distribution and the generated distribution; α, β, and γ are weighting coefficients, satisfying α + β + γ = 1.
[0034] In a further embodiment of the first aspect, the RPA process script generation method further includes a script optimization step: during script execution, recording execution exceptions and efficiency data, updating the video big model and mapping rules through reinforcement learning, and optimizing the quality of subsequent script generation;
[0035] In the script optimization step, an operation behavior knowledge base is established, and high-frequency operation patterns are extracted into reusable script component templates, supporting user-defined component library expansion.
[0036] A second aspect of the present invention provides an RPA process script generation system for automatically executing the RPA process script generation method based on a large model and screen recording scenario described in the first aspect. The RPA process script generation system includes:
[0037] The recording module is used to record screen videos of user operations via RPA software;
[0038] The processing module is used to perform noise reduction, frame rate adjustment, and motion filtering on the recorded video.
[0039] The analysis module contains pre-trained large video models for recognizing actions and objects in videos;
[0040] The template generation module is used to generate action and target templates based on the recognition results;
[0041] The script conversion module is used to map templates to RPA scripts;
[0042] An execution module is used to execute the generated RPA script on the RPA platform.
[0043] In a third aspect, the present application provides a computer-readable storage medium having stored therein at least one executable instruction, which, when executed on an electronic device, causes the electronic device to perform the method for generating an RPA process script based on a large model and a screen recording scene as described in the first aspect.
[0044] Compared with the prior art, the present application has the following beneficial effects:
[0045] (1) Automatically generate RPA scripts: through deep learning and video processing technology, automatically identify the operation steps, interface elements and interaction logic in the screen recording video, and generate the corresponding RPA script without human intervention, greatly improving the script generation efficiency.
[0046] (2) Improve accuracy: use the powerful feature extraction and pattern recognition capabilities of the large model to accurately capture every key detail in the screen recording, ensuring that the generated RPA script is accurate and error-free, reducing errors caused by human factors.
[0047] (3) Adapt to complex processes: the system can handle complex operation processes containing multiple steps, conditional branches and loops, flexibly cope with various business scenarios, and meet the automation needs of different enterprises. BRIEF DESCRIPTION OF DRAWINGS
[0048] Figure 1 Flowchart of the RPA process script generation method in the embodiment.
[0049] Figure 2 Detailed flowchart of video to RPA process automation. DETAILED DESCRIPTION
[0050] In the following description, a large number of specific details are given to provide a more thorough understanding of the present application. However, it is obvious to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, some technical features known in the art are not described in order not to obscure the present application.
[0051] Before describing the following embodiments, first explain the terms appearing in the following.
[0052] LLM (Large Language Model): refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks such as text classification, question answering, dialogue, etc., and are an important way to artificial intelligence.
[0053] RPA(Robotic Process Automation, machine process automation): an application that provides another way to automate the end-user manual operation process by simulating the end-user manual operation mode on the computer.
[0054] Action and target template: an intermediate description pseudo code for describing actions in a video, which can abstractly represent the operation steps and logical flow in the video.
[0055] Target: a specific object or operation target identified and extracted in a video, such as web elements, pictures, buttons, input boxes, etc.
[0056] Action: a mouse action identified in a video, such as clicking, double-clicking, dragging, etc.
[0057] RPA script: a robot process automation script, used to automate the execution of tasks.
[0058] Example 1:
[0059] This embodiment discloses an RPA process script generation method based on large models and screen recording scenarios. The method records the to-be-generated automation scenario through screen recording, annotates the screen recording using the large model's understanding of pictures, and finally converts it into a specific RPA process script using semantic understanding capabilities, thereby realizing the rapid understanding and arrangement of processes in a custom scenario.
[0060] Figure 1 For the RPA process script generation process based on large models and screen recording scenarios, the present invention is implemented by the following steps:
[0061] Step S1: RPA video recording.
[0062] Record the user's operation video through RPA screen recording. Support recording video from different sources (such as computer screen, mobile device screen), ensuring that a wide range of user scenarios and device types can be covered.
[0063] Step S2: Video processing.
[0064] Perform preprocessing operations such as filtering, noise reduction, and video frame on the RPA screen recording video to improve the accuracy and efficiency of subsequent video large model analysis.
[0065] Step S3: Video large model analysis.
[0066] Use a pre-trained video large model to perform in-depth analysis on the preprocessed video, identifying operation action behaviors (such as clicking, dragging, inputting, etc.) and targets (such as buttons, text boxes, pictures, etc.) in the video.
[0067] Step S4: Generate action and target object templates.
[0068] According to the recognition result of the video large model, the corresponding action and target object templates are automatically generated, and each operation behavior and its target object information is described in detail.
[0069] Step S5: RPA script generation.
[0070] The action and target object template information is converted and mapped into an RPA script through a program, including the arrangement of script components, parameter settings, and process logic arrangement, to ensure that the RPA script can accurately simulate the operation behavior in the screen recording video.
[0071] Step S6: RPA script execution.
[0072] The script generated by mapping is executed on the RPA platform to realize the automatic simulation and execution of the operation behavior in the RPA screen recording video.
[0073] Embodiment 2:
[0074] Based on the foregoing embodiment 1, this embodiment discloses some details of the steps.
[0075] Step S1: RPA video recording.
[0076] The user's operation video is recorded through RPA screen recording. It supports recording videos from different sources (such as computer screen, mobile device screen), ensuring that it can cover a wide range of user scenarios and device types.
[0077] Step S2: Video processing.
[0078] The RPA screen recording video is preprocessed, such as filtering, noise reduction, and video frame, to improve the accuracy and efficiency of subsequent video large model analysis.
[0079] In the preprocessing step, an adaptive noise reduction algorithm is used to dynamically adjust the noise reduction parameters according to the video frame rate, preserving key operation details.
[0080] The adaptive noise reduction algorithm uses the following filter model, as shown in equation (1):
[0081]
[0082] In the formula, V t ′ is the t-th video frame after noise reduction; ω i is the time window weight coefficient, satisfying is the mean vector of the neighborhood frame at t; σ is the dynamic noise variance; ||*|| represents the L2 norm of the frame difference.
[0083] Step S3: Video large model analysis.
[0084] The pre-processed video is analyzed in depth using a pre-trained video large model to identify the operation action behaviors (such as clicking, dragging, inputting, etc.) and target objects (such as buttons, text boxes, pictures, etc.) in the video.
[0085] The logical expression of the action and target object template is shown in equation (2):
[0086] T={(A j ,Q j ,C j )|j=1,2,…,M} (2)
[0087] In the equation, T is the template set; A j represents the jth action, which belongs to the action set A={click, input, drag, …}; Q j represents the jth target object, which contains coordinate attributes and type attributes; C j is the conditional constraint.
[0088] The training method of the video large model includes: based on historical screen recording videos and corresponding RPA script templates, the model parameters are optimized through supervised learning to make the model accurately identify actions and target objects;
[0089] The loss function of the video large model training is shown in equation (3):
[0090]
[0091] In the equation, N is the number of training samples; y i is the true label of the ith sample, which contains action and target object type; p is the model prediction probability; θ is the model parameter vector; λ is the L2 regularization coefficient.
[0092] Step S4: Generate action and target object templates.
[0093] According to the recognition results of the video large model, the corresponding action and target object templates are automatically generated, and each operation behavior and its target object information is described in detail.
[0094] The video large model analyzes the video, introduces a probability graph model for action recognition, and the mathematical expression is shown in equation (4):
[0095]
[0096] In the equation, represents the posterior probability of identifying action A t and target object Q t based on the first t frames of video at time t; Action A t and target Q t Generate video sequence ; P(A t , Q t ) is the conditional probability of action A t and target Q t , representing the operation logic association; P(Q t ) is the prior probability of target Q t , obtained by statistical analysis of historical screen recording data; A ′ represents any action in the action set, which belongs to the same action type space as A t , including clicking, inputting, and dragging; O ′ represents any target in the target set, which belongs to the same target type space as Q t , including buttons, text boxes, and pictures.
[0097] Step S5: RPA script generation.
[0098] The action and target template information are converted and mapped into RPA scripts by a program, including the arrangement of script components, parameter settings, and flow logic arrangement, to ensure that the RPA script can accurately simulate the operation behavior in the screen recording video.
[0099] Step S6: RPA script execution.
[0100] The script generated by mapping is executed on the RPA platform to realize the automatic simulation and execution of the operation behavior in the RPA screen recording video.
[0101] Example 3:
[0102] The following operations are all completed on the Chrome browser, and the specific operations are as follows: open the Chrome browser, enter the website address in the input address bar, then press Enter to access the webpage, then click the login in the upper right corner of the homepage, enter the account and password, then click to read and accept the Baidu user agreement, and finally click to log in. Take this operation as an example. The video analysis large model trained and the result verification are used to generate action and target templates, and the RPA script is converted and mapped by a program. The specific implementation steps are as follows:
[0103] Step S1: Install RPA software and record video: the user installs RPA software on his computer, uses the video recording function to record the process of opening the browser, accessing Baidu, clicking to log in and entering the account and password, checking the user agreement, and finally clicking to log in.
[0104] Step S2: Video processing of step 1: The recorded video is pre-processed by the RPA software to remove invalid actions, noise, and video frames, and finally output a high-fidelity video.
[0105] Step S3: The video output by step S2 is sent to a video analysis large model, which can identify each operation step of the user in the video and the corresponding target object, and then output an action and target object template.
[0106] Step S4: Based on the template content output by step S3, the action and target object template are converted into RPA component scripts through an RPA conversion program.
[0107] Step S4-1: Open the browser
[0108] Step S4-2: Enter the website address
[0109] Step S4-3: Click the "Login" button
[0110] Step S4-4: Enter the account
[0111] Step S4-5: Enter the password
[0112] Step S4-6: Check the user agreement
[0113] Step S4-7: Click the "Login" button
[0114] Step S5: Script execution, based on the generated RPA script, run in the RPA execution machine to realize the automatic Baidu login operation.
[0115] The logic behind the method disclosed in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs.
[0116] When the computer instructions or computer program are loaded or executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.
[0117] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the processes described in the flowcharts and / or block diagrams. Figure 1 The processes or functions described in the flowcharts and / or block diagrams Figure 1 The devices that implement the functions specified in one or more flows and / or blocks.
[0118] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction devices that implement the processes described in the flowcharts and / or block diagrams. Figure 1 The processes or functions described in the flowcharts and / or block diagrams Figure 1 The devices that implement the functions specified in one or more flows and / or blocks.
[0119] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the processes described in the flowcharts and / or block diagrams. Figure 1 The processes or functions described in the flowcharts and / or block diagrams Figure 1 The devices that implement the functions specified in one or more flows and / or blocks.
[0120] While the application has been described and illustrated with reference to specific preferred embodiments, it is not intended that it be limited to these particulars. Various changes in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.
Claims
1. A method for generating RPA process scripts based on large models and screen recording scenarios, characterized in that, Includes the following steps: Record screen videos of user operations using RPA software, covering screen sources from different devices; Preprocess the recorded video, including noise reduction, frame rate adjustment, and filtering out invalid actions; Analyze videos using pre-trained large video models to identify operational actions and target objects; Based on the recognition results, action and target templates are generated to describe the operation behavior and target information; The template is mapped to an RPA script, which includes component arrangement, parameter settings and process logic; The generated scripts are executed on the RPA platform to automate operations.
2. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, The training method for the large video model includes: optimizing model parameters through supervised learning based on historical screen recording videos and corresponding RPA script templates, so that the model can accurately identify actions and targets; The loss function for training large video models is: In the formula, N is the number of training samples; y i The true label for the i-th sample includes the action and the type of the target object; θ represents the model prediction probability; θ is the model parameter vector; and λ is the L2 regularization coefficient.
3. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, In the preprocessing step, an adaptive noise reduction algorithm is used to dynamically adjust the noise reduction parameters according to the video frame rate while preserving key operational details. The adaptive noise reduction algorithm uses the following filtering model: In the formula, V t ′ ω is the denoised video frame at time t; i The time window weighting coefficient satisfies Let be the mean vector of the neighboring frames at time t; represents the dynamic noise variance; ||*|| represents the L2 norm of the frame difference.
4. The RPA process script generation method based on large models and screen recording scenarios according to claim 1, characterized in that, In the video big data model analysis, a probabilistic graphical model is introduced for action recognition, mathematically expressed as: In the formula, This indicates that at time t, based on the previous t frames of video... Action A was identified. t and target object Q t The posterior probability; Indicates action A t and target object Q t Generate video sequences The probability of P(A) t Q t Action A t and target object Q t The conditional probability of P(Q) represents the logical correlation of operations; t ) is the target object Q t The prior probability is obtained from statistical analysis of historical screen recording data; A ′ Represents any action in the action set, and is related to A. t Belonging to the same action type space, including click, input, and drag; O ′ Represents any target object in the target object set, and Q t The same target object type space includes buttons, text boxes, and images.
5. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, The actions include mouse clicks, drags, inputs, and swipes, and the targets include web page elements, images, buttons, and input boxes; The logical expressions for the action and target templates are: T={(A j ,Q j ,C j )|j=1,2,…,M} In the formula, T is the template set; A j Q represents the j-th action, belonging to the action set A = {click, input, drag, ...}; j Represents the j-th target object, including coordinate and type attributes; C j These are conditional constraints.
6. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, In the steps of generating action and target templates, intermediate description pseudocode is used to abstractly represent the operation steps and logical flow. The pseudocode includes action type, target coordinates, operation parameters and time sequence.
7. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, In the step of mapping the template to an RPA script, the action and target information in the pseudocode are converted into script components supported by the RPA platform through preset mapping rules. The mapping rules include an action-component mapping table and a parameter conversion algorithm. By dynamically adjusting the mapping rules through reinforcement learning, the state transition equation is: s t+1 =f(s t ,a t ,r t ) The reward function is defined as: In the formula, s t The mapping state at time t includes template features and script parameters; a t The mapping action at time t includes component selection and parameter adjustment; r t Let δ be the reward value at time t; exec A binary variable represents the success of the execution, with a value of 1 for success and 0 for failure; T script For RPA script execution time; T human For manual operation time; KL(P) target ||P generated ) represents the KL divergence between the target probability distribution and the generated distribution; α, β, and γ are weighting coefficients, satisfying α + β + γ = 1.
8. The RPA process script generation method based on large model and screen recording scenario according to claim 1, characterized in that, It also includes script optimization steps: during script execution, execution exceptions and efficiency data are recorded, and the video big model and mapping rules are updated through reinforcement learning to optimize the quality of subsequent script generation; In the script optimization step, an operation behavior knowledge base is established, and high-frequency operation patterns are extracted into reusable script component templates, supporting user-defined component library expansion.
9. An RPA process script generation system, used to automatically execute the RPA process script generation method based on a large model and screen recording scenario as described in any one of claims 1 to 8, characterized in that, The RPA process script generation system includes: The recording module is used to record screen videos of user operations via RPA software; The processing module is used to perform noise reduction, frame rate adjustment, and motion filtering on the recorded video. The analysis module contains pre-trained large video models for identifying actions and objects in the video; The template generation module is used to generate action and target templates based on the recognition results; The script conversion module is used to map templates to RPA scripts; An execution module is used to execute the generated RPA script on the RPA platform.
10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on an electronic device, causes the electronic device to execute the RPA process script generation method based on a large model and screen recording scenario as described in any one of claims 1 to 8.
Citation Information
Patent Citations
RPA process automatic generation method and system based on video analysis
CN113255614A
RPA process video key frame extraction and element positioning method
CN115033740A
RPA and OCR-based demand collection method and device
CN117291516A
Business process automation method and related equipment
CN119829174A
Robotics Process Automation Automatic Enhancement System
US20230037297A1
