Robotic process automation

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for creating user interface control images for robotic automation. One of the methods includes detecting, during a training process, a control selection event; retrieving a first image and a second image; determining a difference image that represents a difference between the first image and the secondimage; determining, using the difference image, a plurality of colors for pixels included in the difference image that are likely located outside of a control location for the control; determining oneor more shapes depicted in the difference image that each have a color other than all of the colors in the plurality of colors; selecting, from the one or more shapes and using an event location, a particular shape as likely representing the control; and storing a cropped image of the control extracted from one of the two or more images.

Description

technical field

[0001] The present disclosure relates generally to robotic process automation. Background technique

[0002] Manual processing performed on a computer can be repetitive and time-consuming. For example, a user may send an email in response to a web form submission, and for each web form submission, manual interaction with the computer may be required to click a button to open a window for composing a new email, click a button to send the web form submission to the Copy the information into the window, click the button to write the text of the email body, and click the button to send the email. Contents of the invention

[0003] In some implementations, the robotic training system may use computer vision and user interface component analysis to identify user interface controls, such as icons, selected during a process automated by the robotic training system. The robot training system can analyze differences between the plurality of images of the user inter...

Images