A computer-implemented method of interacting with a computer application through a
user interface is disclosed. The method includes determining if the computer application is in a state ready to be observed, reasoned, and interacted with; identifying an element in the computer application to interact with; and determining an action to use on the identified element and information required to execute the action; wherein identifying the element in the computer application to interact with includes extracting and storing one of more features of a reference element from a
reference document object model (DOM); and given a target DOM, finding the element in the target DOM that corresponding to the reference element.