SYSTEM AND METHOD FOR DETERMINING GUI INTERACTION INFORMATION OF END-USER DEVICE - Patent application

Abstract

A computer-implemented method for determining GUI interaction information of an end-user device is disclosed, the method includes: analyzing device state information using one or more GUI spy modes to estimate GUI interaction information of the one or more GUI spy modes, classifying the estimated GUI interaction information of the one or more GUI spy modes based on a reference model, and outputting GUI interaction information based on the classification result.

Description

[Technical field]

[0001] The present invention relates to a system and method for determining GUI interaction information of an end-user device. [Background technology]

[0002] A graphical user interface (GUI) of an end-user device allows an operator (i.e., a human operator) to use the end-user device to execute processes, which may include complex data processing and / or system control tasks. While the GUI allows an operator to quickly become familiar with executing new processes, further automation of the processes presents challenges because, from the perspective of an operator interacting with the GUI, the nature of the processes that may be executed is diverse rather than uniform.

[0003] Intelligent process capture and automation platforms such as "Blue Prism Capture" provide a system that represents an evolution of traditional process automation approaches by using a software agent to interact with an end-user device via an existing GUI to execute a given process. Such software agent generates appropriate input commands (to the end-user device) for the GUI of the end-user device to cause the end-user device to execute a given process, thereby enabling automation of said process. In order to be able to generate appropriate input commands for a given process to be executed automatically, the intelligent process capture and automation platform must first "capture" the process during a manual demonstration of the process by an operator. More specifically, for this purpose, the intelligent process capture and automation platform must analyze the state of the end-user device, i.e., the GUI state and various user inputs to the end-user device, during the course of the manual demonstration of the given process, and determine the sequence of GUI interactions that will be made to execute said process.

[0004] Such processes often involve operator interaction with multiple end-user device applications through multiple user inputs, each application having its own GUI elements for interaction with. An intelligent process capture and automation platform such as "Blue Prism Capture" determines the nature of GUI interactions made during manual demonstration of a process through a GUI "spying mode", i.e., a mode that monitors the end-user device's GUI and GUI element interactions. Such GUI spying modes may include low-level APIs, such as Win32, UIA, Browser (html) or Active Accessibility, which directly access GUI element attributes through a programmatic interface. GUI spying modes may also include modes that require post-processing of GUI screenshots, such as using an OCR model to analyze GUI pixel data to obtain visible text information of GUI elements, or using computer vision tools to analyze GUI pixel data to identify and extract GUI element attributes.

[0005] The problem here is that it is unclear which GUI spy mode will be most effective for a given GUI element in a given application. For example, UIA may work well for certain Windows applications, e.g., Excel, but not necessarily for applications such as Pycharm or SAP, in which case it may be necessary to fall back to a computer vision-based approach to determine GUI interaction information. If the GUI spy mode does not work, it may return an error or may return inaccurate results, e.g., the bounding box of the returned GUI element is too large for the interacted GUI element.

[0006] One approach that can be used to get around this issue is to have a human operator select the appropriate GUI spy mode and use an intelligent process capture and automation platform to perform and capture a manual demonstration of the process, which shifts the responsibility from the platform to the human operator, but at the expense of usability, as it takes time to accurately perform and capture a manual demonstration of the process. Summary of the Invention

[0007] It is therefore desirable to provide a method for estimating an optimal GUI spy mode or modes for determining GUI interactions, thereby enabling a more efficient and accurate determination of GUI interactions occurring during a manual demonstration of a process. [Means for solving the problem]

[0008] One embodiment of the present invention is a computer-implemented method for determining end-user device Graphical User Interface (GUI) interaction information, comprising: analyzing the device state information using one or more GUI spy modes to infer GUI interaction information for the one or more GUI spy modes; classifying estimated GUI interaction information of one or more GUI spy modes based on the reference model; and outputting GUI interaction information based on the classification result.

[0009] In embodiments disclosed herein, one or more GUI spy modes include an Application Programming Interface (API) native to the computer, and the estimated GUI interaction information is estimated by accessing the GUI interaction information from the API.

[0010] In further embodiments disclosed herein, one or more of the GUI spy modes include post-processing methods.

[0011] In further embodiments disclosed herein, the post-processing methods include computer vision tools.

[0012] In further embodiments disclosed herein, the reference model includes a heuristic model that is based on predefined rules.

[0013] In further embodiments disclosed herein, the reference model includes a multi-modal deep learning model trained on historical data.

[0014] In further embodiments disclosed herein, analyzing the device state information and classifying the corresponding inferred GUI interaction information is performed serially for multiple GUI spy modes.

[0015] In further embodiments disclosed herein, analyzing the device state information and classifying the corresponding inferred GUI interaction information is performed in parallel for multiple GUI spy modes.

[0016] In further embodiments disclosed herein, classifying the inferred GUI interaction information based on the reference model includes classifying the inferred GUI interaction information as either true or false.

[0017] In a further embodiment disclosed herein, the final GUI spy mode is a computer vision tool and the corresponding inferred GUI interaction information is classified as true.

[0018] In further embodiments disclosed herein, classifying the estimated GUI interaction information is terminated when a true classification is determined, and outputting the GUI interaction information based on the classification result includes outputting the estimated GUI interaction information classified as true.

[0019] In further embodiments disclosed herein, classifying the estimated GUI interaction information based on the reference model includes assigning a score to a subset of the estimated GUI interaction information based on the reference model.

[0020] In further embodiments disclosed herein, outputting GUI interaction information based on the classification results includes outputting the classified inferred GUI information with the highest score.

[0021] In further embodiments disclosed herein, outputting the GUI interaction information based on the classification results includes filtering and aggregating the classified inferred GUI interaction information based on the scores.

[0022] In a further embodiment disclosed herein, the filtering includes filtering out a subset of the categorized inferred GUI interaction information having a score below a predetermined threshold.

[0023] In further embodiments disclosed herein, outputting the GUI interaction information based on the classification result includes aggregating the classified inferred GUI interaction information having a score equal to or greater than a predetermined threshold.

[0024] In further embodiments disclosed herein, outputting GUI interaction information based on the classification results includes outputting a subset of the classified estimated GUI interaction information estimated using one or more of a plurality of GUI spy modes.

[0025] In further embodiments disclosed herein, analyzing device state information and inferring GUI interaction information using one or more GUI spy modes is performed for each instance of user interaction.

[0026] The present invention also provides, as a further embodiment, a non-transitory computer readable medium containing instructions for causing a computer to perform the method embodiments disclosed above. According to a further embodiment of the present invention there is provided a system comprising: a display for displaying the GUI of the end user device;

[0027] and at least one processor connected to a display, the at least one processor configured to execute instructions according to the method embodiments disclosed above. Hereinafter, an embodiment of the present invention will be described by way of example with reference to the accompanying drawings. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 illustrates a schematic diagram of an example of an end-user device system. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of a display for displaying a graphical user interface of an end user device. [Diagram 3] 2 is a flowchart of a method according to an embodiment of the present invention. [Figure 4] FIG. 2 illustrates a schematic diagram of information flow according to an embodiment of the present invention. [Figure 5A] 1 is a flow chart illustrating an example of a series implementation according to an embodiment of the present invention. [Figure 5B] 1 is a flow chart illustrating an example of a parallel implementation according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] The following description and drawings set forth certain illustrative embodiments of the invention.

[0030] The systems and methods described herein operate in the context of a platform for intelligent process capture and automation for end-user devices. The process capture component is responsible for capturing, i.e., recording, a manual demonstration of a given process performed on an end-user device by an operator. Process capture captures GUI interaction information, which includes information necessary to execute the captured process on the end-user device by a series of interactions with the GUI of the end-user device. This GUI interaction information is then used by software agents of the automation portion of the platform for the purpose of automating the captured process. This application relates to the process capture portion of the platform, and more particularly to techniques for accurately and efficiently determining said GUI interaction information during the capture of a manual demonstration of a given process.

[0031] 1 illustrates, in a schematic manner, an example of an end user device 100. One such end user device may be a personal computer, although it will be apparent that the end user device may be other devices such as a tablet, laptop computer, or other handheld device. The end user device 100 includes a computer 102, a display 104, and a user input device 106. The computer may include a storage device 108, a communication device 110, a memory device 112, and a processing device 114.

[0032] The processing device 114 may include memory (e.g., read only memory (ROM) and random access memory (RAM)) for storing processor-executable instructions, and one or more processors for executing the processor-executable instructions. The processing device 114 may also be in communication with the storage device 108. In embodiments of the invention in which the processing device 114 includes more than one processor, the processors may operate in a parallel or distributed manner. The processing device 114 may execute software associated with the operating system of the end user device or other elements of the end user device.

[0033] The communication device 110 may be a device that enables the end user device 100 to communicate with an interface device, such as the user input device 106. The communication device 110 may include one or more wired or wireless transceivers (e.g., WiFi, Bluetooth, and / or Ethernet communication devices) for communicating with other devices within the end user device 100. The communication device 110 may be configured to transmit data to and receive data from a remote processing server or processing device (e.g., a cloud server or processing device).

[0034] The memory device 112 may be any device (eg, a random access memory (RAM) device or a read-only memory (ROM) device) that stores data generated or received by components of the end-user device 100.

[0035] The storage device 108 may be any type of non-volatile data storage device, such as one or more of a hard disk drive, a magnetic disk, an optical disk, a ROM, etc. The storage device 108 may store an operating system that the processing device 114 executes in order for the end-user device 100 to function. The storage device 108 may also store one or more computer programs (or software or instructions or code).

[0036] Display 104 may be any device that outputs visual data representing a graphical user interface (GUI) of an end-user device. The GUI presented on display 104 may allow an operator to interact with the end-user device.

[0037] The user input devices 106 may enable an operator to interact with the end user device's GUI and / or other components within the end user device system 100 and may include a keyboard, a mouse, a trackpad, a trackball, and / or other directional input devices.

[0038] 2 is a schematic diagram of an example of a display 104 displaying a graphical user interface (GUI) 120 of an end-user device 100. The GUI 120 shows a number of applications represented by tabs 122 and windows 124, and includes an operating system. The various applications and operating systems include a number of application-specific GUI elements. The GUI 120 also includes GUI elements that correspond to a peripheral user input device 106, such as a mouse pointer 126. As will be described in more detail below, the GUI interaction information may include bounding boxes 128 that are used to identify GUI elements that can be interacted with to perform a given process. For purposes of illustration, the GUI 120 of FIG. 2 shows a number of bounding boxes 128 that each enclose an interactable GUI element.

[0039] 3 is a flow chart of a method according to an embodiment of the present invention. The steps of method 300 described below are exemplary. In some embodiments, method 300 may perform one or more additional operations not described herein and / or may perform without one or more operations described herein. Furthermore, the operations of method 300 are not limited to the order shown in FIG. 3 and described below.

[0040] Method 300 may be implemented in processing device 114 (e.g., one or more digital processors, analog processors, digital circuits designed to process information, analog circuits designed to process information, state machines, and / or other mechanisms for electronically processing information). Processing device 114 may include one or more devices that perform some or all of the operations of method 300 in response to instructions electronically stored in electronic storage device 108. Processing device 114 may include one or more devices configured to be specifically designed to perform one or more operations of the method through hardware, firmware, and / or software.

[0041] In step 310 shown in Fig. 3, one or more GUI spy modes 420 are used to analyze device state information 410 to infer GUI interaction information 430 of one or more GUI spy modes 420, as shown in Fig. 4, which shows a schematic of an information flow 400 according to an embodiment of the present invention. The device state information 410 may include user input information, e.g., user input information obtained via user input device 106, and GUI state information (e.g., information displayed on GUI 120). The device state information 410 may include information related to one or more of keystrokes, mouse actions, hover-over events, GUI screenshots, xy coordinates of a mouse cursor, along with other user input and / or GUI state variables.

[0042] The GUI interaction information provides information about how an operator interacts with the GUI 120 of the end-user device 100 to perform a given process. The GUI interaction information may include information about GUI element interactions. The GUI interaction information may include one or more coordinates of the bounding box 128 of the interacted GUI element, text information contained within the GUI element, the name of the interacted application, as well as GUI element attributes.

[0043] A GUI spy mode 420 is used to analyze the device state information 410 to estimate GUI interaction information for a given device state information 410, i.e., to obtain estimated GUI interaction information 430. The GUI spy mode 420 provides tools to "spy" or monitor the GUI 120 and GUI element interactions. The GUI spy mode 420 may include a mode for communicating with an application programming interface (API) that provides direct access to GUI element attributes through a program interface stored on the end user device 100. The API provides a software intermediary that allows applications to communicate with each other. Thus, GUI interaction information may be obtained from the API. The end user device 100 may include multiple different APIs.

[0044] The GUI Spy mode 420 may also include various post-processing methods. The post-processing GUI Spy mode may involve post-processing of GUI screenshots. The post-processing GUI Spy mode may use optical character recognition (OCR) on the GUI screenshots to obtain visible text information associated with one or more GUI elements. The post-processing GUI Spy mode may include computer vision tools. The computer vision tools identify GUI elements of the GUI 120 through image analysis techniques such as feature detection, allowing for identification of GUI elements based on known configurations (or appearances) of expected GUI elements. The computer vision tools may use machine learning or deep learning algorithms trained to identify specific GUI elements. The computer vision tools may use optical character recognition techniques to identify text components of identified GUI elements. The computer vision tools may use standard object detection techniques to identify GUI elements.

[0045] Referring again to FIG. 4, device state information 410 is provided as an input to GUI spy mode 420, and the output of GUI spy mode 420 is estimated GUI interaction information 430. If GUI spy mode 420 is unable to estimate GUI interaction information 430, GUI spy mode 420 may return an error (not shown in FIG. 4). Both device state information 410 and GUI interaction information 430, 450 may have a subset of information that matches, i.e., may include multiple variables. Device state information 410 may include device state information 410 acquired and analyzed at a particular time, e.g., for each instance of user interaction. Alternatively, device state information 410 may be acquired within a time window, where the time window may be a time window that starts and ends with a particular user input to the end user device, or may be a pre-determined periodic time window.

[0046] In some examples, the GUI state, i.e., the state displayed on GUI 120, may change before associated device state information 410 is obtained for performing a GUI interaction corresponding to the GUI state change. In some embodiments, device state information 410 (i.e., user input information and GUI state information) may be streamed to a memory device and assigned a corresponding timestamp. Accordingly, when inferring GUI interaction information 430 for a particular GUI interaction corresponding to a GUI state change, GUI spy mode 420 may access and analyze device state information 410 associated with a timestamp immediately preceding the timestamp of device state information 410 associated with the GUI state change.

[0047] Referring again to FIG. 3, in step 320 of the method 300, after classifying the estimated GUI interaction information 430 based on the reference model, in step 330, outputting GUI interaction information 450 based on the classification result. Referring again to FIG. 4, the device state information 410 and the estimated GUI interaction information 430 are provided as inputs to a classifier 440, which generates output GUI interaction information 450 based on the classification result. Classifying the estimated GUI interaction information 430 may include assigning a score to all or a subset of the estimated GUI interaction information 430 based on the reference model. The assigned score may be between 0 and 1 (inclusive), and the assigned score attempts to characterize the accuracy of the estimated GUI interaction information 430. In other embodiments, the classification of the estimated GUI interaction information 430 may include the use of a binary classifier that classifies a subset of the estimated GUI interaction information 430 as either true or false based on a score assigned to all or a subset of the estimated GUI interaction information 430. Once the estimated GUI interaction information 430 has been classified, step 330 of the method 300 of Figure 3 outputs GUI interaction information 450 based on the classification results. This output GUI interaction information 450 may then be used for intelligent process automation purposes.

[0048] The reference model underlying the classification of the estimated GUI interaction information 430 may include a heuristic model based on predefined rules. The predefined rules may define thresholds for the subset of the estimated GUI interaction information 430. For example, one such predefined rule may include checking that the area of ​​the bounding box 128 of a GUI element is below some reasonable threshold where such a GUI element is expected to be bounded. Another example of a predefined rule may include checking for vertical and horizontal lines for pixel values ​​within a given GUI element bounding box 128 provided as part of the estimated GUI interaction information 430 for a given GUI spy mode 420. If there are two vertical lines and two horizontal lines surrounding a mouse action (e.g., a click location), i.e., if the click location is surrounded by a rectangle, this suggests that there is a smaller GUI element bounding box 128 than the one provided by the GUI spy mode 420 and that there was some error in the estimation of the GUI interaction information 430 by the GUI spy mode 420. These lines can be found using classical computer vision methods. These methods may or may not include steps such as binarization, Hough transform, filtering, scaling, Canny edge detection, and component combination. Furthermore, other examples of the predetermined rules include exploiting the fact that GUI elements tend to be noisier (in terms of pixel value variance) than any larger bounding box (which tends to have more blank space). Based on this finding, the predetermined rules may include providing a threshold for the variance of pixel values ​​within the bounding box 128 of a given GUI element. Above a certain threshold, the bounding box 128 can be determined to be accurate. The above examples are not intended to limit the predetermined rules, and it will be clear to one skilled in the art that the above rules merely serve as possible examples of the predetermined rules, and more predetermined rules can be defined.A heuristic model based on predefined rules underlying the classification of the estimated GUI interaction information 430 may include a combination of one or more predefined rules, where the rules "vote" on whether a subset of the estimated GUI interaction information 430 (e.g., GUI element bounding boxes 128) is correct or not, and the model makes a final decision on whether the estimated GUI interaction information 430 is correct or not based on a majority vote.

[0049] In another embodiment, the reference model may include a deep learning model trained on historical data. The historical data may include a large collection of examples across all GUI spy modes used by the intelligent process capture and automation platform and many applications. Each example is composed of three elements: 1) a screenshot of the GUI 120 that displays the application or the desktop of the end user device, 2) inferred GUI interaction information 430, and 3) an assigned score. To train the deep learning model, the historical data must include examples of correct (positive) and incorrect (negative) GUI interaction information 430. Each correct example is assigned a score of 1.0 and each incorrect example is assigned a score of 0.0, which may be done manually or automatically. The historical data is divided into two different data sets: a training set and a test set. The deep learning model may include a residual neural network. The deep learning model may be trained by learning to classify the training set portion of the historical data. Training is performed by updating the weights of the deep learning model with an algorithm called backpropagation. The training procedure makes multiple passes over the training set. The deep learning model can be evaluated on a test set portion of the historical data. The evaluation process may evaluate the performance of the deep learning model on the test set according to a set of performance metrics including accuracy, precision, and recall.

[0050] The method 300 of FIG. 3 can be implemented in multiple ways depending on the operator's preferences and / or the computational requirements and constraints of the operating context. That is, the method of the present invention can include a serial implementation, a parallel implementation, or a combination thereof. As used herein, serial implementation and parallel implementation refer to a) analyzing the device state information 410 using multiple GUI spy modes 420 to estimate GUI interaction information 430 for the multiple GUI spy modes 420, and b) classifying the estimated GUI interactions 430 for the multiple GUI spy modes 420 based on a reference model. In other words, in a serial implementation, the estimation of GUI interaction information and subsequent classification of the estimated GUI interaction information 430 are performed sequentially for each of the multiple GUI spy modes 420. In a parallel implementation, the estimation of GUI interaction information is performed simultaneously for the multiple GUI spy modes 420, followed by classification of the estimated GUI interaction information 430 for the multiple GUI spy modes 420.

[0051] 5A is a flow chart illustrating an example of a serial implementation of a method 500 according to an embodiment of the present invention. When computing resources are low, an operator may select a serial implementation according to this exemplary embodiment. In step 510, the method is initialized with data in the form of device state information 410 and a collection of GUI spy modes 420 to be used in the serial manner.

[0052] In step 520, the device state information 410 is analyzed using a currently selected GUI spy mode 420 from the set of GUI spy modes to infer GUI interaction information 430 for the currently selected GUI spy mode 420. In a first iteration of the method of this embodiment, the currently selected GUI spy mode 420 is the first GUI spy mode 420 from the set of GUI spy modes. The set of GUI spy modes may be arranged arbitrarily or according to a predetermined rule, except for the last GUI spy mode 420 in the set.

[0053] In decision step 530, a check is made to determine whether the currently selected GUI spy mode 420 is the last GUI spy mode 420 in the set of GUI spy modes. In this embodiment of the method of the present invention, the last GUI spy mode 420 in the set of GUI spy modes is reserved for a computer vision tool. The computer vision tool serves as a fallback GUI spy mode 420 in case all other GUI spy modes in the set of GUI spy modes are unable to obtain accurate GUI interaction information. For a computer vision tool, the result of the GUI spy mode, i.e., the estimated GUI interaction information 430, is always classified as true. Thus, if the check determines that the currently selected GUI spy mode 420 is the last GUI spy mode 420 in the set of GUI spy modes, the method 500 proceeds to step 540, where the estimated GUI interaction information 430 estimated by the computer vision tool is output (as the GUI interaction information 450 based on the classification result) and the method 500 ends. If the check determines that the currently selected GUI spy mode 420 is not the last GUI spy mode 420 in the collection, ie, a computer vision tool, then the method 500 proceeds to step 550 .

[0054] At step 550, the inferred GUI interaction information 430 of the currently selected GUI spy mode 420 is classified based on the reference model, as described above with reference to FIGS.

[0055] At decision step 560, a check is performed to determine the nature of the classification of the estimated GUI interaction information 430 for the currently selected GUI spy mode 420. If the estimated GUI interaction information 430 is classified as true, the method 500 proceeds to step 570 where the estimated GUI interaction information 430 estimated by the currently selected GUI spy mode 420 is output (as GUI interaction information 450 based on the classification result) and the method 500 ends. If the check determines that the estimated GUI interaction information 430 is classified as false, the method proceeds to step 580 where the estimated GUI interaction information 430 of the currently selected GUI spy mode 420 is discarded and the selected GUI spy mode 420 in the set of GUI spy modes is shifted to the next GUI spy mode 420 in the set of GUI spy modes and the method 500 proceeds to step 520 for the next iteration of the method 500. The method 500 is repeated until completion, ie, upon the first "true" classification of the estimated GUI interaction information 430, and subsequent output of the estimated GUI interaction information 450 based on the classification results.

[0056] In the embodiment of method 500 illustrated in FIG. 5A , in a serial implementation of the method, not all GUI spy modes 420 in the set of GUI spy modes may be used because the method 500 may terminate before the last GUI spy mode 420 in the set of GUI spy modes is used; however, as described in more detail below, a serial implementation of the method of the present invention may also be implemented such that all GUI spy modes are used, regardless of whether a “true” classification of inferred GUI interaction information 430 occurs before the last GUI spy mode 420 in the set of GUI spy modes is reached.

[0057] 5B is a flow chart illustrating an example of a parallel implementation of a method 600 in accordance with an embodiment of the present invention. In step 610, the method is initialized with data in the form of device state information 410 and a collection of GUI spy modes to be employed in a parallel manner.

[0058] In step 620, the device state information 410 is analyzed using each of the GUI spy modes 420 of the set of GUI spy modes simultaneously to infer GUI interaction information 430 for each of the GUI spy modes 420 of the set of GUI spy modes.

[0059] In step 630, the inferred GUI interaction information 430 for each of the GUI spy modes 420 is simultaneously classified based on the reference model, as described above with reference to FIGS.

[0060] In step 640, GUI interaction information 450 based on the classification results for each of the GUI spy modes 420 in the set of GUI spy modes is output. The output can be determined based on the classification results in several ways, as described below. The following ways in which the output can be determined based on the classification results also apply to a serial implementation of the method of the present invention, in which all GUI spy modes 420 in the set of GUI spy modes are used, regardless of whether a "true" classification of the inferred GUI interaction information 430 occurred before using the last GUI spy mode 420 in the set of GUI spy modes.

[0061] As described above, classification of the estimated GUI interaction 430 information for a given GUI spy mode 420 may include assigning a score between 0 and 1 (inclusive) to all or a subset of the estimated GUI interaction 430 information based on the reference model. Classification-based GUI interaction information 450 may then be output based on this score. Estimated GUI interaction information 430 may also be classified as true or false for each of the GUI spy modes 420. GUI interaction information 450 may then be output based on an aggregation of estimated GUI interaction information 430 classified as true.

[0062] In one embodiment, for a given GUI spy mode 420, the inferred GUI interaction information 430 with the highest sum of assigned scores is used as the output GUI interaction information.

[0063] In another embodiment, the output GUI interaction information, i.e., GUI interaction information based on the classification results 450, may be composed of a subset of estimated GUI interaction information 430 from the multiple GUI spy modes, for example, based on a highest score for each subset of estimated GUI interaction information 430 across the entire set of estimated GUI interaction information 430 for the multiple GUI spy modes. For example, the classification results may suggest that an application name for a particular GUI interaction may be obtained more accurately from one of the API GUI spy modes, while an xy coordinate of a bounding box of an interacted GUI element may be obtained more accurately from a GUI spy mode including a computer vision tool. Accordingly, the output GUI interaction information 450 may be composed as a subset of estimated GUI interaction information 430 from one or more of the multiple GUI spy modes.

[0064] As another embodiment, the entire or a subset of estimated GUI interaction information 430 with scores below a certain threshold or classified as false may be filtered out from estimated GUI interaction information 430. Output GUI interaction information 450 may then be based on an aggregation of estimated GUI interaction information 430 that was not filtered, i.e., with assigned scores equal to or above a certain threshold or classified as true.

[0065] Aggregating estimated GUI interaction information 430 for multiple unfiltered GUI Spy modes may include, for example, obtaining the average xy coordinates of the mouse cursor 126, or the average coordinates of the bounding box 128 of a particular GUI element, or the mode of the application name, from the remaining estimated GUI interaction information 430.

[0066] As described above, the operational context of the present invention is a platform for intelligent process capture and automation, which involves the "capture" or recording of a manual demonstration of a given process. The method of the present invention can be implemented in several ways with respect to the manual demonstration of a process, depending on the operator's preferences and / or the computational requirements and constraints of the operational context.

[0067] In one embodiment, the method 300 of the present invention may be performed during a manual demonstration process, i.e., the method 300 may be performed for a given GUI interaction as an operator is interacting with an end user device to perform a manual demonstration of a process to be automated.

[0068] Alternatively, the method 300 of the present invention may be performed once the manual demonstration process is completed. An operator may employ this embodiment of the present invention if it is necessary to reserve computing resources during the manual demonstration of a process. In such an example, the relevant device state information is streamed to the memory device 112 and stored for later access and processing as described above.

[0069] A manual demonstration of the process may include successive operator interactions with multiple applications. In such an example, the method 300 according to an embodiment of the present invention may be performed for a first application after switching to a second application. For example, an operator may interact with Microsoft Excel and then Microsoft Edge. In such an example, the method 300 may be performed for GUI interactions with Microsoft Excel once the operator begins interacting with Microsoft Edge. In such an example, the associated device state information 410 may be streamed and stored in the memory device 112 for later access and processing as described above.

[0070] Furthermore, if the output GUI interaction information 450 for a given application is determined using the estimated GUI interactions 430 from a single GUI spy mode 420, then in subsequent uses of that application, an embodiment of the method 300 of the present invention may use only that GUI spy mode 420 to determine GUI interaction information. For example, a particular API may be determined to be suitable for estimating GUI interaction information 430 for a particular application. Thus, whenever that application is used during the manual demonstration process, an embodiment of the method 300 of the present invention may use only the single GUI spy mode 420 to analyze the device state information 410 and estimate the GUI interaction information 430 for that application. This estimated GUI interaction information 430 may be classified as true and then output.

[0071] Below is a numbered list of embodiments that may be claimed.

[0072] Embodiment 1 - A computer-implemented method for determining GUI interaction information of an end user device, comprising: analyzing the device state information using one or more GUI spy modes to infer GUI interaction information for the one or more GUI spy modes; classifying the estimated GUI interaction information of the one or more GUI spy modes based on the reference model; and outputting GUI interaction information based on the classification result.

[0073] Embodiment 2 - The method of embodiment 1, wherein the one or more GUI spy modes include an Application Programming Interface (API) native to the computer.

[0074] Embodiment 3 - The method of embodiment 2, wherein the estimated GUI interaction information is estimated by accessing the GUI interaction information from an API.

[0075] Embodiment 4 - A method according to any of the preceding embodiments, wherein one or more GUI spy modes include a post-processing method.

[0076]

[0026] Embodiment 5 - The method according to embodiment 4, wherein the post-processing method includes a computer vision tool.

[0077] Embodiment 6 - A method according to any of the preceding embodiments, wherein the estimated GUI interaction information is estimated by performing optical character recognition on the GUI.

[0078] Embodiment 7 - A method according to any of the preceding embodiments, wherein the reference model includes a heuristic model based on predetermined rules.

[0079] Embodiment 8 - The method according to any one of embodiments 1 to 6, wherein the reference model includes a multi-modal deep learning model trained on past data.

[0080] Embodiment 9 - A method according to any of the preceding embodiments, wherein using one or more GUI spy modes to analyze device state information and estimate GUI interaction information includes returning an error if the estimate cannot be determined.

[0081] Embodiment 10 - A method according to any of the preceding embodiments, wherein analyzing the device state information and classifying the corresponding estimated GUI interaction information are performed serially for multiple GUI spy modes.

[0082] Embodiment 11 - A method according to any one of embodiments 1 to 9, wherein analyzing the device state information and classifying the corresponding estimated GUI interaction information are performed in parallel for multiple GUI spy modes.

[0083] Embodiment 12 - A method according to any of the above-mentioned embodiments, wherein classifying the estimated GUI interaction information based on the reference model includes classifying the estimated GUI interaction information as either true or false.

[0084] Embodiment 13 - The method according to embodiment 12, wherein the final GUI spy mode is a computer vision tool and the corresponding estimated GUI interaction information is classified as true.

[0085]

[0036] Embodiment 14 - The method according to embodiment 12, wherein classifying the estimated GUI interaction information is terminated when a true classification is determined.

[0086] Embodiment 15 - In the method according to embodiment 12, outputting GUI interaction information based on the classification result includes outputting a first estimated GUI interaction information classified as true.

[0087] -Embodiment 16 - A method according to any one of embodiments 1 to 11, wherein classifying the estimated GUI interaction information based on the reference model includes assigning a score to a subset of the estimated GUI interaction information based on the reference model.

[0088] Embodiment 17-A method according to embodiment 16, wherein outputting GUI interaction information based on the classification result includes outputting classified estimated GUI interaction information having the highest score.

[0089]

[0046] In the method according to embodiment 18-embodiment 16, outputting GUI interaction information based on the classification result includes filtering and aggregating the classified estimated GUI interaction information based on a score.

[0090]

[0046] In the method of embodiment 19-embodiment 18, the filtering includes filtering out a subset of the classified estimated GUI interaction information having a score below a predetermined threshold.

[0091] In the method according to any one of embodiments 20 to 16, outputting GUI interaction information based on the classification result includes aggregating classified estimated GUI interaction information having a score equal to or greater than a predetermined threshold.

[0092] -Embodiment 21 - A method according to any one of the above-mentioned embodiments, wherein outputting GUI interaction information based on the classification results includes outputting a subset of classified estimated GUI interaction information estimated using one or more of a plurality of GUI spy modes.

[0093] -Embodiment 22 - A method according to any one of the above-mentioned embodiments, wherein the method is performed during a manual demonstration process.

[0094] -Embodiment 23 - A method as described in embodiment 22, which is executed when a manual demonstration process is completed.

[0095] Embodiment 24 - A method according to embodiment 22 or embodiment 23, wherein the manual demonstration includes successive interactions with multiple applications.

[0096] -Embodiment 25 - In the method according to embodiment 24, the method is performed for the first application after switching to the second application.

[0097] Embodiment 26 - A method according to any of the above-described embodiments, wherein if the output GUI interaction information of an application is estimated using a single GUI spy mode, the single GUI spy mode is also used to analyze device state information during subsequent use of the application.

[0098] -Embodiment 27 - A method according to any one of the above-mentioned embodiments, wherein classifying the estimated GUI interaction information based on the reference model includes passing the device state information and the estimated GUI interaction information to a classifier and outputting classified GUI interaction information.

[0099] Embodiment 28 - A method according to any one of the above-mentioned embodiments, wherein the device state information includes a matching subset of the device state information.

[0100] In the method according to any one of embodiments 29 to 7, the predetermined rule defines a threshold value for the estimated GUI interaction information.

[0101] In the method according to any one of embodiments 30 to 8, the multi-modal deep learning model trained on the historical data includes a residual neural network.

[0102] -Embodiment 31 - A method according to any one of the above-mentioned embodiments, wherein analyzing device state information and inferring GUI interaction information using one or more GUI spy modes is performed for each instance of user interaction.

[0103] -Embodiment 32 - A method according to any one of embodiments 1 to 30, wherein analyzing device state information and estimating GUI interaction information using one or more GUI spy modes is performed at regular intervals.

[0104] Embodiment 33 - A method according to any one of the above-mentioned embodiments, wherein the device status information includes device status information obtained at a particular time.

[0105] Embodiment 34 - A method according to any one of embodiments 1 to 32, wherein the device status information includes device status information obtained within a time window.

[0106] Embodiment 35 - A method according to any one of the preceding embodiments, wherein the device state information is stored in memory together with a corresponding timestamp.

[0107] -Embodiment 36 - In the method described in embodiment 35, analyzing device state information and inferring GUI interaction information using one or more GUI spy modes includes analyzing device state information stored in memory using one or more GUI spy modes.

[0108] Embodiment 37 - A method according to any one of the above-mentioned embodiments, wherein the device state information includes user input information and GUI state information.

[0109] Embodiment 38 - A method according to any one of the above-mentioned embodiments, wherein the GUI interaction information includes GUI element interaction information.

[0110] -Embodiment 39 - A non-transitory computer-readable medium comprising instructions for causing a computer to execute the method according to any one of embodiments 1 to 38.

[0111] -Embodiment 40- a display for displaying the GUI of the end user device; A system comprising at least one processor connected to a display, the at least one processor being configured to execute instructions based on a method according to any one of embodiments 1 to 38.

Claims

1. A method implemented by a computer for determining graphical user interface (GUI) interaction information of an end-user device, comprising: analyzing device state information using one or more GUI spy modes and estimating GUI interaction information of the one or more GUI spy modes; classifying the estimated GUI interaction information of the one or more GUI spy modes based on a reference model; outputting GUI interaction information based on the classification result of the classification; A method including the above steps.

2. The method according to claim 1, wherein the one or more GUI spy modes include an application programming interface (API) native to a computer, and the estimated GUI interaction information is estimated by accessing GUI interaction information from the API.

3. In the method according to claim 1, the one or more GUI spy modes include a post-processing method.

4. In the method according to claim 3, the post-processing method includes computer vision tools.

5. In the method according to claim 1, the reference model includes a discovery model based on predetermined rules.

6. In the method according to claim 1, the reference model includes a multimodal deep learning model trained with past data.

7. In the method according to claim 1, analyzing the device state information and classifying the corresponding estimated GUI interaction information are performed serially for a plurality of GUI spy modes.

8. In the method according to claim 1, analyzing the device state information and classifying the corresponding estimated GUI interaction information are performed in parallel for a plurality of GUI spy modes.

9. In the method according to claim 1, classifying the estimated GUI interaction information based on the reference model includes classifying the estimated GUI interaction information as either true or false.

10. In the method according to claim 9, the last GUI spy mode is a computer vision tool, and the corresponding estimated GUI interaction information is classified as true.

11. In the method according to claim 9, classifying the estimated GUI interaction information ends when the true classification is determined, and outputting the GUI interaction information based on the classification result includes outputting the estimated GUI interaction information classified as true, the method.

12. In the method according to claim 1, classifying the estimated GUI interaction information based on a reference model includes assigning scores to subsets of the estimated GUI interaction information based on the reference model, the method.

13. In the method according to claim 12, outputting the GUI interaction information based on the classification result includes outputting the classified estimated GUI interaction information with the highest score, the method.

14. In the method according to claim 12, outputting the GUI interaction information based on the classification result includes filtering and aggregating the classified estimated GUI interaction information based on the score, the method.

15. In the method according to claim 14, the filtering includes excluding a subset of the classified estimated GUI interaction information with a score less than a predetermined threshold, the method.

16. In the method according to claim 12, outputting the GUI interaction information based on the classification result includes aggregating the classified estimated GUI interaction information with a score equal to or higher than a predetermined threshold, the method.

17. In the method according to claim 1, outputting the GUI interaction information based on the classification result includes outputting a subset of the classified estimated GUI interaction information estimated using one or more of the plurality of GUI spy modes, the method.

18. In the method according to claim 1, analyzing device state information using the one or more GUI spy modes and estimating the GUI interaction information is performed for each instance of user interaction, the method.

19. A non-transitory computer-readable medium including instructions for causing a computer to execute the method according to any one of claims 1 to 18.

20. A display for displaying a GUI of an end-user device, At least one processor connected to the display, the at least one processor being configured to execute instructions based on the method according to any one of claims 1 to 18, a system comprising the same.

Images