Method and system for re-identification in a user interface

Abstract

A method (100) for performing description mining of a concept is provided, comprising the steps of: obtaining (S110) a plurality of basis descriptions, each of which is associated with a user interface, UI, element corresponding to a type of UI element represented by the concept; generating (S120) at least one of a local description of the concept or a global description of the concept based on the plurality of basis descriptions, the local description of the concept comprising description components specific to a respective UI, and the global description of the concept comprising description components not specific to a UI; storing (S130) at least one of the generated local description and the global description in association with a concept identifier corresponding to the concept.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for re-identification in a user interface. More specifically, the present disclosure relates to a system and a computer-implemented method for performing concept description mining and performing a re-identification check of user interface elements in a current user interface.

Background Art

[0002] Computer processes guided by humans are ubiquitous across many technical and effort areas. Modern graphical user interfaces (GUIs) have proven to be very valuable in enabling human operators to use computer systems to perform often complex data processing and / or system control tasks. However, GUIs often enable a human operator to quickly become accustomed to performing new tasks, but there is a high wall to any further automation of the tasks.

[0003] Conventional workflow automation typically takes tasks performed by an operator using a GUI and automates them so that a computer system can perform the same tasks without significant redesign of the underlying software used to perform the tasks. Initially, this required exposing the application programming interface (API) of the software so that a script could be manually manipulated to perform the required functions of the software to perform the required tasks.

[0004] Robotic Process Automation (RPA) is a new form of business process automation technology that utilizes software robots or artificial intelligence (AI) workers. In traditional workflow automation tools, software developers use internal application program interfaces to generate a list of actions for automating tasks. In contrast, RPA systems develop action lists by having users watch tasks being performed in the application's graphical user interface and then execute automation by directly repeating those tasks on the graphical user interface. The RPA robot can then generate the appropriate input commands for the GUI to cause a given process to be executed by the computer system. This enables the automation of processes and turns manned processes into unmanned processes. The advantages of such an approach are numerous, including greater scalability that allows multiple RPA robots to execute the same tasks across multiple computer systems, and strong reproducibility such that the possibility of human error in a given process is reduced or eliminated.

Summary of the Invention

[0005] Existing approaches for automation typically rely on extracting unique identifiers for user interface (UI) elements and building automation based on these identifiers. Examples of such identifiers for web-based applications are Extensible Markup Language Path Language (XPath) expressions and Cascading Style Sheet (CSS) selectors, which can also be combined with various properties of the UI elements. The unique identifiers are usually manually defined by RPA developers during application modeling or workflow development. However, these unique identifiers can be easily affected by changes to the UI. For example, an XPath expression can be invalidated by the addition of a new ancestor (e.g., a container not visible to the user) or a change in the identifier of an existing ancestor element.

[0006] Some identifiers of UI elements are more robust to changes in the UI, and domain experts may use such identifiers to identify UI elements. While doing so, experts can use trial and error in addition to domain knowledge. The methods and systems proposed herein are inspired by this. The methods and systems are based on the idea of maintaining the representation of the UI over time and leveraging different versions of the UI to automatically extract robust and consistent identifiers by pattern mining techniques. Also, according to some embodiments described herein, domain knowledge is partially captured as a conceptual hierarchy that can be understood as an ontology of UI elements.

[0007] The embodiments described herein are directed to methods and systems for performing descriptive concept mining and performing re-identification checks of UI elements within the current UI.

[0008] According to one aspect of the present disclosure, a computer-implemented method for performing descriptive concept mining is provided. The method comprises: obtaining a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by a concept; generating at least one of a local description of the concept or a global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept comprises descriptive components specific to each UI and the global description of the concept comprises descriptive components not specific to the UI; associating by storing at least one of the generated local description or global description with a concept identifier corresponding to the concept.

[0009] According to another aspect of the present disclosure, in a current UI, a computer-implemented method for performing a re-identification check of user interface (UI) elements is provided. The method comprises: for a UI element, extracting at least one of a stored first local description or a stored first global description based on a concept identifier representing the UI element; determining, by querying a representation of the current UI, whether each description can be used to re-identify the UI element using at least one stored first local description and the stored first global description.

[0010] According to another aspect of the present disclosure, a system for performing concept description mining is provided. The system comprises: an acquisition unit configured to acquire a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by a concept; a generation unit configured to generate at least one of a local description of the concept or a global description of the concept based on the plurality of reference descriptions, the local description of the concept comprising description components specific to each UI, and the global description of the concept comprising description components not specific to the UI; a storage unit configured to store at least one of the generated local description or global description associated with a concept identifier corresponding to the concept.

[0011] According to another aspect of the present disclosure, a system for performing a re-identification check of user interface (UI) elements in a current UI is provided. The system comprises: an acquisition unit configured to extract, for a UI element, at least one of a stored first local description or a stored first global description based on a concept identifier representing the UI element; an interrogation unit configured to determine, by querying a representation of the current UI using at least one stored first local description and the stored first global description, whether each description can be used to re-identify the UI element.

[0012] Optional features are described in the appended dependent claims.

[0013] These and other aspects of the disclosure will be better recognized and understood when considered in conjunction with the following description and the accompanying drawings. The following description sets forth various embodiments of the disclosure and numerous specific details thereof, given by way of example and not limitation. Within the scope of the disclosure, many substitutions, modifications, additions, or rearrangements are possible, and the disclosure includes all such substitutions, modifications, additions, or rearrangements.

Brief Description of the Drawings

[0014] Embodiments of the disclosure will be described by way of example only with reference to the accompanying drawings:

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0015] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments may be practiced without these specific details. In other instances, well-known materials or methods have not been described in detail to avoid obscuring the present disclosure.

[0016] FIG. 1 is a flowchart of a computer-implemented method for performing description mining of a UI concept (simply referred to as "concept" herein) according to one embodiment of the present disclosure.

[0017] Method 100 begins at step S110 where a plurality of reference descriptions are obtained. Each of the plurality of reference descriptions is associated with a user interface (UI) element corresponding to the type of UI element represented by the concept.

[0018] In the context of the present embodiment, descriptions (such as reference descriptions, local descriptions, and global descriptions) may indicate at least one of the location and attributes of each UI element or each concept. For example, in some embodiments, each of the plurality of reference descriptions obtained in step S110 may be in the form of one of a Cascading Style Sheet (CSS) selector, an Extended Markup Language Path Language (XPath) expression, an image, and / or audio. The image may be, for example, computer vision (CV).

[0019] As an example, the concept can represent a UI element of the "OK button" type. In this example, each of the plurality of reference descriptions may be a description for a different "OK button", for example, for different UIs and / or different versions of the UI. For example, the plurality of reference descriptions may be XPath expressions for the OK buttons of different UIs and / or different versions of a particular UI.

[0020] In some embodiments, the plurality of reference descriptions may comprise at least one of a local history reference description and a non-local reference description. In these embodiments, the local history reference description is specific to the current UI (e.g., the UI being used by the user), and the non-local reference description is not specific to the current UI. For example, the local history reference description corresponds to a UI element that is part of the current UI. For example, the non-local reference description corresponds to a UI element that is not part of the current UI but is similar to the current UI.

[0021] Next, in step S120, based on the plurality of reference descriptions obtained in step S110, at least one of a local description of the concept and a global description of the concept is generated. The local description of the concept includes description components specific to each UI, e.g., the current UI. The global description of the concept includes components that are not specific to the UI, and thus, the global description can be reused in different UIs (e.g., different applications or web pages). In some cases, a plurality of local / global descriptions corresponding to different parallel ways of identifying each concept / UI element can be generated in step S120. For example, the generated descriptions can include XPath expressions, CSS selectors, CV descriptions, etc. In these examples, even if one form / type of description becomes invalid, another form / type can be used for the purpose of identifying the UI element. Further, in some embodiments, the former description (i.e., the invalidated description) can be updated using data from the UI element identified using the latter (i.e., the description used to identify the UI element).

[0022] In some embodiments, generating a local description in S120 may include updating an existing local description of the concept. Alternatively or additionally, in some embodiments, generating a global description in step S120 may include updating an existing global description of the concept. These existing descriptions may be stored in a database (e.g., the "UI concept description" storage shown in FIG. 5).

[0023] Generating the local and / or global description may include performing pattern mining with respect to at least a plurality of reference descriptions. More specifically, in some embodiments, generating a local description of the concept in S120 may include determining a common description component from a plurality of local history reference descriptions. Alternatively or additionally, generating a global description of the concept in S120 may include determining a common description component from a plurality of non-local reference descriptions. In these embodiments, determining the common description component may be performed using a trained machine learning model. For example, determining the common description component may be performed using some machine learning techniques such as a neural network based on computer vision, a decision tree using structured information from a DOM tree (e.g., tags, text content), or a version space method based on the features of UI elements.

[0024] Subsequently, in step S130, at least one of the local description and the global description generated in step S120 is stored so as to be associated with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept within a set of concepts. For example, the concept identifier can indicate the order of types of UI elements represented by concepts in an ontology of a plurality of different types of UI elements. The set of concepts may be a concept hierarchy in which concepts are presented based on the types of UI elements represented by the concepts. For example, an "OK button" and a "login button" are sub - concepts corresponding to the more general concept of "send button" within the hierarchy. The concept hierarchy can represent the domain knowledge of experts in the field of automation, more specifically, robotic process automation.

[0025] In some embodiments, method 100 may further include obtaining user input for accepting or rejecting at least one of the generated local description and the generated global description. In these embodiments, the step of storing the generated local description and / or the generated global description in step S130 may be performed only when user input for accepting each generated local description or generated global description is obtained.

[0026] As described above with respect to S120, in some embodiments, generating a local description of a concept in S120 can include determining a common description component from a plurality of local history reference descriptions. In these embodiments, storing the local description generated in S130 can include storing, as at least a part of the local description, the determined common description component.

[0027] Also, as described above with respect to S120, in some embodiments, generating a global description of a concept at S120 can include determining a common description component from a plurality of non-local reference descriptions. In these embodiments, storing the global description generated at S130 can include storing the determined common description component as at least a part of the global description.

[0028] Although not shown in the flowchart of FIG. 1, in some embodiments, method 100 can further include, for each of the plurality of reference descriptions obtained at step S110, obtaining a corresponding UI representation. In these embodiments, generating a local description of a concept or a global description of a concept at step S120 can further be based on the interrelationship between the reference description and the corresponding UI representation.

[0029] The UI representation in some embodiments can comprise at least one of a structured representation of each UI and an unstructured representation of each UI. For example, the structured representation can include a Document Object Model (DOM) tree. As another example, the unstructured representation of a UI can comprise a screenshot of each UI.

[0030] Furthermore, although not shown in the flowchart of FIG. 1, in some embodiments, method 100 can further include generating an updated UI description by removing information content from the original UI description before obtaining a description of the corresponding UI for each reference description. Thus, in these embodiments, the obtained representation can be the updated representation (instead of the original representation). As an example, the original UI representation can comprise a DOM tree, and the information content (to be removed) can comprise text nodes and / or image nodes within the DOM tree. The updated representation can be regarded as a "privacy-guaranteed" version of the UI representation. This is because information content that may raise privacy concerns has been removed from this version.

[0031] In some embodiments, method 100 may further comprise determining whether a predetermined trigger condition is met before generating at least one of a local description and a global description of the concept in step S120. In these embodiments, generating at least one of the local description and the global description in step S120 may be performed only if it is determined that the predetermined trigger condition is met. The predetermined trigger condition may be associated with at least one of a predetermined time interval, storage of new concept instances of the concept, the position of each UI element (e.g., spatial coordinates, or an identifier such as a CSS selector, or a Win32 object attribute), a concept instance including the representation of each UI, and an indicative user input. The new concept instance may refer to a concept instance associated with a UI element that cannot be identified by the existing description stored for the corresponding concept identifier.

[0032] As an example of a predetermined trigger condition, the step of generating the local and / or global description may be manually triggered by receiving a user input that requires descriptive mining to be performed. As another example, the step of generating the local and / or global description may be triggered by a specific time interval / frequency, for example, a predefined command to perform descriptive mining every Monday.

[0033] Although not shown in FIG. 1, in some embodiments, method 100 may further have the following: a step of obtaining a user input, wherein the user input corresponds to assigning concept identifiers to first UI elements in respective UIs; a step of storing concept instances of the concept identifiers. The concept instance may include a representation of each UI, a description of the first UI element, and a concept identifier. For example, a concept example of the "OK button" concept may include a representation of the UI including the OK button (e.g., a DOM tree), an XPath representation of the OK button, and a concept identifier of the "OK button" (which may be the "OK button" itself or any identifier). Thus, each UI element in the user interface can be regarded as an instance of a concept in the context of the present disclosure. As described above, storing the (new) concept instance can be a predetermined trigger condition for initiating the generation of at least one of a local description and a global description.

[0034] FIG. 2 is a flowchart of a method for performing a re-identification check of UI elements in a current UI according to one embodiment of the present disclosure. In some embodiments, the UI element may be part of an action in an automated workflow. Also, in some embodiments, performing a re-identification check of UI elements as described below may be part of the execution of an automated workflow.

[0035] Method 200 starts at step S210 where at least one of a stored first local description and a stored first global description is obtained for the UI element. This obtaining is based on a concept identifier representing the UI element. For example, if the UI element is an "OK button", it may be associated with a concept identifier that identifies the "OK button" as a related concept. The first local description and / or the first global description may be generated for the concept based on the method described above with reference to FIG. 1.

[0036] As described above, in some embodiments, the UI element can be part of an action in an automated workflow. In these embodiments, in step S210, before retrieving at least one of the stored first local description and the stored first global description, the method may further include retrieving the automated workflow. The automated workflow includes at least an action related to the UI element. In a practical implementation, the automated workflow can include a plurality of actions, and each action includes a plurality of UI elements. Retrieving the automated workflow may include receiving a user input corresponding to an action in the automated workflow. For example, the user input corresponds to an operation for creating a corresponding action in the automated workflow.

[0037] Returning to FIG. 2, in step S220, the current UI representation is queried using at least one stored first local description and the stored first global description, thereby determining whether each description can be used to re-identify the UI element. For example, the first local description may be an XPath description, and in step S220, the DOM tree representing the current UI may be queried to determine whether there is a match for this XPath description within the DOM tree. If a match is determined, the re-identification of the UI element is successful. On the other hand, if no match is determined, the re-identification of the UI element is not successful.

[0038] In the context of the present disclosure, it should be understood that the terms "re-identify" and "re-identifying" may refer to the process of identifying UI elements from a UI (e.g., by finding a match within the UI representation), or more specifically, from a UI representation (e.g., a DOM tree). Thus, it is not a prerequisite for a UI element to have been previously "identified" in order for it to be "re-identified" in step S220. Further, the terms "re-identify" and "re-identifying" may refer to scenarios where a UI element cannot be identified using local / global descriptions, and thus, different local / global descriptions are used to perform the "re-identification" (e.g., by finding a match in the UI representation). This will be explained in more detail below with respect to scenarios where a first local / global description cannot be used to re-identify a UI element and a second local / global description is retrieved for the re-identification check.

[0039] In some embodiments where both a stored first local description and a stored first global description are available, the method may include retrieving at least the first local description based on a concept identifier in step S210 and querying a presentation of the current UI using the first local description in step S220. In this way, the method can prioritize the use of the local description over the global description in the re-identification check, and since the local description is specific to the current UI, a more reliable result can be provided.

[0040] In some embodiments, the method may further include storing a representation of the current UI and the first local description and / or the first global description so as to be associated with the concept identifier if it is determined in step S220 that the first local description and / or the first global description may be used to re-identify a UI element.

[0041] Alternatively, if it is determined in step S220 that the UI elements cannot be re-identified using the first local description and / or the first global description, the method may further include discarding the first local description and / or the first global description that cannot be used for re-identification. Instead of or in addition to this, if it is determined in step S220 that the UI elements cannot be re-identified using the first local description and / or the first global description, the method may further include: based on the concept identifier, retrieving at least one of the stored second local description and the stored second global description, where the second local description is different from the first local description and the second global description is different from the second global description; querying the current UI representation using at least one of the second local description and the second global description to determine whether the UI elements can be re-identified using each description.

[0042] As yet another alternative, in some embodiments, if it is determined that the UI elements cannot be re-identified using the first local description and / or the first global description, the method may further comprise requesting user input to indicate the UI elements in the current UI, and / or performing concept description mining. The description mining may be performed according to the method described above with reference to FIG. 1.

[0043] It will be understood that in some embodiments of the present disclosure, the methods shown in FIGS. 1 and 2 can, in certain embodiments, be combined to enable both description mining and re-identification checking to be performed. An exemplary process in which both description mining and re-identification checking are performed is shown in FIG. 5 and will be described in more detail with reference to the following drawings.

[0044] FIG. 3 is a block diagram of a system for performing concept description mining according to an embodiment of the present disclosure.

[0045] As shown in FIG. 3, a system 300 is provided that includes an acquisition unit 310, a generation unit 320, and a storage unit 330.

[0046] The acquisition unit 310 is configured to acquire a plurality of reference descriptions. Each of the plurality of reference descriptions is associated with a UI element corresponding to the type of UI element represented by a concept.

[0047] As described above with reference to FIG. 1, descriptions (such as reference descriptions, local descriptions, and global descriptions) can indicate at least one of the location and attributes of each UI element or each concept. For example, in some embodiments, each of the plurality of reference descriptions acquired by the acquisition unit 310 can be in the form of one of a cascading style sheet (CSS) selector, an extended markup language path language (XPath) expression, an image, and / or an audio description. The image description can be, for example, a computer vision (CV) description.

[0048] In some embodiments, the plurality of reference descriptions can include at least one of a local history reference description and a non-local reference description. In these embodiments, the local history reference description is specific to the current UI, and the non-local reference description is not specific to the current UI. For example, the local history reference description can correspond to a UI element that is part of the current UI. For example, the non-local reference description can correspond to a UI element that is not part of the current UI but is similar to the current UI.

[0049] The generation unit 320 generates at least one of a local description of a concept and a global description of the concept based on a plurality of reference descriptions acquired by the acquisition unit 310. The local description of the concept includes description components specific to each user interface (UI). The global description of the concept includes description components that are not specific to the UI, and thus the global description can be reused in different UIs (e.g., different applications or web pages). The generation of the local and / or global description may involve performing pattern mining with respect to at least the plurality of reference descriptions. In some cases, a plurality of local / global descriptions corresponding to different parallel methods for identifying each concept / UI element may be generated by the generation unit 320. Thus, in some embodiments, if one form / type of description becomes invalid, another form / type of description can be used instead to identify the UI element. Also, the invalidated description can be updated using data from the UI element identified using other descriptions.

[0050] In some embodiments, the generation unit 320 may be configured to generate the local description by updating an existing local description of the concept. Alternatively or in addition, in some embodiments, the generation unit 320 may be configured to generate the global description by updating an existing global description for the concept. These existing descriptions may be stored in a UI concept description storage device (e.g., the "UI concept description" storage device shown in FIG. 5).

[0051] The generation unit 320 may be configured to generate a local description for a concept by determining common description components from a plurality of local history criteria descriptions. Alternatively or in addition, the generation unit 320 may be configured to generate a global description for a concept by determining common description components from a plurality of non-local criteria descriptions. Further, in these embodiments, the generation unit 320 may be configured to determine common description components using a trained machine learning model. For example, determining the common description components may be performed using several machine learning techniques, such as a neural network based on computer vision, a decision tree using structured information from a DOM tree (e.g., tags, text content), or a version space method based on features of UI elements.

[0052] The storage unit 330 is configured to store at least one of the generated local and global descriptions in association with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept within a set of concepts. For example, the concept identifier can indicate the order of types of UI elements represented by concepts in an ontology of a plurality of different types of UI elements. The set of concepts may be a concept hierarchy in which concepts are presented based on the types of UI elements represented by the concepts. For example, an "OK button" and a "login button" are subordinate concepts corresponding to the more general concept of "submit button" within the hierarchy. The concept hierarchy can represent the domain knowledge of experts in the field of automation, more specifically, robotic process automation.

[0053] In some embodiments, the acquisition unit 310 may be further configured to acquire user input that accepts or rejects at least one of the generated local description and the generated global description. In these embodiments, when the generation unit 310 acquires user input that accepts each generated local description or generated global description, the storage unit 330 is configured to store only the generated local description and / or the generated global description.

[0054] As described above, in some embodiments, the generation unit 320 may be configured to generate a local description of a concept by determining a common description component from a plurality of local history reference descriptions. In these embodiments, the storage unit 330 may be configured to store the generated local description by storing the common description component determined as at least a part of the local description.

[0055] Also, as described above, in some embodiments, the generation unit 320 may be configured to generate a global description of a concept by determining a common description component from a plurality of non-local reference descriptions. In these embodiments, the storage unit 330 may be configured to store the generated global description by storing the determined common description component as at least a part of the global description.

[0056] In some embodiments, the acquisition unit 310 may be further configured to acquire a corresponding UI representation for each of the plurality of reference descriptions. In these embodiments, the generation unit 320 may be configured to generate a local description of a concept or a global description of a concept further based on the correlation between the reference description and the corresponding UI representation.

[0057] In some embodiments, the UI representation may comprise at least one of the structured representation of each UI and the unstructured representation of each UI. For example, the structured representation can include a Document Object Model (DOM) tree. As another example, the unstructured representation of the UI may comprise screenshots of each UI.

[0058] In some embodiments, the generation unit 320 may be configured to generate an updated UI representation by removing information content from the original UI representation. This operation may be performed before the acquisition unit 310 acquires the corresponding UI representation for each reference description. In these embodiments, the acquired representation may be the updated representation. For example, the original UI representation can include a DOM tree, and the information content (to be removed) can include text nodes and / or image nodes within the DOM tree. The updated representation can be regarded as a "privacy-guaranteed" version of the UI representation. This is because information content that may raise privacy concerns has been removed from this version.

[0059] Although not shown in FIG. 3, in some embodiments, the system 300 may further comprise a determination unit configured to determine whether a predetermined trigger condition is met before the generation unit 320 generates at least one of the local description and the global description of the concept. In these embodiments, the generation unit 320 may be configured to execute the generation of at least one of the local description and the global description only when determined by the determination unit that the predetermined trigger condition is met. The predetermined trigger condition may be associated with at least one of a predetermined time interval, the storage of a new concept instance of the concept, the position of each UI element (e.g., spatial coordinates) and the concept instance including the representation of each UI, and an indicative user input. The new concept instance can refer to a concept instance associated with a UI element that cannot be identified by the existing description stored for the corresponding concept identifier.

[0060] As an example of a predefined trigger condition, the generation of local and / or global descriptions by the generation unit 3120 can be manually triggered by receiving user input that requires the execution of description mining. As another example, the generation of local and / or global descriptions can be triggered by a specific time interval / frequency, for example, a predefined command for performing description mining on each Monday.

[0061] In some embodiments, the acquisition unit 310 can be further configured to acquire user input. The user input corresponds to the assignment of a concept identifier to a first UI element in each UI. Further, in these embodiments, the storage unit 330 can be further configured to store concept instances of the concept identifiers. The concept instances include the representation of each UI, the description of the first UI element, and the concept identifier. For example, a concept instance of the "OK button" concept can include the representation of the UI (e.g., DOM tree) including the OK button, the XPath representation of the OK button, and the concept identifier of the "OK button" (which can be the "OK button" itself or any identifier). Therefore, each UI element in the user interface can be regarded as an instance of a concept in the context of the present disclosure. As described above, storing (new) concept instances can be a predefined trigger condition for starting the generation of at least one of the local description and the global description.

[0062] Although not shown in FIG. 3, in some embodiments, the components including the acquisition unit 310, the generation unit 320, and the storage unit 330 can be implemented as a combination of a processing circuit and a memory within the system 300.

[0063] FIG. 4 is a block diagram of a system for performing a re-identification check of UI elements in a current UI according to an embodiment of the present disclosure. In some embodiments, the UI element can be part of an action of an automated workflow.

[0064] As shown in FIG. 4, a system 400 is provided that includes an acquisition unit 410 and a query unit 420. The acquisition unit 410 is configured to search for at least one of a stored first local description and a stored first global description for a UI element based on a concept identifier representing the UI element. This search is based on the concept identifier representing the UI element. For example, if the UI element is an "OK button", the UI element can be associated with a concept identifier that identifies "OK button" as a related concept. The first local description and / or the first global description may be generated for the concept based on the method described above with reference to FIG. 1, or may be generated by the system 300 in the manner described with reference to FIG. 3.

[0065] As described above, in some embodiments, the UI element can be part of an action in an automated workflow. In these embodiments, the system 400 can further include an acquisition unit (not shown in FIG. 4) configured to acquire an automated workflow before the acquisition unit 410 retrieves at least one of the stored first local description and the stored first global description, and the automated workflow includes at least an action involving the UI element. In a practical implementation, the automated workflow can include a plurality of actions, and each action can include a plurality of UI elements. The acquisition unit can be configured to acquire the automated workflow by receiving a user input corresponding to an action in the automated workflow. For example, the user input can correspond to an operation for creating a corresponding action in the automated workflow.

[0066] The query unit 420 is configured to query the representation of the current UI using at least one stored first local description and a stored first global description, and to determine whether each description can be used to re-identify a UI element. For example, the first local description may be an XPath description, and the query unit 420 may be configured to query the DOM tree representing the current UI to determine whether there is a match for this XPath description within the DOM tree. If a match is determined, the re-identification of the UI element is successful. On the other hand, if no match is determined, the re-identification of the UI element is not successful. In some embodiments, performing the re-identification check of the UI element by the query unit 420 may be regarded as part of the execution of an automated workflow.

[0067] As described above with reference to FIG. 2, the terms "re-identify" and "re-identifying" may refer to the process of identifying a UI element from the UI, or more specifically from the UI representation, without necessarily requiring that the UI element has been "identified" previously. Further, the terms "re-identify" and "re-identifying" may refer to a scenario where a UI element cannot be identified using local / global descriptions, and thus different local / global descriptions are used to perform "re-identification".

[0068] In some embodiments where both the stored first local description and the stored first global description are available, the acquisition unit 410 may be configured to retrieve at least the first local description based on a concept identifier. Also, the query unit 420 may be configured to query the presentation of the current UI using the first local description. In this way, the system 400 can prioritize the use of local descriptions over global descriptions in the re-identification check, and since the local description is specific to the current UI, it can provide a more reliable result.

[0069] In some embodiments, the system 400 may further comprise a memory unit configured to store the current UI representation and the first local description and / or the first global description in association with a concept identifier, if it is determined by the query unit 420 that the first local description and / or the first global description can be used to re-identify UI elements.

[0070] Alternatively, the system 400 may further comprise a discard unit configured to discard the first local description and / or the first global description if it is determined by the query unit 420 that the first local description and / or the first global description cannot be used to re-identify UI elements. Further, alternatively or additionally, if it is determined by the query unit 420 that the UI elements cannot be re-identified using the first local description and / or the first global description, the acquisition unit 410 may be further configured to retrieve at least one of the stored second local description and the stored second global description based on the concept identifier. The second local description is different from the first local description, and the second global description is different from the second global description. Further, the query unit 420 may be further configured to query the current UI representation using at least one of the second local description and the second global description to determine whether each description can be used to re-identify UI elements.

[0071] As yet another alternative, in some embodiments, the system 400 may further comprise a request unit configured to request user input to indicate UI elements in the current UI and / or to perform concept description mining if it is determined by the query unit 420 that the first local description and / or the first global description cannot be used to re-identify UI elements. The description mining may be performed according to the method described above with reference to FIG. 1 and / or by the system 300 described above with reference to FIG. 3.

[0072] Although not shown in FIG. 4, in some embodiments, the components including the acquisition unit 410 and the query unit 420 may be implemented as a combination of a processing circuit and a memory within the system 400.

[0073] In some embodiments of the present disclosure, the functions of the systems 300 and 400 shown in FIGS. 3 and 4 can be combined as a single system in a particular embodiment, and it will be understood that such a system enables both performing descriptive mining and re-identification checking.

[0074] FIG. 5 is a diagram showing an exemplary process for performing descriptive mining of concepts and performing re-identification checking of UI elements within the current UI according to one embodiment of the present disclosure.

[0075] As shown in FIG. 5, the method is shown by several steps numbered (i) to (xi) in the figure for ease of description. These steps will be described in more detail below with reference to a user who is an automation developer using an RPA editor to construct a workflow. In the context of this embodiment, the user is constructing a workflow for the open application UI by using specific UI elements within the application UI.

[0076] (i) The user highlights the UI element of interest. In the example shown in FIG. 5, this is the OK button of the application UI.

[0077] (ii) The RPA editor or system (e.g., the system 300 in FIG. 3) captures the highlighted element. In other words, the highlighted element is located within the current application UI.

[0078] (iii) As described above with reference to FIG. 1, for example, a UI element can be regarded as an instance or example of a concept. For example, the OK button in the current UI as shown in FIG. 5 can be regarded as an instance of the "OK button" concept or an instance of the "login button" concept, and both of them can be regarded as sub - concepts of the "send button" concept. Before the user highlights each instance on the UI, these concepts can be created and placed at appropriate positions within the concept hierarchy. Alternatively, an existing concept hierarchy can be directly used by the user.

[0079] (iv) The representation of the entire UI and the positions of the highlighted UI elements within the UI are stored as instances in the "UI representation and highlighted objects" storage together with concept identifiers (sometimes called "concept IDs"). The representation of the UI may not be structured (e.g., a screenshot of the UI), may be structured (e.g., a DOM tree), or may be both. The positions of the elements of interest within the UI representation are in the form of spatial coordinates or identifiers that uniquely identify it, such as CSS selectors or XPath expressions. If there are privacy concerns, a structured representation of the UI such as the DOM can be used, but all information content (e.g., text and image nodes) is removed. The following table shows an example of the information saved as an instance for the "OK button" concept. For each application or website, instances can be stored separately for each concept.

[0080]

Table 1

[0081] (v) In this example, storing a new concept instance in the "UI representation and highlighted objects" database can trigger descriptive mining by a UI concept description miner (sometimes simply called the "description miner" in this specification).

[0082] (vi) The description miner creates and / or updates local and global descriptions of related concepts. For example, if the OK button highlighted in step (i) belongs to the SAP website, the description miner can create and / or update the description(s) of the "OK button" concept for this particular website, as well as the global description(s) that can be used as the default description(s) of the "OK button" concept for other websites and applications. The global description can be generated using all the stored information across different websites and applications for this particular concept, while the local description can be generated for each website or application. The description miner may be composed of a number of specialized miners in some embodiments. For example, the XPath miner can create and / or update the XPath expressions for each UI element, while the CSS miner can create and / or update CSS selectors. Similarly, the computer vision (CV) miner can use the screen shot(s) of the UI to create the CV description(s) of the concept both locally and globally. This means that the description miner can generate different types of descriptions for the same concept and store them in the "UI concept description" storage device. The "mined" descriptions can be reviewed by a human user, and as a result, the mined descriptions can be accepted or rejected after review.

[0083] Even if, in many cases, the elements of interest can be automatically identified using existing descriptions, it would still be useful to create an efficient and robust (local / global) description over time. For example, an XPath description may already be sufficient to find an instance of a submit button on a website, but a describer miner can use some of the stored UI descriptions of the same website to generate a more robust CSS selector. Thus, even if the stored XPath expression becomes invalid in the future, the mined CSS selector can be used to self-heal the workflow and re-identify the intended UI elements.

[0084] (vii) Each mined description is paired with a relevant concept identifier (i.e., corresponding to the concept representing the type of each UI element), and thus they can be queried and accessed through the corresponding concept identifier.

[0085] (viii) The user creates a workflow by using the concept identifier and refers to the corresponding element of interest. Thus, the identifier of the UI element is separated from the workflow description. This allows the description of the UI element (i.e., the concept) to be changed later without making any changes to the existing workflow.

[0086] (ix) The digital worker can execute the workflow completely or partially during workflow design time to check whether the workflow or a part of the workflow is operating properly. While doing so, the concept is queried to retrieve the description of the corresponding UI element.

[0087] (x) The digital worker re-identifies the elements of interest using the retrieved descriptions (i.e., XPath expressions, CSS selectors, etc.). If the elements are correctly re-identified, the UI representation and the automatically re-identified elements are stored for future use, for example, to be used to discover better descriptions. If the elements cannot be re-identified, the user may be asked to re-highlight the elements.

[0088] (xi) During the normal execution of the workflow, if the existing descriptions (searched based on the concept identifiers) cannot be used to re-identify the UI elements, the human operator can be asked to highlight the unidentified elements on the screen: Steps (ix) and (x) can be repeated later, and if possible, the execution of the workflow can be resumed.

[0089] Of course, this description is merely an example, and changes and modifications can be made to the described embodiments without departing from the scope of the present disclosure as defined in the claims.

[0090] The technology has been described in detail for purposes of illustration based on what is currently considered to be the most practical and preferred embodiments, but such detailed description is for that purpose only, and the technology is not limited to the disclosed embodiments. On the contrary, it is to be understood that it is intended to cover modifications and equivalent configurations within the scope of the appended claims. For example, it is to be understood that the technology contemplates that, to the extent possible, one or more features of any implementation form can be combined with one or more features of any other implementation form.

[0091] Throughout this specification, references to "one embodiment", "an embodiment", "one example" or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the invention. Thus, the phrases "in one embodiment", "in an embodiment", "one example" or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable combination and / or sub-combination in one or more embodiments or examples. Additionally, the drawings provided herein are for the purpose of illustration to those skilled in the art and it is understood that the drawings are not necessarily drawn to scale.

[0092] Embodiments according to the present disclosure may be embodied as an apparatus, a method, or a computer program product. For example, in some embodiments, when a program is executed by a processor, a computer program product may be provided that includes instructions for causing the processor to execute the methods described with respect to FIGS. 1 and / or 2. As another embodiment, a computer-readable storage medium may be provided that includes instructions for causing a computer to execute the steps of the methods described in FIGS. 1 and / or 2 when executed by the computer. Thus, the present embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software aspects and hardware aspects that are generally referred to herein as "modules" or "systems". Furthermore, embodiments of the present disclosure can take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0093] Although described in connection with an exemplary computing system environment, embodiments of the present disclosure operate in numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with aspects of the present disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable household appliances, mobile phones, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0094] Any combination of one or more computer-usable or computer-readable media may be utilized. For example, the computer-readable media may include one or more of a portable computer diskette, hard disk, random access memory (RAM) device, read-only memory (ROM) device, erasable programmable read-only memory (EPROM or flash memory) device, portable compact disc read-only memory (CDROM), optical storage device, and magnetic storage device. Computer program code for carrying out operations of embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0095] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of code that comprises one or more executable instructions for implementing the specified logical function. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions. These computer program instructions may be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner so as to produce a product that includes instruction means for implementing the functions / operations specified in the flowchart and / or block diagram.

[0096] The following is a list of embodiments of the present disclosure.

[0097] 1. A computer-implemented method for performing concept description mining, comprising: obtaining a plurality of reference descriptions, each of the plurality of reference descriptions being associated with an element corresponding to a type of UI element represented by the concept in a user interface, UI; generating at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, the local description of the concept comprising description components specific to each UI, and the global description of the concept comprising description components not specific to the UI; storing at least one of the generated local description and global description so as to be associated with a concept identifier corresponding to the concept; A method having the above.

[0098] 2. The concept identifier uniquely identifies the concept within the concept set, and the concept set is an ontology of a plurality of different types of UI elements, the method according to Embodiment 1.

[0099] 3. The plurality of reference descriptions include at least a local history reference description and a non-local reference description, the local history reference description is specific to the current UI, and the non-local reference description is not specific to the current UI, the method according to Embodiment 2.

[0100] 4. Generating a local description includes updating an existing local description of a concept, and / or generating a global description includes updating an existing global description of a concept, the method according to any one of the foregoing embodiments.

[0101] 5. The description indicates at least one of the location and attributes of each UI element or each concept, the method according to any one of the foregoing embodiments.

[0102] 6. Each of the plurality of reference descriptions is in one form of a cascading style sheet, CSS, selector, extended markup language path language, XPath, description formula, image and / or audio description, the method according to any one of the foregoing embodiments.

[0103] 7. The method further includes, for each of the plurality of reference descriptions, obtaining a corresponding UI representation, and the step of generating a local description about a concept or a global description about a concept is further based on the correlation between the reference description and the corresponding UI representation, the method according to any one of the above embodiments.

[0104] 8. The UI representation includes at least one of a structured representation of each UI and an unstructured representation of each UI, the method according to Embodiment 7.

[0105] 9. The method according to embodiment 8, wherein the structured representation of the UI includes a document object model, DOM, tree, and / or the unstructured representation of the UI includes screenshots of the respective UIs.

[0106] 10. Before obtaining the corresponding UI representation of each reference description, further comprising the step of generating an updated UI representation by removing information content from the original UI representation, and the obtained representation is the updated representation. The method according to any one of embodiments 7 to 9.

[0107] 11. The method according to embodiment 10, wherein the original UI representation includes a DOM tree, and the information content includes text nodes and / or image nodes in the DOM tree.

[0108] 12. Generating the local description for the concept includes determining common description components from a plurality of local history reference descriptions, and storing the generated local description includes storing the determined common description components as at least part of the local description. The method according to any one of the foregoing embodiments.

[0109] 13. Generating the global description for the concept includes determining common description components from a plurality of non-local reference descriptions, and storing the generated global description includes storing the determined common description components as at least part of the global description. The method according to any one of the foregoing embodiments.

[0110] 14. Determining the common description components is performed using a trained machine learning model. The method according to embodiment 12 or embodiment 13.

[0111] 15. Before generating at least one of the local description and the global description for the concept, comprising the step of determining whether a predetermined trigger condition is satisfied. The generation of at least one of the local description and the global description is executed only when it is determined that a predetermined trigger condition is satisfied. A method according to any one of the preceding embodiments.

[0112] 16. The method according to embodiment 15, wherein the predetermined trigger condition is associated with at least one of the following: A predetermined time interval; Remembering new concept instances of a concept, the concept instances including the position of each UI element and the representation of each UI; An indication user input.

[0113] 17. The method further comprises Obtaining a user input, the user input corresponding to the assignment of the concept identifier to a first UI element in each UI, Storing a concept instance for the concept identifier, the concept instance comprising the representation of each UI, the description of the first UI element, and the concept identifier; and has The storing of the concept instance is a predetermined trigger condition The method according to embodiment 15 or embodiment 16.

[0114] 18. The method further comprises obtaining a user input for accepting or rejecting at least one of the generated local description and the generated global description, Storing the generated local description and / or the generated global description is executed only by obtaining a user input for accepting each generated local description or the generated global description. The method according to any one of the above embodiments.

[0115] 19. A computer-implemented method for performing a re-identification check of user interface, UI, elements in a current UI For a UI element, obtaining at least one of a stored first local description and a stored first global description based on a concept identifier representing the UI element; Using at least one stored first local description and a stored first global description to query a representation of a current UI element to determine whether each description can be used to re-identify the UI element; A method having the above.

[0116] 20. The UI element is part of an action in an automated workflow. Before retrieving the at least one of the stored first local description and the stored first global description, the method further includes retrieving the automated workflow, where the automated workflow includes at least an action involving the UI element. The method according to embodiment 19.

[0117] 21. Retrieving the automated workflow includes receiving user input corresponding to the action in the automated workflow. The method according to embodiment 20.

[0118] 22. Executing the re-identification check of the UI element is part of the execution of the retrieved automated workflow. The method according to any one of embodiments 19 to 21.

[0119] 23. Both the stored first local description and the stored first global description are available. The method includes retrieving at least the first local description based on the concept identifier and querying a presentation of the current UI using the first local description. The method according to any one of embodiments 19 to 22.

[0120] 24. If it is determined that the first local description and / or the first global description can be used to re-identify the UI element, further comprising, in association with the concept identifier, storing the current UI representation and the first local description and / or the first global description, the method according to any one of embodiments 19 to 23.

[0121] 25. If it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, a step of obtaining at least one of the stored second local description and the stored second global description based on the concept identifier, wherein the second local description is different from the first local description and the second global description is different from the first global description; a step of querying the current UI representation using at least one of the second local description and the second global description to determine whether each description can be used to re-identify the UI element; The method according to any one of embodiments 19 to 23, comprising:

[0122] 26. If it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, further comprising requesting user input to indicate the UI element within the current UI, the method according to any one of embodiments 19 to 23.

[0123] 27. If it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, further comprising performing concept description mining according to any one of embodiments 1 to 18, the method according to any one of embodiments 19 to 23.

[0124] 28. The method according to any one of embodiments 19 to 27, wherein the first local description and / or the first global description is generated according to any one of embodiments 1 to 18.

[0125] 29. A system for performing descriptive mining on a concept, comprising: An acquisition unit configured to acquire a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with an element corresponding to a type of a UI element represented by the concept in a user interface, UI; A generation unit configured to generate at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept includes description components specific to each UI, and the global description of the concept includes description components not specific to the UI; A storage unit configured to store at least one of the generated local description and global description associated with a concept identifier corresponding to the concept; A system comprising the above.

[0126] 32. A system for performing a re-identification check of elements in a current user interface, UI, comprising: An acquisition unit configured to retrieve at least one of a stored first local description and a stored first global description for a UI element based on a concept identifier representing the UI element; A query unit configured to query a representation of the current UI using at least one stored first local description and a stored first global description to determine whether respective descriptions can be used to re-identify the UI element; A system comprising the above.

[0127] 33. A computer-readable storage medium that, when executed by a computer, causes the computer to execute the steps of any one of the methods of Embodiments 1 to 28.

Claims

1. A computer-implemented method (100) for performing conceptual description mining, comprising: obtaining a plurality of reference descriptions (S110), each of the plurality of reference descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; generating at least one of a local description of the concept or a global description of the concept based on the plurality of reference descriptions (S120), the local description of the concept comprising description components specific to each UI, and the global description of the concept comprising description components not specific to the UI; storing at least one of the generated local description and the global description so as to be associated with a concept identifier corresponding to the concept (S130); A method having the above steps.

2. The method according to claim 1, wherein the concept identifier uniquely identifies the concept in a concept set, and the concept set is an ontology of a plurality of different types of UI elements.

3. The method according to claim 1, wherein each of the plurality of reference descriptions is in one form of a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) description formula, an image and / or an audio description.

4. The method further comprises, for each of the plurality of reference descriptions, obtaining a corresponding UI representation, and the step of generating the local description of the concept or the global description of the concept is further based on the correlation between the reference description and the corresponding UI representation. The method according to claim 1.

5. The method further comprises generating an updated UI representation by removing information content from the original UI representation before obtaining the description of the corresponding UI for each reference description, wherein the obtained representation is the updated representation. The method according to claim 4.

6. The step of generating the local description of the concept comprises determining common description components from a plurality of local history reference descriptions, and / or the step of generating the global description of the concept comprises determining common description components from a plurality of non-local reference descriptions. The method according to claim 1.

7. The method further has a step of determining whether a predetermined trigger condition is satisfied, prior to the step of generating at least one of the local description or the global description of the concept. The step of generating at least one of the local description or the global description is executed only when it is determined that the predetermined trigger condition is satisfied. The method according to claim 1.

8. The method further includes a step of obtaining user input, wherein the user input corresponds to assigning the concept identifier to a first UI element in each UI; a step of storing a concept instance of the concept identifier, wherein the concept instance includes the representation of each UI, the description of the first UI element, and the concept identifier; and has storing the concept instance is a predetermined trigger condition. The method according to claim 7.

9. The method further has a step of obtaining user input for accepting or rejecting at least one of the generated local description and the generated global description. The step of storing the generated local description and / or the generated global description is executed only by obtaining user input for accepting each of the generated local description or the generated global description. The method according to claim 1.

10. A computer-implemented method (200) for performing a re-identification check of a user interface (UI) element in a current UI, comprising: a step of obtaining, for the UI element, at least one of a stored first local description or a stored first global description based on a concept identifier representing the UI element, wherein each of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions is associated with a UI element corresponding to a type of the UI element corresponding to the concept identifier, the stored first local description includes a description component specific to each UI, and the stored first global description includes a description component not specific to the UI (step S210); Determining whether each description can be used to query a current UI element using at least one of the stored first local description and the stored first global description to re-identify the UI element (S220); A method having the same. **Claim 11** The method further includes, when it is determined that the first local description and / or the first global description can be used to re-identify the UI element, storing the current UI representation and the first local description and / or the first global description so as to be associated with the concept identifier. The method according to claim 10. **Claim 12** The method further includes, when it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, Obtaining at least one of a stored second local description and a stored second global description based on the concept identifier, wherein the second local description is different from the first local description and the second global description is different from the first global description; Querying the current UI representation using at least one of the second local description and the second global description to determine whether each description can be used to re-identify the UI element; The method according to claim 10, having the same. **Claim 13** The method further includes, when it is determined that the first local description and / or the first global description cannot be used to re-identify the UI element, Requesting user input indicating the UI element within the current UI; Performing description mining of the concept according to claim 1; Performing at least one of the above; The method according to claim 10. **Claim 14** A system (300) for performing description mining of a concept, An acquisition unit (310) configured to acquire a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a user interface (UI) element corresponding to the type of UI element represented by the concept. Acquisition unit; A generation unit (320) configured to generate at least one of the local description of the concept or the global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept includes description components specific to each UI, and the global description of the concept includes description components not specific to the UI; A storage unit (330) configured to store at least one of the generated local description and global description so as to be associated with a concept identifier corresponding to the concept; A system comprising the same.

15. A system (400) for performing a re-identification check of user interface (UI) elements in a current UI, An acquisition unit configured to acquire at least one of a stored first local description or a stored first global description for the UI element based on a concept identifier representing the UI element, wherein each of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions is associated with a UI element corresponding to the type of the UI element corresponding to the concept identifier, the stored first local description includes description components specific to each UI, and the stored first global description includes description components not specific to the UI. Acquisition unit (410); A query unit (420) configured to query the representation of the current UI using at least one of the stored first local description or the stored first global description to determine whether each description can be used to re-identify the UI element; A system comprising the same.

16. A computer-readable storage medium that, when executed by a computer, includes instructions for causing the computer to execute the method according to any one of claims 1 to 13.

Images