Methods and systems for reidentification in a user interface

Abstract

A computer-implemented method (100) for performing description mining of a concept is provided. The method includes: acquiring (S110) a plurality of reference descriptions, each of which is associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; generating (S120) at least one of a local description and a global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept includes a description component specific to the corresponding UI, and wherein the global description of the concept includes a description component not specific to the UI; and storing (S130) the generated local and global descriptions such that they are associated with a concept identifier corresponding to the concept.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to methods and systems for re-identification in user interfaces. More specifically, the present disclosure relates to systems and computer-implemented methods for performing concept description mining and for performing re-identification checks of user interface elements in a current user interface. BACKGROUND

[0002] Human-guided computer processes are ubiquitous in many technical and activity fields. Modern graphical user interfaces (GUIs) have proven invaluable in allowing human operators to use computer systems to perform often complex data processing and / or system control tasks. However, while GUIs generally allow operators to quickly become accustomed to performing new tasks, they provide a high barrier to any further automation of the task.

[0003] Traditional workflow automation aims to take tasks that are typically performed by operators and automate them using a GUI, such that a computer system can perform the same task without requiring extensive re-engineering of the underlying software used to perform the task. Initially, this requires exposing the software’s application programming interface (API) so that scripts can be manually designed to perform the required functionality of the software in order to perform the required task.

[0004] Robotic process automation (RPA) is an emerging form of business process automation technology that utilizes software robots or artificial intelligence (AL) workers. In traditional workflow automation tools, a software developer produces a list of actions using an internal application interface to automate a task. In contrast, RPA systems develop a list of actions by observing a user performing a task in an application’s graphical user interface, and then perform the automation by repeating those tasks directly on the graphical user interface. RPA robots can then generate appropriate input commands for the GUI to cause a computer system to perform a given process. This enables process automation, turning a manned process into an unmanned process. The advantages of this approach are numerous and include greater scalability allowing multiple RPA robots to perform the same task across multiple computer systems, and greater repeatability as the potential for human error is reduced or even eliminated in a given process. SUMMARY

[0005] Existing automation approaches typically rely on extracting unique identifiers of user interface (UI) elements and building automations based on these identifiers. Examples of such identifiers for web-based applications are Extensible Markup Language Path Language (XPath) expressions and Cascading Style Sheet (CSS) selectors, which can also be combined with various properties of the UI elements. The unique identifiers are typically defined manually by RPA developers during application modeling or workflow development. However, these unique identifiers can be susceptible to changes in the UI. For example, an XPath expression can be invalidated by the addition of new sources (e.g., a container that is not visible to the user) or changes to the identifiers of existing source elements.

[0006] Some identifiers of UI elements are more robust to changes in the UI, and domain experts can use such identifiers to identify UI elements. In doing so, the experts can use trial and error outside of domain knowledge. The methods and systems presented herein are inspired from this - they are based on the idea that representations of the UI are kept over time, and these possibly different versions of the UI are leveraged to automatically extract more robust and consistent identifiers using pattern mining techniques. Furthermore, according to some embodiments described herein, domain knowledge can be partially captured as a hierarchy of concepts, which can be understood as an ontology of UI elements.

[0007] Embodiments described herein are directed to methods and systems for performing description mining of concepts and for performing re-identification checks of UI elements in a current UI.

[0008] According to one aspect of the disclosure, a computer-implemented method for performing description mining of concepts is provided. The method includes obtaining a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface (UI) element, the UI element corresponding to a type of UI elements represented by the concept; generating, based on the plurality of reference descriptions, at least one of a local description of the concept and a global description of the concept, wherein the local description of the concept includes description components that are specific to a respective UI, and wherein the global description of the concept includes description components that are not specific to a UI; and storing the at least one of the generated local description and global description such that it is associated with a concept identifier corresponding to the concept.

[0009] According to another aspect of the present disclosure, there is provided a computer- implemented method for performing re-identification checking of a user interface (UI) element in a current UI. The method comprises retrieving, for the UI element, at least one of a stored first local description and a stored first global description based on a concept identifier representing the UI element; and querying a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

[0010] According to another aspect of the present disclosure, there is provided a system for performing description mining for a concept. The system comprises an obtaining unit configured to obtain a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface (UI) element, the UI element corresponding to a type of UI element represented by the concept; a generating unit configured to generate at least one of a local description for the concept and a global description for the concept based on the plurality of reference descriptions, wherein the local description for the concept comprises a description component specific to a respective UI, and wherein the global description for the concept comprises a description component not specific to a UI; and a storing unit configured to store the at least one of the generated local description and global description such that it is associated with a concept identifier corresponding to the concept.

[0011] According to another aspect of the present disclosure, there is provided a system for performing re-identification checking of a user interface (UI) element in a current UI. The system comprises a retrieving unit configured to retrieve, for the UI element, at least one of a stored first local description and a stored first global description based on a concept identifier representing the UI element; and a querying unit configured to query a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

[0012] Optional features are set out in the appended dependent claims.

[0013] These and other aspects of the present disclosure will become better understood and appreciated when considered in connection with the following description and accompanying drawings. The following description, while indicating various embodiments of the present disclosure and numerous specific details thereof, is given by way of illustration and not of limitation. Many substitutions, modifications, additions or rearrangements can be made within the scope of the present disclosure, and the present disclosure includes all such substitutions, modifications, additions or rearrangements. BRIEF DESCRIPTION OF DRAWINGS

[0014] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings:

[0015] Figure 1 is a flowchart of a computer-implemented method for performing description mining of a UI concept according to embodiments of the present disclosure;

[0016] Figure 2 is a flowchart of a method for performing re-identification checking of UI elements in a current UI according to embodiments of the present disclosure

[0017] Figure 3 is a block diagram of a system for performing description mining of a concept according to embodiments of the present disclosure;

[0018] Figure 4 is a block diagram of a system for performing re-identification checking of UI elements in a current UI according to embodiments of the present disclosure; and

[0019] Figure 5 is a diagram illustrating exemplary processes for performing description mining of a concept and performing re-identification checking of UI elements in a current UI according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments can be practiced without the specific details. In other instances, well-known materials or methods have not been described in detail in order to avoid obscuring the present disclosure.

[0021] Figure 1 is a flowchart of a computer-implemented method for performing description mining of a UI concept (herein referred to simply as "concept") according to embodiments of the present disclosure.

[0022] The method 100 begins at step S110, at which a plurality of reference descriptions are acquired. Each of the plurality of reference descriptions is associated with a user interface (UI) element corresponding to a type of UI element represented by the concept.

[0023] In the context of the present embodiments, a description (e.g., a reference description, a partial description, and a global description) can represent at least one of a location and a property of a respective UI element or a respective concept. For example, in some embodiments, each of the plurality of reference descriptions acquired at step S110 can be in the form of one of the following: a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) expression, and an image and / or audio description. An image description can be, for example, a computer vision (CV) description.

[0024] As an example, the concept in question can represent an "OK button" type UI element. In this example, the plurality of reference descriptions each can be a description of a different "OK button," e.g., from a plurality of different UIs and / or a plurality of different versions of one UI. For example, the plurality of reference descriptions can be XPath expressions of OK buttons of a plurality of different UIs and / or a plurality of different versions of one specific UI.

[0025] In some embodiments, the plurality of reference descriptions can include at least one of a local history reference description and a non-local reference description. In these embodiments, a local history reference description is specific to a current UI (e.g., the UI being used by the user), while a non-local reference description is not specific to the current UI. For example, a local history reference description can correspond to a UI element that is part of the current UI. For example, a non-local reference description can correspond to a UI element that is not part of the current UI, but rather corresponds to a UI similar to the current UI.

[0026] Then, at step S120, at least one of a local description of the concept and a global description of the concept is generated based on the plurality of reference descriptions obtained at step S110. A local description of the concept includes description components that are specific to a respective UI (e.g., the current UI). A global description of the concept includes description components that are not specific to a UI, and thus the global description can be reused in different UIs (e.g., different applications or web pages). In some cases, at step S120, a plurality of local / global descriptions can be generated corresponding to different parallel ways of identifying the respective concept / UI element. For example, the generated descriptions can include XPath expressions, CSS selectors, CV descriptions, etc. In these examples, even if one form / type of description becomes invalid, another form / type can still be used for the purpose of identifying the UI element. Moreover, in some embodiments, a previous description (i.e., the one that becomes invalid) can be updated by using data of the UI element identified using the latter description (i.e., the one used for the purpose of identifying the UI element).

[0027] In some embodiments, generating a local description at S120 can include updating an existing local description of the concept. Alternatively, or additionally, in some embodiments, generating a global description at step S120 can include updating an existing global description of the concept. These existing descriptions can be stored in a database (e.g., the "UI Concept Descriptions" store shown in FIG. 1). Figure 5

[0028] ​The generation of the local and / or global description can involve performing pattern mining with respect to at least the plurality of reference descriptions. In more detail, in some embodiments, generating a local description of a concept at S120 can comprise determining common description components from a plurality of local historical reference descriptions. Alternatively, or additionally, generating a global description of a concept at S120 can comprise determining common description components from a plurality of non-local reference descriptions. In these embodiments, determining common description components can be performed using a trained machine learning model. For example, determining common description components can be performed using a variety of machine learning techniques, such as a computer vision based neural network, a decision tree using structured information from the DOM tree (e.g. tags, text content), or a version space approach based on features of UI elements.

[0029] Subsequently, at step S130, at least one of the local description and the global description generated at step S120 is stored such that it is associated with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept in a set of concepts. For example, the concept identifier can indicate an order of the type of UI element represented by the concept in an ontology of a plurality of different types of UI elements. The set of concepts can be a concept hierarchy in which concepts are presented based on the type of UI element represented by the concept, for example, an "OK button" and a "login button" would be sub-concepts under a more general concept of a "submit button" falling in the hierarchy. The concept hierarchy can represent domain knowledge of an expert in the field of automation, more specifically, domain knowledge of an expert in the field of robotic process automation.

[0030] In some embodiments, the method 100 can further comprise obtaining a user input accepting or rejecting the at least one of the generated local description and the generated global description. In these embodiments, storing the generated local description and / or the generated global description at step S130 can only be performed upon obtaining a user input accepting the respective generated local description or generated global description.

[0031] As described above with reference to S120, in some embodiments, generating a local description of a concept at S120 can comprise determining common description components from a plurality of local historical reference descriptions. In these embodiments, storing the generated local description at S130 can comprise storing the determined common description components as at least part of the local description.

[0032] Further, as described above with reference to S120, in some embodiments, generating a global description of a concept at S120 can comprise determining common description components from a plurality of non-local reference descriptions. In these embodiments, storing the generated global description at S130 can comprise storing the determined common description components as at least part of the global description.

[0033] Although Figure 1 not shown in the flowchart of FIG. 1 1, in some embodiments, the method 100 can further comprise, for each of the plurality of reference descriptions acquired at step S1 10, acquiring a corresponding UI representation. In these embodiments, generating the local description of the concept or the global description of the concept at step S120 can be further based on the inter-relationships between the reference descriptions and the corresponding UI representations.

[0034] In some embodiments, the UI representation can comprise at least one of a structured representation of the respective UI and an unstructured representation of the respective UI. For example, the structured representation can comprise a Document Object Model (DOM) tree. As another example, the unstructured representation of the UI can comprise a screenshot of the respective UI.

[0035] Further, although Figure 1 not shown in the flowchart of FIG. 1 1, in some embodiments, the method 100 can further comprise, prior to acquiring the representation of the corresponding UI of the respective reference description, generating an updated UI representation by removing information content from the original UI representation. Thus, in these embodiments, the acquired representation can be the updated representation (rather than the original representation). As an example, the original UI representation can comprise a DOM tree, and the information content (which is removed) can comprise text nodes and / or image nodes in the DOM tree. The updated representation can be considered a "privacy-protected" version of the UI representation, as the information content that can raise privacy concerns has been removed from this version.

[0036] In some embodiments, the method 100 can further comprise, prior to generating at least one of the local description of the concept and the global description of the concept at step S120, determining whether a predetermined triggering condition is satisfied. In these embodiments, the generation of at least one of the local description and the global description at step S120 is only performed if it is determined that the predetermined triggering condition is satisfied. The predetermined triggering condition can be associated with at least one of: a predetermined time interval, storage of a new concept instance of the concept, a location (e.g. spatial coordinates, or an identifier such as a CSS selector, or a Win32 object property) of a concept instance comprising a respective UI element and a representation of the respective UI, and an indication of a user input. The new concept instance can refer to a concept instance associated with a UI element that cannot be identified by an existing description stored with respect to a corresponding concept identifier.

[0037] As an example of the predetermined triggering condition, the generation of the local and / or global description can be manually triggered by receiving a user input requesting the execution of the description mining. As another example, the generation of the local and / or global description can be triggered by a predefined instruction to execute the description mining at a specific time interval / frequency (e.g. every Monday).

[0038] WhileFigure 1 not shown, but in some embodiments, the method 100 can further include obtaining a user input corresponding to assigning the concept identifier to a first UI element in the respective UI, and storing a concept instance of the concept identifier. The concept instance can include a representation of the respective UI, a description of the first UI element, and the concept identifier. For example, a concept instance of the "OK button" concept can include a representation (e.g., DOM tree) of the UI containing the OK button, an XPath expression of the OK button, and the concept identifier of the "OK button" (which can be the "OK button" itself or any identifier). Thus, each UI element in a user interface can be considered an instance of a concept in the context of the present disclosure. As mentioned above, the storage of the (new) concept instance can be a predetermined trigger condition for initiating the generation of at least one of the local description and the global description.

[0039] Figure 2 is a flowchart of a method for performing a re-identification check of a UI element in a current UI according to embodiments of the present disclosure. In some embodiments, the UI element can be part of an action of an automation workflow. Further, in some embodiments, performing the re-identification check of the UI element as described below can be part of the execution of the automation workflow.

[0040] The method 200 starts at step S210, where at least one of a stored first local description and a stored first global description is retrieved for a UI element. This retrieval is based on a concept identifier representing the UI element. For example, if the UI element is an "OK button", it can be associated with a concept identifier that identifies the "OK button" as the associated concept. The first local description and / or the first global description can be generated for this concept based on the methods described above with reference to Figure 1

[0041] As mentioned above, in some embodiments, the UI element can be part of an action of an automation workflow. In these embodiments, prior to retrieving at least one of the stored first local description and the stored first global description at step S210, the method can further include obtaining an automation workflow that includes at least an action involving the UI element. In practical implementations, the automation workflow can include multiple actions, each involving multiple UI elements. Obtaining the automation workflow can include receiving a user input corresponding to creating the corresponding action in the automation workflow. For example, the user input can correspond to an operation of creating the corresponding action in the automation workflow.

[0042] Returning to Figure 2 ​At step S220, the stored first partial description and the stored first global description are used to query the representation of the current UI in order to determine whether the UI element can be re-identified using the respective description. For example, the first partial description can be an XPath expression, and at step S220, the DOM tree representing the current UI can be queried to determine whether there is a match for that XPath expression in the DOM tree. If it is determined that there is a match, the re-identification of the UI element is successful. On the other hand, if it is determined that there is no match, the re-identification of the UI element is not successful.

[0043] In the context of the present disclosure, it will be appreciated that the terms "re-identify" and "re-identification" can refer to the process of identifying a UI element from a UI (e.g. by finding a match in a UI representation) or more specifically from a UI representation (e.g. a DOM tree). Thus, it is not a prerequisite that the UI element to be "re-identified" at step S220 has already been "identified". Furthermore, the terms "re-identify" and "re-identification" can also refer to scenarios in which a UI element cannot be identified using a partial / global description and thus "re-identification" is performed using a different partial / global description (e.g. by finding a match in a UI representation). This will be explained in more detail below in relation to scenarios in which a UI element cannot be re-identified using the first partial / global description and a second partial / global description is retrieved for the re-identification check.

[0044] In some embodiments in which both the stored first partial description and the stored first global description are available, the method can comprise retrieving at least the first partial description based on the concept identifier at step S210, and querying the representation of the current UI using the first partial description at step S220. In this way, the method can prioritise the use of a partial description over a global description in the re-identification check, which can provide more reliable results as a partial description is specific to the current UI.

[0045] In some embodiments, the method can further comprise, if it is determined at step S220 that the UI element can be re-identified using the first partial description and / or the first global description, storing the representation of the current UI and the first partial description and / or the first global description such that they are associated with the concept identifier.

[0046] Alternatively, if it is determined at step S220 that the UI element cannot be re-identified using the first local description and / or the first global description, the method can further comprise discarding the first local description and / or the first global description that cannot be used for re-identification. Alternatively or additionally, if it is determined at step S220 that the UI element cannot be re-identified using the first local description and / or the first global description, the method can further comprise retrieving at least one of: a stored second local description and a stored second global description based on the concept identifier, the second local description being different from the first local description and the second global description being different from the second global description; and querying the representation of the current UI using the at least one of the second local description and the second global description to determine whether the UI element can be re-identified using the respective description.

[0047] As yet another alternative, in some embodiments, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, the method can further comprise requesting user input indicative of a UI element in the current UI and / or performing description mining of the concept. The description mining can be performed in accordance with the method as described above with reference to Figure 1 .

[0048] It will be appreciated that the methods shown in Figure 1 and Figure 2 may be combined in certain embodiments to allow both description mining and re-identification checking to be performed. An example process in which description mining and re-identification checking are performed is shown in Figure 5 and will be explained in more detail below with reference to the accompanying drawings.

[0049] Figure 3 is a block diagram of a system for performing description mining of a concept in accordance with embodiments of the present disclosure.

[0050] As shown in Figure 3 , there is provided a system 300 comprising an obtaining unit 310, a generating unit 320 and a storing unit 330.

[0051] The obtaining unit 310 is configured to obtain a plurality of reference descriptions. Each of the plurality of reference descriptions is associated with a UI element corresponding to a type of UI element represented by the concept.

[0052] As described above with reference to Figure 1The explained, descriptive (e.g., reference descriptions, local descriptions, and global descriptions) can indicate at least one of a location and a property of the respective UI element or the respective concept. For example, in some embodiments, each of the plurality of reference descriptions acquired by the acquisition unit 310 can be in the form of one of: a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) expression, and an image and / or audio description. The image description can be, for example, a computer vision (CV) description.

[0053] In some embodiments, the plurality of reference descriptions can include at least one of a local historical reference description and a non-local reference description. In these embodiments, the local historical reference description is specific to the current UI, while the non-local reference description is not specific to the current UI. For example, the local historical reference description can correspond to a UI element that is part of the current UI. For example, the non-local reference description can correspond to a UI element that is not part of the current UI, but rather corresponds to a UI similar to the current UI.

[0054] The generation unit 320 is configured to generate at least one of a local description of a concept and a global description of a concept based on the plurality of reference descriptions acquired by the acquisition unit 310. The local description of a concept includes description components that are specific to a respective user interface (UI). The global description of a concept includes description components that are not specific to a UI, and thus the global description can be reused in different UIs (e.g., different applications or web pages). The generation of the local and / or global descriptions can involve performing pattern mining with respect to at least the plurality of reference descriptions. In some cases, multiple local / global descriptions corresponding to different parallel ways of identifying the respective concept / UI element can be generated by the generation unit 320. Thus, in some embodiments, if one form / type of description becomes invalid, another form / type of description can be used instead to identify the UI element. Furthermore, data of the UI element identified using the other description can be used to update the description that became invalid.

[0055] In some embodiments, the generation unit 320 can be configured to generate the local description by updating an existing local description of the concept. Alternatively or additionally, in some embodiments, the generation unit 320 can be configured to generate the global description by updating an existing global description of the concept. These existing descriptions can be stored in a UI concept description storage (e.g., the "UI Concept Descriptions" storage shown in FIG. 1). Figure 5

[0056] ​The generating unit 320 can be configured to generate a local description of the concept by determining a common description component from a plurality of local historical reference descriptions. Alternatively or additionally, the generating unit 320 can be configured to generate a global description of the concept by determining a common description component from a plurality of non-local reference descriptions. Furthermore, in these embodiments, the generating unit 320 can be configured to determine the common description component using a trained machine learning model. For example, determining the common description component can be performed using a variety of machine learning techniques, such as a computer vision-based neural network, a decision tree using structured information from the DOM tree (e.g., tags, text content), or a version space method based on features of the UI elements.

[0057] The storing unit 330 is configured to store at least one of the generated local description and the generated global description in association with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept in a set of concepts. For example, the concept identifier can indicate an order of a type of UI element represented by the concept in an ontology of a plurality of different types of UI elements. The set of concepts can be a concept hierarchy in which concepts are presented based on a type of UI element represented by the concept, e.g., an “OK button” and a “login button” would be sub-concepts under a more general concept of “submit button” falling in the hierarchy. The concept hierarchy can represent domain knowledge of an expert in the field of automation, more specifically, domain knowledge of an expert in the field of robotic process automation.

[0058] In some embodiments, the obtaining unit 310 can be further configured to obtain a user input accepting or rejecting the at least one of the generated local description and the generated global description. In these embodiments, the storing unit 330 is configured to store the generated local description and / or the generated global description only if the obtaining unit 310 obtains a user input accepting the respective generated local description or generated global description.

[0059] As mentioned above, in some embodiments, the generating unit 320 can be configured to generate a local description of the concept by determining a common description component from a plurality of local historical reference descriptions. In these embodiments, the storing unit 330 can be configured to store the generated local description by storing the determined common description component as at least a part of the local description.

[0060] Furthermore, as mentioned above, in some embodiments, the generating unit 320 can be configured to generate a global description of the concept by determining a common description component from a plurality of non-local reference descriptions. In these embodiments, the storing unit 330 can be configured to store the generated global description by storing the determined common description component as at least a part of the global description.

[0061] In some embodiments, the obtaining unit 310 can be further configured to obtain, for each of the plurality of reference descriptions, a corresponding UI representation. In these embodiments, the generating unit 320 can be configured to generate the local description of the concept or the global description of the concept further based on the inter-relationship between the reference description and the corresponding UI representation.

[0062] In some embodiments, the UI representation can comprise at least one of a structured representation of the respective UI and an unstructured representation of the respective UI. For example, the structured representation can comprise a Document Object Model (DOM) tree. As another example, the unstructured representation of the UI can comprise a screenshot of the respective UI.

[0063] In some embodiments, the generating unit 320 can be configured to generate an updated UI representation by removing information content from an original UI representation. This operation can be performed prior to the obtaining unit 310 obtaining the representation of the corresponding UI for the respective reference description. In these embodiments, the obtained representation can be the updated representation. For example, the original UI representation can comprise a DOM tree, and the information content (which is removed) can comprise text nodes and / or image nodes in the DOM tree. The updated representation can be considered as a “privacy-protected” version of the UI representation, as the information content that can cause privacy issues has been removed from this version.

[0064] Although Figure 3 Although not shown in FIG. 3, in some embodiments, the system 300 can further comprise a determining unit configured to determine whether a predetermined triggering condition is satisfied prior to the generating unit 320 generating at least one of the local description of the concept and the global description of the concept. In these embodiments, the generating unit 320 can be configured such that the generation of at least one of the local description and the global description is performed only if the predetermined triggering condition is determined to be satisfied by the determining unit. The predetermined triggering condition can be associated with at least one of: a predetermined time interval, a storage of a new concept instance of the concept, a location (e.g. spatial coordinates) of the concept instance comprising a respective UI element and a representation of the respective UI, and an indication of a user input. The new concept instance can refer to a concept instance associated with a UI element that cannot be identified by an existing description stored with respect to a corresponding concept identifier.

[0065] As an example of the predetermined triggering condition, the generation of the local and / or global description by the generating unit 320 can be manually triggered by a user input that requests the execution of the description mining. As another example, the generation of the local and / or global description can be triggered by a predefined instruction that executes the description mining at a specific time interval / frequency (e.g. every Monday).

[0066] In some embodiments, the acquisition unit 310 may also be configured to acquire user input corresponding to assigning the concept identifier to a first UI element in the corresponding UI. Furthermore, in these embodiments, the storage unit 330 may also be configured to store a concept instance of the concept identifier, which includes: a representation of the corresponding UI, a description of the first UI element, and the concept identifier. For example, a concept instance of the "OK button" concept may include a representation of the UI containing the OK button (e.g., a DOM tree), an XPath expression for the OK button, and a concept identifier for the "OK button" (which may be the "OK button" itself or any identifier). Therefore, each UI element in the user interface can be considered an instance of a concept within the context of this disclosure. As described above, the storage of (new) concept instances may be a predetermined triggering condition for initiating the generation of at least one of a local description and a global description.

[0067] Although Figure 3 Although not shown in the figure, in some embodiments, the components including the acquisition unit 310, the generation unit 320 and the storage unit 330 may be implemented as a combination of processing circuitry and memory in system 300.

[0068] Figure 4 This is a block diagram of a system for performing a re-identification check on UI elements in a current UI, according to embodiments of the present disclosure. In some embodiments, the UI elements may be part of an automated workflow action.

[0069] like Figure 4 As shown, a system 400 including a retrieval unit 410 and a query unit 420 is provided. The retrieval unit 410 is configured to retrieve at least one of a stored first local description and a stored first global description for a UI element based on a conceptual identifier representing the UI element. This retrieval is based on the conceptual identifier representing the UI element. For example, if the UI element is an "OK button," it can be associated with a conceptual identifier that identifies the "OK button" as an associated concept. The first local description and / or the first global description can be based on the above references. Figure 1 The methods described are conceptually generated, or they may be derived from system 300 by reference. Figure 3 It was generated in the manner described.

[0070] As described above, in some embodiments, UI elements can be part of the actions of an automated workflow. In these embodiments, system 400 may also include an acquisition unit ( Figure 4(Not shown in the image), it is configured to acquire an automated workflow before the retrieval unit 410 retrieves at least one of a stored first local description and a stored first global description, the automated workflow including at least actions involving UI elements. In a practical implementation, the automated workflow may include multiple actions, each involving multiple UI elements. The acquisition unit may be configured to acquire the automated workflow by receiving user input corresponding to an action in the automated workflow. For example, the user input may correspond to an operation that creates a corresponding action in the automated workflow.

[0071] The query unit 420 is configured to query the representation of the current UI using at least one of a stored first local description and a stored first global description to determine whether the UI element can be re-identified using the corresponding description. For example, the first local description may be an XPath expression, and the query unit 420 may be configured to query the DOM tree representing the current UI to determine if a match for that XPath expression exists in the DOM tree. If a match is found, the re-identification of the UI element is successful. Conversely, if no match is found, the re-identification of the UI element is unsuccessful. In some embodiments, the UI element re-identification check performed by the query unit 420 can be considered as part of the execution of an automated workflow.

[0072] As mentioned above Figure 2 As explained, the terms "re-identification" and "re-recognition" can refer to the process of identifying UI elements from the UI, or more specifically from the UI representation, without requiring that the UI elements have been previously "identified." Furthermore, the terms "re-identification" and "re-recognition" can also refer to scenarios where local / global descriptions cannot be used to identify UI elements, and therefore different local / global descriptions are used to perform "re-identification."

[0073] In some embodiments where both the first local description and the first global description stored are available, the retrieval unit 410 may be configured to retrieve at least the first local description based on a concept identifier. Furthermore, the query unit 420 may be configured to use the first local description to query a representation of the current UI. In this way, the system 400 can prioritize the use of the local description over the global description during re-identification checks, which can provide more reliable results because the local description is specific to the current UI.

[0074] In some embodiments, system 400 may further include a storage unit configured to store a representation of the current UI and the first local description and / or the first global description, such that they are associated with a concept identifier, if query unit 420 determines that UI elements can be re-identified using the first local description and / or the first global description.

[0075] Alternatively, the system 400 can further comprise a discarding unit configured to discard the first local description and / or the first global description if it is determined by the querying unit 420 that the UI element cannot be re-identified using the first local description and / or the first global description. Additionally, or alternatively, the retrieving unit 410 can be further configured to retrieve at least one of a stored second local description and a stored second global description based on the concept identifier, the second local description being different from the first local description and the second global description being different from the second global description, if it is determined by the querying unit 420 that the UI element cannot be re-identified using the first local description and / or the first global description. Furthermore, the querying unit 420 can be further configured to query the representation of the current UI using at least one of the second local description and the second global description to determine whether the UI element can be re-identified using the respective description.

[0076] As yet another alternative, in some embodiments, the system 400 can further comprise a requesting unit configured to request a user input indicative of a description mining of the UI element in the current UI and / or of the performed concept if it is determined by the querying unit 420 that the UI element cannot be re-identified using the first local description and / or the first global description. The description mining can be performed according to the method described above with reference to Figure 1 Figure 3 and / or by the system 300 described above with reference to

[0077] Although Figure 4 not shown in Figs. 1 to 3, in some embodiments, the components comprising the retrieving unit 410 and the querying unit 420 can be implemented as a combination of processing circuitry and memory in the system 400.

[0078] It is to be understood that the functionality of the systems 300 and 400 shown in Figs. 1 to 3 can be combined in certain embodiments into a single system to allow such a system to perform both description mining and re-identification checking. Figure 3 Figure 4 It is to be understood that the functionality of the systems 300 and 400 shown in Figs. 1 to 3 can be combined in certain embodiments into a single system to allow such a system to perform both description mining and re-identification checking.

[0079] Figure 5 Fig. 4 is a diagram illustrating an exemplary process for performing a description mining of a concept and performing a re-identification check of a UI element in a current UI according to an embodiment of the present disclosure.

[0080] As Figure 5 shown in Figs. 1 to 3, the process is illustrated by a plurality of steps numbered (i) to (xi) in the diagram for ease of reference. These steps are explained in more detail below with reference to a user, who can be an automation developer using an RPA editor to build a workflow. In the context of the present embodiment, the user is establishing a workflow with respect to an open application UI by using a particular UI element in the application UI. ​​

[0081] (i) The user highlights UI elements of interest. Figure 5 In the example shown, this is the OK button on the application UI.

[0082] (ii) RPA editor or system (e.g., Figure 3 The system (300) captures the highlighted element. In other words, the highlighted element is located within the current application UI.

[0083] (iii) See above for reference Figure 1 As explained, UI elements can be considered instances or examples of concepts. For example, such as... Figure 5 The OK button in the current UI shown can be considered an instance of the "OK button" concept or the "login button" concept, both of which can be considered sub-concepts of the "submit button" concept. Users can create these concepts and place them in appropriate positions within the concept hierarchy before highlighting the corresponding instances on the UI. Alternatively, the existing concept hierarchy can be used directly by the user.

[0084] (iv) The entire UI representation and the positions of highlighted UI elements within the UI, along with a concept identifier (which may also be referred to as a "concept ID"), are stored as instances in the "UI Representation and Highlighted Objects" store. The UI representation can be unstructured (e.g., a screenshot of the UI), or it can be structured (e.g., a DOM tree), or both. The position of an element of interest within the UI representation can be in the form of spatial coordinates or a unique identifier, such as a CSS selector or an XPath expression. If privacy concerns exist, a structured representation of the UI, such as the DOM, can be used, but all informational content (e.g., text and image nodes) is removed. The table below provides an example of the information stored in an instance under the "OK Button" concept. For each application or website, instances can be stored separately for each concept.

[0085]

[0086]

[0087] Table 1 - Examples of stored instances of the concept "OK button"

[0088] (v) In this example, the storage of new concept instances in the "UI representation and highlighted objects" database can be triggered by the UI concept description miner (which may be referred to as the "description miner" in this document).

[0089] (vi) The description miners create and / or update local and global descriptions of the relevant concepts. For example, if the OK button highlighted in step (i) belongs to the SAP website, the description miners can create and / or update a description of the "OK button" concept for that particular website, and a global description that can be used as a default description of the "OK button" concept in other websites and applications. For this particular concept, the global description can be generated using all the stored information across different websites and applications, while the local description can be generated per website or per application. In some embodiments, the description miners can be composed of multiple specialized miners. For example, XPath miners can create and / or update XPath expressions of the corresponding UI elements, while CSS miners can create and / or update CSS selectors. Similarly, computer vision (CV) miners can use screenshots of the UI and locally and globally create CV descriptions of the concepts. This means that the description miners can produce different types of descriptions of the same concept and store them in the "UI concept descriptions" store. The "mined" descriptions can be scrutinized by human users so that they can be accepted or rejected after inspection.

[0090] In many cases, even if the element of interest can be automatically located using existing descriptions, it is still useful to create more efficient and robust (local / global) descriptions over time. For example, XPath expressions can have been sufficient to locate instances of the submit button on a website, but the description miners can use multiple stored UI representations of the same website to produce more robust CSS selectors. Thus, even if the stored XPath expressions become invalid in the future, the mined CSS selectors can be used to self-heal the workflow and re-identify the intended UI elements.

[0091] (vii) Each mined description is paired with a relevant concept identifier (i.e., a concept identifier corresponding to the concept that represents the type of the corresponding UI element), so they can be queried and accessed by the corresponding concept identifier.

[0092] (viii) The user creates workflows by referring to the corresponding elements of interest using the concept identifiers. Thus, the identifiers of the UI elements and the workflow descriptions are decoupled. This allows subsequent modifications of the descriptions (i.e., concepts) of the UI elements without making any changes to the existing workflows.

[0093] (ix) Digital workers can execute workflows, in whole or in part, during the workflow design time to check whether the workflow or parts of the workflow are working properly. In doing so, the concepts are queried to retrieve the descriptions of the corresponding UI elements.

[0094] (x) The digital worker uses the retrieved description (i.e., XPath expression, CSS selector, etc.) to re-identify the element of interest. If the element is correctly re-identified, the UI representation and the automatically re-identified element are stored for future use, e.g., for discovering a better description. If the element cannot be re-identified, the user can be asked to highlight the element again.

[0095] (xi) During normal execution of the workflow, if the UI element cannot be re-identified using the existing description (retrieved based on the concept identifier), a request can be created for the operator to highlight on screen the element that was not re-identified, steps (ix) and (x) can be repeated afterwards, and the execution of the workflow can be resumed if possible.

[0096] It is to be understood that the description is only by way of example; alterations and modifications can be made to the described embodiments without departing from the scope of the disclosure as defined in the claims.

[0097] While for the purpose of illustration the present technology has been described in detail on the basis of currently considered the most practical and preferred implementations, it is to be understood that such details are only for that purpose and that the present technology is not limited to the implementations disclosed, but is intended to cover modifications and equivalent arrangements included within the scope of the appended claims. For example, it is to be understood that the present technology contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

[0098] Throughout this specification, reference has been made to "an embodiment" or "one embodiment" or "an example" meaning that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the disclosure. The appearance of the phrases "in one embodiment" or "in an embodiment" or "an example" or "one example" in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics can be combined in any suitable

[0099] Embodiments according to the present disclosure can be implemented as a device, a method, or a computer program product. For example, in some embodiments, a computer program product can be provided that includes instructions that, when executed by a processor, cause the processor to perform a method as described. Figure 1 and / or Figure 2 the described method. As another example, in some embodiments, a computer-readable storage medium can be provided that includes instructions, which, when executed by a computer, cause the computer to perform as described.Figure 1 and / or Figure 2 The steps of the methods described in the specification can be implemented by modules or systems. Accordingly, embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "module" or "system." Furthermore, embodiments of the present disclosure can take the form of a computer program product on any tangible computer readable medium having computer

[0100] Although described in connection with an exemplary computing system environment, embodiments of the disclosure can be used in numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with aspects of the disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0101] Any combination of one or more computer usable or computer readable medium(s) can be utilized. For example, a computer readable medium can include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or flash memory) device, a portable compact disc read-only memory (CDROM), an optical storage device, and a magnetic storage device. Computer program code embodied in a computer readable medium can be written in any combination of one or more programming languages.

[0102] The flow diagrams and the block diagrams illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that each block of the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by special-purpose hardware-based systems that perform the specified functions or acts or combinations of special-purpose hardware and computer instructions.

[0103] The following is a list of embodiments of the present disclosure:

[0104] 1. A computer-implemented method for performing descriptive mining of a concept, the method comprising:

[0105] obtaining a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface, UI, element, the UI element corresponding to a type of UI element represented by the concept;

[0106] generating, based on the plurality of reference descriptions, at least one of a local description of the concept and a global description of the concept, wherein the local description of the concept comprises a description component that is specific to a respective UI, and wherein the global description of the concept comprises a description component that is not specific to a UI; and

[0107] storing the generated at least one of the local description and the global description such that it is associated with a concept identifier corresponding to the concept.

[0108] 2. The method of embodiment 1, wherein the concept identifier uniquely identifies the concept in a set of concepts, the set of concepts being an ontology of a plurality of different types of UI elements.

[0109] 3. The method of embodiment 2, wherein the plurality of reference descriptions comprises at least one of a local historical reference description and a non-local reference description, wherein a local historical reference description is specific to a current UI, and wherein a non-local reference description is not specific to a current UI.

[0110] 4. The method of any one of the preceding embodiments, wherein generating a local description comprises updating an existing local description of the concept, and / or wherein generating a global description comprises updating an existing global description of the concept.

[0111] 5. The method of any one of the preceding embodiments, wherein a description is indicative of at least one of a position and a property of a respective UI element or a respective concept.

[0112] 6. The method of any one of the preceding embodiments, wherein each of the plurality of reference descriptions is in the form of one of: a Cascading Style Sheet, CSS, selector, an eXtensible Markup Language, XML, path language, XPath, expression, and an image and / or audio description.

[0113] 7. The method of any one of the preceding embodiments, further comprising, for each of the plurality of reference descriptions, obtaining a corresponding UI representation,

[0114] wherein generating the local description of the concept or the global description of the concept is further based on an interrelation between the reference description and the corresponding UI representation.

[0115] 8. The method of embodiment 7, wherein the UI representation comprises at least one of a structured representation of the respective UI and an unstructured representation of the respective UI.

[0116] 9. The method of embodiment 8, wherein the structured representation of the UI comprises a Document Object Model, DOM, tree, and / or wherein the unstructured representation of the UI comprises a screenshot of the respective UI.

[0117] 10. The method of any one of embodiments 7 to 9, further comprising, prior to obtaining the respective reference description’s representation of the corresponding UI,

[0118] generating an updated UI representation by removing information content from the original UI representation,

[0119] wherein the obtained representation is the updated representation.

[0120] 11. The method of embodiment 10, wherein the original UI representation comprises a DOM tree, and wherein the information content comprises text nodes and / or image nodes in the DOM tree.

[0121] 12. The method of any one of the preceding embodiments, wherein generating the local description of the concept comprises determining a common description component from a plurality of local historical reference descriptions, and

[0122] wherein storing the generated local description comprises storing the determined common description component as at least part of the local description.

[0123] 13. The method of any one of the preceding embodiments, wherein generating the global description of the concept comprises determining a common description component from a plurality of non-local reference descriptions, and

[0124] wherein storing the generated global description comprises storing the determined common description component as at least part of the global description.

[0125] 14. The method of embodiment 12 or embodiment 13, wherein determining the common description component is performed using a trained machine learning model.

[0126] 15. The method of any one of the preceding embodiments, further comprising, prior to generating at least one of the local description and the global description of the concept:

[0127] determining whether a predetermined trigger condition is met,

[0128] wherein the generating of at least one of the local description and the global description is only performed if it is determined that the predetermined trigger condition is met.

[0129] 16. The method according to embodiment 15, wherein the predetermined trigger condition is associated with at least one of:

[0130] - a predetermined time interval;

[0131] - storage of a new concept instance of the concept, wherein the concept instance comprises a position of a respective UI element and a representation of a respective UI; and

[0132] - instructing a user input.

[0133] 17. The method according to embodiment 15 or embodiment 16, further comprising: obtaining a user input, wherein the user input corresponds to assigning the concept identifier to a first UI element in a respective UI,

[0134] storing a concept instance of the concept identifier, the concept instance comprising: a representation of the respective UI, a description of the first UI element and the concept identifier;

[0135] wherein the storage of the concept instance is the predetermined trigger condition.

[0136] 18. The method according to any one of the preceding embodiments, further comprising: obtaining a user input accepting or rejecting at least one of a generated local description and a generated global description,

[0137] wherein the storing of the generated local description and / or the generated global description is only performed upon obtaining the user input accepting the respective generated local description or the generated global description.

[0138] 19. A computer-implemented method for performing a re-identification check for a UI element in a current user interface, UI, the method comprising:

[0139] retrieving, for the UI element, at least one of a stored first local description and a stored first global description based on a concept identifier representing the UI element; and

[0140] querying a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

[0141] 20. The method according to embodiment 19, wherein the UI element is part of an action of an automated workflow, and wherein prior to retrieving the at least one of the stored first local description and the stored first global description, the method further comprises obtaining an automated workflow, wherein the automated workflow comprises at least an action involving the UI element.

[0142] 21. The method of example 20, wherein obtaining the automation workflow comprises receiving user input corresponding to actions in the automation workflow.

[0143] 22. The method of any one of examples 19-21, wherein performing the re- identification check of the UI element is part of execution of the obtained automation workflow.

[0144] 23. The method of any one of examples 19-22, wherein both the stored first partial description and the stored first global description are available, and wherein the method comprises retrieving at least the first partial description based on the concept identifier, and querying the representation of the current UI using the first partial description.

[0145] 24. The method of any one of examples 19-23, further comprising, if it is determined that the UI element can be re-identified using the first partial description and / or the first global description, storing the representation of the current UI and the first partial description and / or the first global description such that they are associated with the concept identifier.

[0146] 25. The method of any one of examples 19-23, further comprising, if it is determined that the UI element cannot be re-identified using the first partial description and / or the first global description:

[0147] retrieving at least one of: a stored second partial description and a stored second global description based on the concept identifier, wherein the second partial description is different from the first partial description, and wherein the second global description is different from the first global description; and

[0148] querying the representation of the current UI using the at least one of the second partial description and the second global description to determine whether the UI element can be re-identified using the respective description.

[0149] 26. The method of any one of examples 19-23, further comprising, if it is determined that the UI element cannot be re-identified using the first partial description and / or the first global description, requesting user input indicative of the UI element in the current UI.

[0150] 27. The method of any one of examples 19-23, further comprising, if it is determined that the UI element cannot be re-identified using the first partial description and / or the first global description, performing description mining of the concept according to any one of examples 1-18.

[0151] 28. The method of any one of embodiments 19 to 27, wherein the first local description and / or the first global description is generated according to any one of embodiments 1 to 18.

[0152] 29. A system for performing description mining of a concept, the system comprising:

[0153] an acquisition unit configured to acquire a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface, UI, element, the UI element corresponding to a type of UI element represented by the concept;

[0154] a generation unit configured to generate, based on the plurality of reference descriptions, at least one of a local description of the concept and a global description of the concept, wherein the local description of the concept comprises a description component specific to a respective UI, and wherein the global description of the concept comprises a description component that is not specific to a UI; and

[0155] a storage unit configured to store the generated at least one of a local description and a global description in association with a concept identifier corresponding to the concept.

[0156] 32. A system for performing re-identification checking of a UI element in a current user interface, UI, the system comprising:

[0157] a retrieval unit configured to retrieve, based on a concept identifier representing the UI element, at least one of a stored first local description and a stored first global description for the UI element; and

[0158] a querying unit configured to query a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

[0159] 33. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method of any one of embodiments 1 to 28.

Claims

1. A computer-implemented method (100) for performing descriptive mining of a concept, the method comprising: obtaining (SI 10) a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface, UI, element, the UI element corresponding to a type of UI element represented by the concept; generating (S120) at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept comprises description components specific to a respective UI, and wherein the global description of the concept comprises description components not specific to a UI; and storing (S130) the at least one of the generated local description and the global description such that it is associated with a concept identifier corresponding to the concept.

2. The method (100) of claim 1, wherein the concept identifier uniquely identifies the concept in a set of concepts, the set of concepts being an ontology of a plurality of different types of UI elements.

3. The method (100) according to any one of the preceding claims, wherein each of the plurality of reference descriptions is in the form of one of: a Cascading Style Sheet, CSS, selector, an eXtensible Markup Language, XML, path language, XPath, expression, and an image and / or audio description.

4. The method (100) of claim 1, further comprising: for each of the plurality of reference descriptions, obtaining a corresponding UI representation, wherein generating the local description of the concept or the global description of the concept is further based on an interrelation between the reference description and the corresponding UI representation.

5. The method (100) of claim 4, further comprising, prior to obtaining the representation of the corresponding UI of the respective reference description: generating an updated UI representation by removing informational content from an original UI representation, wherein the obtained representation is the updated representation.

6. The method (100) of claim 1, wherein generating the local description of the concept comprises determining common description components from a plurality of local historical reference descriptions, and / or wherein generating the global description of the concept comprises determining common description components from a plurality of non-local reference descriptions.

7. The method (100) of claim 1, further comprising, prior to generating at least one of the local description of the concept and the global description of the concept: determining whether a predetermined trigger condition is met, wherein the generating of at least one of the local description and the global description is only performed if it is determined that the predetermined trigger condition is met.

8. The method (100) of claim 7, further comprising: obtaining a user input, wherein the user input corresponds to assigning the concept identifier to a first UI element in a respective UI, storing a concept instance of the concept identifier, the concept instance comprising: a representation of the respective UI, a description of the first UI element, and the concept identifier; wherein the storing of the concept instance is the predetermined trigger condition.

9. The method (100) of claim 1, further comprising: obtaining a user input accepting or rejecting the at least one of the generated local description and the generated global description, wherein the storing of the generated local description and / or the generated global description is only performed if the user input accepting the respective generated local description or generated global description is obtained.

10. A computer-implemented method (200) for performing re-identification checking of a UI element in a current user interface, UI, the method comprising: retrieving (S210), for the UI element, at least one of a stored first local description and a stored first global description based on a concept identifier representing the UI element, wherein each of the at least one of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a UI element corresponding to a type of UI element corresponding to the concept identifier, wherein the stored first local description comprises description components specific to a respective UI, and wherein the stored first global description comprises description components not specific to a UI; and querying (S220) a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

11. The method (200) of claim 10, further comprising, if it is determined that the UI element can be re-identified using the first local description and / or the first global description, storing the representation of the current UI and the first local description and / or the first global description such that they are associated with the concept identifier.

12. The method (200) of claim 10, further comprising, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description: retrieving at least one of a stored second local description and a stored second global description based on the concept identifier, wherein the second local description is different from the first local description, and wherein the second global description is different from the second global description; and querying the representation of the current UI using the at least one of the second local description and the second global description to determine whether the UI element can be re-identified using the respective description.

13. The method (200) of claim 10, further comprising, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, performing at least one of: requesting a user input indicative of the UI element in the current UI; and performing description mining of the concept according to any of claims 1 to 9.

14. A system (300) for performing description mining of a concept, the system comprising: an acquisition unit (310) configured to acquire a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface, UI, element, the UI element corresponding to a type of UI element represented by the concept; and a description mining unit (320) configured to perform description mining of the concept based on the plurality of reference descriptions. a generating unit (320) configured to generate at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, wherein the local description of the concept comprises description components specific to a respective UI, and wherein the global description of the concept comprises description components not specific to a UI; and a storing unit (330) configured to store the at least one of the generated local description and global description such that it is associated with a concept identifier corresponding to the concept.

15. A system (400) for performing a re-identification check for a UI element in a current user interface, UI, the system comprising: a retrieving unit (410) configured to retrieve at least one of a stored first local description and a stored first global description for the UI element based on a concept identifier representing the UI element, wherein each of the at least one of the stored first local description and the stored first global description is generated based on a plurality of reference descriptions, each of the plurality of reference descriptions being associated with a UI element corresponding to a type of UI element corresponding to the concept identifier, wherein the stored first local description comprises description components specific to a respective UI, and wherein the stored first global description comprises description components not specific to a UI; and a querying unit (420) configured to query a representation of the current UI using the at least one of the stored first local description and the stored first global description to determine whether the UI element can be re-identified using the respective description.

16. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 13.

Images