Method and system for re-identification in a user interface - Patents.com

Abstract

A method (100) for performing description mining of a concept is provided, comprising the steps of: obtaining (S110) a plurality of basis descriptions, each of which is associated with a user interface, UI, element corresponding to a type of UI element represented by the concept; generating (S120) at least one of a local description of the concept or a global description of the concept based on the plurality of basis descriptions, the local description of the concept comprising description components specific to a respective UI, and the global description of the concept comprising description components not specific to a UI; storing (S130) at least one of the generated local description and the global description in association with a concept identifier corresponding to the concept.

Description

[Technical field]

[0001] The present disclosure relates to methods and systems for re-identification in a user interface. More specifically, the present disclosure relates to a system and computer-implemented method for performing description mining of concepts and performing re-identification checks of user interface elements in a current user interface. [Background technology]

[0002] Human-guided computer processes are ubiquitous across many fields of technology and endeavor. Modern graphical user interfaces (GUIs) have proven invaluable in enabling human operators to use computer systems to perform often complex data processing and / or system control tasks. However, while GUIs often allow human operators to quickly become familiar with performing new tasks, there are often high barriers to any further automation of the task.

[0003] Traditional workflow automation aims to take tasks that are typically performed by an operator using a GUI and automate them so that a computer system can perform the same tasks without significant redesign of the underlying software used to perform the tasks. In the early days, this required exposing an Application Programming Interface (API) for the software so that scripts could be manually manipulated to perform the required functions of the software to perform the required tasks.

[0004] Robotic Process Automation (RPA) is an emerging form of business process automation technology that utilizes software robots or artificial intelligence (AI) workers. In traditional workflow automation tools, software developers generate lists of actions to automate tasks using internal application program interfaces. In contrast, RPA systems develop action lists by watching a user perform tasks in the graphical user interface of an application, and then perform the automation by repeating those tasks directly on the graphical user interface. The RPA robot then generates the appropriate input commands for the GUI to cause a given process to be executed by a computer system. This enables the automation of the process, turning an attended process into an unattended process. The advantages of such an approach are numerous and include greater scalability, which allows multiple RPA robots to perform the same tasks across multiple computer systems, and strong repeatability, such that the possibility of human error in a given process is reduced or eliminated. Summary of the Invention

[0005] Existing approaches for automation typically rely on extracting unique identifiers for user interface (UI) elements and building automation based on these identifiers. Examples of such identifiers for web-based applications are Extensible Markup Language Path Language (XPath) expressions and Cascading Style Sheets (CSS) selectors, which may also be combined with various properties of UI elements. Unique identifiers are typically defined manually by RPA developers during application modeling or workflow development. However, these unique identifiers can easily be affected by UI changes. For example, XPath expressions can be invalidated by the addition of a new ancestor (e.g., a container invisible to the user) or by changes in the identifiers of existing ancestor elements.

[0006] Some identifiers of UI elements are more robust to changes in the UI, and domain experts may use such identifiers to identify UI elements. While doing so, experts can use trial and error in addition to domain knowledge. The method and system proposed herein are inspired by this. The method and system are based on the idea of ​​maintaining a representation of the UI over time and leveraging different versions of the UI to automatically extract robust and consistent identifiers by pattern mining techniques. Also, according to some embodiments described herein, the domain knowledge is partially captured as a concept hierarchy that can be understood as an ontology of UI elements.

[0007] The embodiments described herein are directed to methods and systems for performing concept description mining and performing re-identification checks of UI elements within a current UI.

[0008] According to one aspect of the present disclosure, a computer-implemented method for performing description mining of a concept is provided, the method including: obtaining a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; generating at least one of a local description for the concept or a global description for the concept based on the plurality of criteria descriptions, the local description for the concept comprising respective UI-specific description components and the global description for the concept comprising non-UI-specific description components; storing at least one of the generated local description or global description, thereby associating it with a concept identifier corresponding to the concept.

[0009] According to another aspect of the present disclosure, a computer-implemented method is provided for performing a re-identification check of a user interface (UI) element in a current UI, the method including: retrieving, for a UI element, at least one of a stored first local description or a stored first global description based on a concept identifier representing the UI element; determining whether the respective descriptions can be used to re-identify the UI element using at least one of the stored first local description and the stored first global description by querying a representation of the current UI.

[0010] According to another aspect of the present disclosure, a system for performing description mining of a concept is provided, the system comprising: an acquisition unit configured to acquire a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; a generation unit configured to generate at least one of a local description of the concept or a global description of the concept based on the plurality of criteria descriptions, the local description of the concept comprising respective UI-specific description components and the global description of the concept comprising non-UI-specific description components; and a storage unit configured to store at least one of the generated local description or global description to be associated with a concept identifier corresponding to the concept.

[0011] According to another aspect of the present disclosure, a system for performing a re-identification check of a user interface (UI) element in a current UI is provided, the system comprising: an acquisition unit configured to retrieve at least one of a stored first local description or a stored first global description for the UI element based on a concept identifier representing the UI element; a query unit configured to query a representation of the current UI using the at least one stored first local description and the stored first global description to determine whether the respective descriptions can be used to re-identify the UI element.

[0012] Optional features are set out in the accompanying dependent claims.

[0013] These and other aspects of the present disclosure will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. The following description sets forth various embodiments of the present disclosure and numerous specific details thereof, which are given by way of illustration and not by way of limitation. Many substitutions, modifications, additions, or rearrangements may be made within the scope of the present disclosure, and the present disclosure includes all such substitutions, modifications, additions, or rearrangements. [Brief description of the drawings]

[0014] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 is a flowchart of a computer-implemented method for performing description mining of UI concepts according to an embodiment of the present disclosure; [Diagram 2] 1 is a flowchart of a method for performing a re-identification check for a UI element in a current UI according to an embodiment of the present disclosure. [Diagram 3] 1 is a block diagram of a system for performing concept description mining according to an embodiment of the present disclosure; [Figure 4] FIG. 1 is a block diagram of a system for performing a re-identification check of a UI element in a current UI, according to one embodiment of the present disclosure; [Diagram 5] FIG. 1 illustrates an exemplary process for performing concept description mining and performing a re-identification check of UI elements in a current UI, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that the specific details are not necessary to practice the embodiments. In other instances, well-known materials or methods have not been described in detail to avoid obscuring the present disclosure.

[0016] FIG. 1 is a flowchart of a computer-implemented method for performing description mining of UI concepts (referred to herein simply as "concepts") according to one embodiment of the present disclosure.

[0017] The method 100 begins at step S110 where a plurality of criteria descriptions are obtained, each of the plurality of criteria descriptions being associated with a user interface (UI) element that corresponds to a type of UI element represented by the concept.

[0018] In the context of this embodiment, the descriptions (such as the reference description, the local description, and the global description) may indicate at least one of the location and attributes of the respective UI elements or respective concepts. For example, in some embodiments, each of the multiple reference descriptions obtained in step S110 may be in the form of one of a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) expression, an image, and / or an audio. The image may be, for example, a computer vision (CV).

[0019] As an example, a concept may represent a UI element of type "OK button." In this example, the multiple criteria descriptions may each be descriptions for a different "OK button," e.g., different UIs and / or different versions of a UI. For example, the multiple criteria descriptions may be XPath expressions of OK buttons for different UIs and / or different versions of a particular UI.

[0020] In some embodiments, the multiple reference descriptions may comprise at least one of a local historical reference description and a non-local reference description. In these embodiments, the local historical reference description is specific to a current UI (e.g., a UI being used by a user) and the non-local reference description is not specific to the current UI. For example, the local historical reference description corresponds to a UI element that is part of the current UI. For example, the non-local reference description corresponds to a UI element that is not part of the current UI but is similar to the current UI.

[0021] Next, in step S120, a local description for the concept and / or a global description for the concept are generated based on the multiple reference descriptions obtained in step S110. The local description of the concept includes description components specific to the respective UI, e.g., the current UI. The global description of the concept includes components that are not specific to the UI, and thus the global description can be reused in different UIs (e.g., different applications or web pages). In some cases, multiple local / global descriptions corresponding to different parallel ways to identify respective concepts / UI elements can be generated in step S120. For example, the generated descriptions can include XPath expressions, CSS selectors, CV descriptions, etc. In these examples, even if one form / type of description becomes invalid, another form / type can be used for identifying the UI element. Furthermore, in some embodiments, the former description (i.e., the invalidated description) can be updated using data from the UI element identified using the latter (i.e., the description used to identify the UI element).

[0022] In some embodiments, generating the local description in S120 may include updating an existing local description of the concept. Alternatively or additionally, in some embodiments, generating the global description in step S120 may include updating an existing global description of the concept. These existing descriptions may be stored in a database (e.g., the "UI Concept Description" storage shown in FIG. 5).

[0023] The generation of the local and / or global description may include performing pattern mining on at least the plurality of reference descriptions. More specifically, in some embodiments, the generation of the local description of the concept in S120 may include determining a common description component from the plurality of local historical reference descriptions. Alternatively or additionally, the generation of the global description of the concept in S120 may include determining a common description component from the plurality of non-local reference descriptions. In these embodiments, the determination of the common description component may be performed using a trained machine learning model. For example, the determination of the common description component may be performed using several machine learning techniques, such as a neural network based on computer vision, a decision tree using structured information from a DOM tree (e.g., tags, text content), or a version space method based on features of UI elements.

[0024] Subsequently, in step S130, at least one of the local description and the global description generated in step S120 is stored so as to be associated with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept in a concept collection. For example, the concept identifier can indicate an order of the types of UI elements represented by the concept in an ontology of multiple different types of UI elements. The concept collection can be a concept hierarchy in which concepts are presented based on the types of UI elements represented by the concept. For example, "OK button" and "Login button" are sub-concepts that fall under the more general concept of "Submit button" in the hierarchy. The concept hierarchy can represent the domain knowledge of experts in the field of automation, and more specifically, robotic process automation.

[0025] In some embodiments, method 100 may further include obtaining user input accepting or rejecting at least one of the generated local description and the generated global description. In these embodiments, storing the generated local description and / or the generated global description in step S130 may be performed only upon obtaining user input accepting the respective generated local description or the generated global description.

[0026] As described above with respect to S120, in some embodiments, generating the local description of the concept in S120 can include determining a common description component from the multiple local historical reference descriptions. In these embodiments, storing the generated local description in S130 can include storing the determined common description component as at least a portion of the local description.

[0027] Also, as described above with respect to S120, in some embodiments, generating a global description for the concept in S120 can include determining a common description component from the multiple non-local reference descriptions. In these embodiments, storing the generated global description in S130 can include storing the determined common description component as at least a portion of the global description.

[0028] 1, in some embodiments, method 100 may further include obtaining a corresponding UI representation for each of the multiple criteria descriptions obtained in step S110. In these embodiments, generating the local description of the concept or the global description of the concept in step S120 may further be based on interrelationships between the criteria descriptions and the corresponding UI representations.

[0029] The UI representations in some embodiments may comprise at least one of a structured representation of the respective UI and an unstructured representation of the respective UI. For example, the structured representation may include a Document Object Model (DOM) tree. As another example, the unstructured representation of the UI may comprise a screenshot of the respective UI.

[0030] Additionally, although not shown in the flowchart of FIG. 1 , in some embodiments, the method 100 may further include generating an updated UI description by removing information content from the original UI description before obtaining a corresponding UI description for each reference description. Thus, in these embodiments, the obtained representation may be the updated representation (instead of the original representation). As an example, the original UI representation may comprise a DOM tree, and the information content (that is removed) may comprise text nodes and / or image nodes in the DOM tree. The updated representation may be considered a “privacy-assured” version of the UI representation, since information content that may raise privacy concerns has been removed from this version.

[0031] In some embodiments, the method 100 may further comprise determining whether a predefined trigger condition is met before generating at least one of the local description and the global description for the concept in step S120. In these embodiments, generating at least one of the local description and the global description in step S120 may be performed only if it is determined that the predefined trigger condition is met. The predefined trigger condition may be associated with at least one of a predefined time interval, storage of a new concept instance of the concept, a position of the respective UI element (e.g., spatial coordinates, or an identifier such as a CSS selector, or a Win32 object attribute), a concept instance including a representation of the respective UI, and an indication user input. The new concept instance may refer to a concept instance associated with a UI element that cannot be identified by an existing description stored for the corresponding concept identifier.

[0032] As one example of a predefined trigger condition, the step of generating local and / or global descriptions may be triggered manually by receiving a user input requesting that description mining be performed. As another example, the step of generating local and / or global descriptions may be triggered by a predefined instruction to perform description mining at a particular interval / frequency, e.g., every Monday.

[0033] Although not shown in FIG. 1, in some embodiments, the method 100 may further include: obtaining a user input, the user input corresponding to assigning a concept identifier to a first UI element in a respective UI; storing a concept instance of the concept identifier. The concept instance may comprise a representation of the respective UI, a description of the first UI element, and a concept identifier. For example, a concept example of an "OK button" concept may include a representation of a UI including an OK button (e.g., a DOM tree), an XPath representation of the OK button, and a concept identifier of the "OK button" (which may be the "OK button" itself or any identifier). Thus, each UI element in a user interface may be considered as an instance of a concept in the context of the present disclosure. As mentioned above, storing a (new) concept instance may be a predefined trigger condition for initiating generation of at least one of a local description and a global description.

[0034] 2 is a flowchart of a method for performing a re-identification check of a UI element in a current UI, according to one embodiment of the present disclosure. In some embodiments, the UI element may be part of an action of an automated workflow. Also, in some embodiments, performing a re-identification check of a UI element as follows may be part of the execution of an automated workflow.

[0035] Method 200 begins at step S210, where at least one of a stored first local description and a stored first global description is obtained for a UI element. The obtaining is based on a concept identifier representing the UI element. For example, if the UI element is an "OK button", it may be associated with a concept identifier that identifies "OK button" as an associated concept. The first local description and / or the first global description may have been generated for the concept based on the method described above with reference to FIG. 1.

[0036] As mentioned above, in some embodiments, the UI element may be part of an action of an automated workflow. In these embodiments, before retrieving at least one of the stored first local description and the stored first global description in step S210, the method may further include retrieving an automated workflow. The automated workflow includes at least an action related to the UI element. In a practical implementation, the automated workflow may include multiple actions, each action including multiple UI elements. The retrieving the automated workflow may include receiving a user input corresponding to an action in the automated workflow. For example, the user input corresponds to an operation for creating a corresponding action in the automated workflow.

[0037] Returning to FIG. 2, in step S220, the representation of the current UI is queried using at least one of the stored first local description and the stored first global description to determine whether each description can be used to re-identify the UI element. For example, the first local description may be an XPath description, and in step S220, the DOM tree representing the current UI may be queried to determine whether there is a match to this XPath description in the DOM tree. If it is determined that there is a match, the re-identification of the UI element is successful. On the other hand, if it is determined that there is no match, the re-identification of the UI element is not successful.

[0038] It should be understood that in the context of this disclosure, the terms "re-identification" and "re-identifying" may refer to the process of identifying a UI element from a UI (e.g., by finding a match in a UI representation) or, more specifically, from a UI representation (e.g., a DOM tree). Thus, it is not a prerequisite that a UI element has been previously "identified" in order for the UI element to be "re-identified" in step S220. Furthermore, the terms "re-identification" and "re-identifying" may also refer to a scenario in which a UI element cannot be identified using a local / global description and thus a different local / global description is used to perform the "re-identification" (e.g., by finding a match in a UI representation). This will be explained in more detail below with respect to a scenario in which a first local / global description cannot be used to re-identify a UI element and a second local / global description is retrieved for the re-identification check.

[0039] In some embodiments in which both a stored first local description and a stored first global description are available, the method may include retrieving at least the first local description based on the concept identifier in step S210, and querying a presentation of the current UI using the first local description in step S220. In this manner, the method may prioritize use of the local description over the global description in re-identification checks, which may provide more reliable results because the local description is specific to the current UI.

[0040] In some embodiments, the method may further include storing, in step S220, the representation of the current UI and the first local description and / or the first global description to be associated with the concept identifier if it is determined that the first local description and / or the first global description can be used to re-identify the UI element.

[0041] Alternatively, if it is determined in step S220 that the UI element cannot be re-identified using the first local description and / or the first global description, the method may further include discarding the first local description and / or the first global description that cannot be used for re-identification. Alternatively or additionally, if it is determined in step S220 that the UI element cannot be re-identified using the first local description and / or the first global description, the method may further include: retrieving at least one of the stored second local description and the stored second global description based on the concept identifier, where the second local description is different from the first local description and the second global description is different from the second global description; querying the representation of the current UI using at least one of the second local description and the second global description to determine whether the UI element can be re-identified using the respective description.

[0042] As yet another alternative, in some embodiments, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, the method may further comprise requesting user input to indicate the UI element in the current UI and / or performing description mining of concepts. Description mining may be performed according to the method described above with reference to FIG.

[0043] It will be appreciated that in some embodiments of the present disclosure, the methods shown in Figures 1 and 2 may be combined in certain embodiments to allow both description mining and re-identification checks to be performed. An exemplary process in which both description mining and re-identification checks are performed is shown in Figure 5 and will be described in more detail with reference to the following figures.

[0044] FIG. 3 is a block diagram of a system for performing concept description mining according to an embodiment of the present disclosure.

[0045] As shown in FIG. 3, a system 300 is provided, which includes an acquisition unit 310, a generation unit 320, and a storage unit 330.

[0046] The obtaining unit 310 is configured to obtain a plurality of criteria descriptions, each of which is associated with a UI element corresponding to a type of the UI element represented by the concept.

[0047] As described above with reference to FIG. 1, the descriptions (such as the reference description, the local description, and the global description) may indicate at least one of the location and attributes of the respective UI elements or respective concepts. For example, in some embodiments, each of the multiple reference descriptions acquired by the acquisition unit 310 may be in the form of one of a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) expression, an image and / or an audio description. The image description may be, for example, a Computer Vision (CV) description.

[0048] In some embodiments, the multiple reference descriptions may include at least one of a local historical reference description and a non-local reference description. In these embodiments, the local historical reference description is specific to the current UI and the non-local reference description is not specific to the current UI. For example, the local historical reference description may correspond to a UI element that is part of the current UI. For example, the non-local reference description may correspond to a UI element that is similar to the current UI but is not part of the current UI.

[0049] The generating unit 320 generates at least one of a local description of the concept and a global description of the concept based on the multiple criteria descriptions obtained by the obtaining unit 310. The local description of the concept includes description components specific to a respective user interface (UI). The global description of the concept includes description components that are not specific to a UI, and thus the global description can be reused in different UIs (e.g., different applications or web pages). The generation of the local and / or global description may involve performing pattern mining on at least the multiple criteria descriptions. In some cases, multiple local / global descriptions corresponding to different parallel ways to identify respective concepts / UI elements may be generated by the generating unit 320. Thus, in some embodiments, if one form / type of description becomes invalid, another form / type of description may be used instead to identify the UI element. Also, the invalid description may be updated with data from UI elements identified using the other description.

[0050] In some embodiments, the generating unit 320 may be configured to generate the local description by updating an existing local description of the concept. Alternatively or additionally, in some embodiments, the generating unit 320 may be configured to generate the global description by updating an existing global description for the concept. These existing descriptions may be stored in a UI concept description store (e.g., the "UI concept description" store shown in FIG. 5).

[0051] The generating unit 320 may be configured to generate a local description for a concept by determining a common description component from a plurality of local historical reference descriptions. Alternatively or additionally, the generating unit 320 may be configured to generate a global description for a concept by determining a common description component from a plurality of non-local reference descriptions. Furthermore, in these embodiments, the generating unit 320 may be configured to determine the common description component using a trained machine learning model. For example, determining the common description component may be performed using several machine learning techniques, such as a neural network based on computer vision, a decision tree using structured information from a DOM tree (e.g., tags, text content), or a version space method based on features of UI elements.

[0052] The storage unit 330 is configured to store at least one of the generated local and global descriptions in association with a concept identifier corresponding to the concept. In some embodiments, the concept identifier can uniquely identify a concept in a concept collection. For example, the concept identifier can indicate an order of types of UI elements represented by the concept in an ontology of multiple different types of UI elements. The concept collection can be a concept hierarchy in which concepts are presented based on the types of UI elements represented by the concept. For example, "OK button" and "Login button" are subconcepts that fall under the more general concept of "Submit button" in the hierarchy. The concept hierarchy can represent the domain knowledge of experts in the field of automation, and more specifically, robotic process automation.

[0053] In some embodiments, the obtaining unit 310 may be further configured to obtain a user input accepting or rejecting at least one of the generated local description and the generated global description. In these embodiments, the storage unit 330 is configured to store only the generated local description and / or the generated global description when the generating unit 310 obtains a user input accepting the respective generated local description or the generated global description.

[0054] As mentioned above, in some embodiments, the generating unit 320 may be configured to generate a local description of a concept by determining a common description component from a plurality of local historical reference descriptions. In these embodiments, the storage unit 330 may be configured to store the generated local description by storing the determined common description component as at least a portion of the local description.

[0055] Also, as mentioned above, in some embodiments, the generating unit 320 may be configured to generate a global description of a concept by determining a common description component from a plurality of non-local reference descriptions. In these embodiments, the storage unit 330 may be configured to store the generated global description by storing the determined common description component as at least a portion of the global description.

[0056] In some embodiments, the obtaining unit 310 may be further configured to obtain, for each of the plurality of criteria descriptions, a corresponding UI representation. In these embodiments, the generating unit 320 may be configured to generate a local description of the concept or a global description of the concept further based on the interrelationship between the criteria descriptions and the corresponding UI representation.

[0057] The UI representations in some embodiments may comprise at least one of a structured representation of the respective UI and an unstructured representation of the respective UI. For example, the structured representation may include a Document Object Model (DOM) tree. As another example, the unstructured representation of the UI may comprise a screenshot of the respective UI.

[0058] In some embodiments, the generating unit 320 may be configured to generate the updated UI representation by removing information content from the original UI representation. This operation may be performed before the obtaining unit 310 obtains a representation of the corresponding UI for each reference description. In these embodiments, the obtained representation may be the updated representation. For example, the original UI representation may include a DOM tree, and the information content (that is removed) may include text nodes and / or image nodes in the DOM tree. The updated representation may be considered a "privacy-guaranteed" version of the UI representation because information content that may raise privacy concerns has been removed from this version.

[0059] Although not shown in FIG. 3, in some embodiments, the system 300 may further comprise a determination unit configured to determine whether a predefined trigger condition is satisfied before the generation unit 320 generates at least one of the local description and the global description of the concept. In these embodiments, the generation unit 320 may be configured such that the generation of at least one of the local description and the global description is performed only if the predefined trigger condition is determined by the determination unit to be satisfied. The predefined trigger condition may be associated with at least one of a predefined time interval, storage of a new concept instance of the concept, the concept instance including the position (e.g., spatial coordinates) and the respective UI representation of the UI element, and an instruction user input. The new concept instance may refer to a concept instance associated with a UI element that cannot be identified by an existing description stored for the corresponding concept identifier.

[0060] As one example of a predefined trigger condition, the generation of local and / or global descriptions by the generation unit 3120 may be triggered manually by receiving a user input requesting that description mining be performed. As another example, the generation of local and / or global descriptions may be triggered by a predefined instruction to perform description mining at a particular interval / frequency, for example every Monday.

[0061] In some embodiments, the acquiring unit 310 may be further configured to acquire a user input. The user input corresponds to the assignment of a concept identifier to a first UI element in the respective UI. Furthermore, in these embodiments, the storage unit 330 may be further configured to store a concept instance of the concept identifier. The concept instance comprises a representation of the respective UI, a description of the first UI element, and the concept identifier. For example, a concept example of an "OK button" concept may include a representation of a UI including an OK button (e.g., a DOM tree), an XPath representation of the OK button, and a concept identifier of the "OK button" (which may be the "OK button" itself or any identifier). Thus, each UI element in the user interface may be considered as an instance of a concept in the context of the present disclosure. As mentioned above, storing a (new) concept instance may be a predefined trigger condition for initiating the generation of at least one of the local description and the global description.

[0062] Although not shown in FIG. 3, in some embodiments, components including the acquisition unit 310, the generation unit 320, and the storage unit 330 may be implemented as a combination of processing circuitry and memory within the system 300.

[0063] 4 is a block diagram of a system for performing a re-identification check of a UI element in a current UI, in accordance with one embodiment of the present disclosure. In some embodiments, the UI element may be part of an action of an automated workflow.

[0064] As shown in Fig. 4, a system 400 is provided, comprising an acquisition unit 410 and a query unit 420. The acquisition unit 410 is configured to search at least one of the stored first local description and the stored first global description for the UI element based on a concept identifier representing the UI element. The search is based on the concept identifier representing the UI element. For example, if the UI element is an "OK button", the UI element may be associated with a concept identifier that identifies the "OK button" as a related concept. The first local description and / or the first global description may have been generated for the concept based on the method described above with reference to Fig. 1 or may have been generated by the system 300 in the manner described with reference to Fig. 3.

[0065] As mentioned above, in some embodiments, the UI element may be part of an action of the automated workflow. In these embodiments, the system 400 may further include an acquisition unit (not shown in FIG. 4 ) configured to acquire an automated workflow before the acquisition unit 410 retrieves at least one of the stored first local description and the stored first global description, the automated workflow including at least an action involving a UI element. In a practical implementation, the automated workflow may include multiple actions, each action including multiple UI elements. The acquisition unit may be configured to acquire the automated workflow by receiving a user input corresponding to an action in the automated workflow. For example, the user input may correspond to an operation for creating a corresponding action in the automated workflow.

[0066] The query unit 420 is configured to query a representation of the current UI using at least one of the stored first local description and the stored first global description to determine whether the respective description can be used to re-identify a UI element. For example, the first local description may be an XPath description, and the query unit 420 may be configured to query a DOM tree representing the current UI to determine whether there is a match for this XPath description in the DOM tree. If it is determined that there is a match, the re-identification of the UI element is successful. On the other hand, if it is determined that there is no match, the re-identification of the UI element is not successful. In some embodiments, performing a re-identification check of the UI element by the query unit 420 may be considered as part of the execution of an automated workflow.

[0067] 2, the terms "re-identification" and "re-identifying" may refer to the process of identifying a UI element from a UI, or more specifically, from a UI representation, without necessarily requiring that the UI element have previously been "identified." Additionally, the terms "re-identification" and "re-identifying" may also refer to scenarios where the UI element cannot be identified using the local / global description, and thus a different local / global description is used to perform the "re-identification."

[0068] In some embodiments where both the stored first local description and the stored first global description are available, the acquisition unit 410 may be configured to retrieve at least the first local description based on the concept identifier, and the query unit 420 may be configured to query the presentation of the current UI using the first local description. In this manner, the system 400 may prioritize the use of the local description over the global description in a re-identification check, and may provide more reliable results since the local description is specific to the current UI.

[0069] In some embodiments, the system 400 may further comprise a storage unit configured to store the representation of the current UI and the first local description and / or the first global description in association with the concept identifier if the query unit 420 determines that the first local description and / or the first global description can be used to re-identify the UI element.

[0070] Alternatively, the system 400 may further comprise a discarding unit configured to discard the first local description and / or the first global description when it is determined by the query unit 420 that the first local description and / or the first global description cannot be used to re-identify the UI element. Further, alternatively or additionally, when it is determined by the query unit 420 that the UI element cannot be re-identified using the first local description and / or the first global description, the obtaining unit 410 may be further configured to retrieve at least one of the stored second local description and the stored second global description based on the concept identifier. The second local description is different from the first local description, and the second global description is different from the second global description. Furthermore, the query unit 420 may be further configured to query the representation of the current UI using at least one of the second local description and the second global description to determine whether the respective description can be used to re-identify the UI element.

[0071] As yet another alternative, in some embodiments, the system 400 may further comprise a requesting unit configured to request a user input to indicate a UI element in the current UI and / or to perform description mining of concepts if it is determined by the querying unit 420 that the first local description and / or the first global description cannot be used to re-identify the UI element. The description mining may be performed according to the method described above with reference to FIG. 1 and / or by the system 300 described above with reference to FIG.

[0072] Although not shown in FIG. 4, in some embodiments, components including the acquisition unit 410 and the query unit 420 may be implemented as a combination of processing circuitry and memory within the system 400.

[0073] It will be appreciated that in some embodiments of the present disclosure, the functionality of systems 300 and 400 shown in FIGS. 3 and 4 may be combined in certain embodiments into a single system, such a system enabling both description mining and re-identification checks to be performed.

[0074] FIG. 5 illustrates an exemplary process for performing concept description mining and performing re-identification checks of UI elements in a current UI, according to one embodiment of the present disclosure.

[0075] As shown in Figure 5, the method is illustrated by several steps, numbered (i) to (xi) in the figure for ease of description. These steps are explained in more detail below with reference to a user, an automation developer, using an RPA editor to build a workflow. In the context of the present embodiment, the user is building a workflow for an open application UI by using certain UI elements in the application UI.

[0076] (i) The user highlights a UI element that they are interested in. In the example shown in Figure 5, this is the OK button on the application UI.

[0077] (ii) The RPA editor or system (e.g., system 300 of FIG. 3) captures the highlighted element, in other words, the highlighted element is located within the current application UI.

[0078] (iii) As explained above with reference to FIG. 1, for example, UI elements can be considered as instances or examples of concepts. For example, the OK button in the current UI as shown in FIG. 5 can be considered as an instance of the "OK button" concept or an instance of the "Login button" concept, both of which can be considered as sub-concepts of the "Submit button" concept. The user can create these concepts and place them in the appropriate positions in the concept hierarchy before highlighting the respective instances on the UI. Alternatively, an existing concept hierarchy can be used directly by the user.

[0079] (iv) A representation of the entire UI and the location of the highlighted UI element within the UI are stored with a concept identifier (sometimes called a "concept ID") as an instance in the "UI Representation and Highlighted Objects" storage. The representation of the UI may be unstructured (e.g., a screenshot of the UI), structured (e.g., a DOM tree), or both. The location of the element of interest within the UI representation is in the form of a spatial coordinate, or an identifier that uniquely identifies it, such as a CSS selector or XPath expression. If there are privacy concerns, a structured representation of the UI, such as a DOM, may be used, but with all of the information content (e.g., text and image nodes) removed. The following table shows an example of information stored in an instance for an "OK button" concept. For each application or website, an instance may be stored separately for each concept.

[0080] [Table 1]

[0081] (v) In this example, storing a new concept instance in the “UI Representation and Highlighted Objects” database can trigger description mining by a UI Concept Description Miner (sometimes referred to herein simply as the “Description Miner”).

[0082] (vi) The description miner creates and / or updates local and global descriptions of the relevant concepts. For example, if the OK button highlighted in step (i) belongs to a SAP website, the description miner may create and / or update the description(s) of the “OK button” concept for this particular website as well as the global description(s) that may be used as default description(s) of the “OK button” concept for other websites and applications. The global description may be generated using all stored information across different websites and applications for this particular concept, while the local description may be generated per website or per application. The description miner may be composed of multiple specialized miners in some embodiments. For example, an XPath miner may create and / or update XPath expressions for each UI element, while a CSS miner may create and / or update CSS selectors. Similarly, a Computer Vision (CV) miner may use UI screenshot(s) and create both local and global CV description(s) of the concept. This means that the description miner may generate various types of descriptions for the same concept and store them in the “UI concept description” store. The "mined" descriptions can be vetted by human users, so that the mined descriptions can be accepted or rejected after review.

[0083] In many cases, even if an element of interest can be automatically identified using existing descriptions, it would still be useful to create a description that is efficient and robust (local / global) over time. For example, an XPath description may already be sufficient to find a submit button instance on a website, but the description miner can use several stored UI descriptions of the same website to generate a more robust CSS selector. Thus, even if the stored XPath expression becomes invalid in the future, the mined CSS selector can be used to self-heal the workflow and re-identify the intended UI element.

[0084] (vii) Each mined description is paired with an associated concept identifier (i.e., one that corresponds to a concept representing the type of the respective UI element), so that they can be queried and accessed through the corresponding concept identifier.

[0085] (viii) A user creates a workflow by using a concept identifier to reference the corresponding element of interest. Thus, the identifiers of the UI elements and the workflow description are separated. This allows the description of the UI elements (i.e., concepts) to be subsequently modified without making any changes to the existing workflow.

[0086] (ix) Digital workers can execute workflows fully or partially during workflow design time to check if the workflow or parts of the workflow are working properly. While doing so, concepts are queried to retrieve descriptions of corresponding UI elements.

[0087] (x) The digital worker re-identifies the element of interest using the retrieved description (i.e., XPath expression, CSS selector, etc.). If the element is correctly re-identified, the UI representation and the automatically re-identified element are stored for future use, e.g., for use in finding a better description. If the element cannot be re-identified, the user may be asked to highlight the element again.

[0088] (xi) During normal execution of the workflow, if an existing description (searched for based on the concept identifier) ​​cannot be used to re-identify the UI element, a request can be made to a human operator to highlight the unidentified element on the screen: steps (ix) and (x) can be repeated at a later time and, if possible, execution of the workflow can be resumed.

[0089] It will be appreciated that this description is by way of example only, and that changes and modifications can be made to the described embodiments without departing from the scope of the present disclosure, as defined in the claims.

[0090] While the present technology has been described in detail for illustrative purposes based on what are currently considered to be the most practical and preferred embodiments, it should be understood that such detailed description is for that purpose only and that the present technology is not limited to the disclosed embodiments, but on the contrary, is intended to cover modifications and equivalent arrangements within the scope of the appended claims. For example, it should be understood that the present technology contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.

[0091] Throughout this specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with an embodiment or example is included in at least one embodiment of the invention. Thus, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" in various places throughout this specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in any suitable combinations and / or subcombinations in one or more embodiments or examples. In addition, it will be understood that the drawings provided herein are for purposes of explanation to persons skilled in the art, and that the drawings are not necessarily drawn to scale.

[0092] The embodiments according to the present disclosure may be embodied as an apparatus, a method, or a computer program product. For example, in some embodiments, a computer program product may be provided that includes instructions that, when executed by a processor, cause the processor to perform the method described with respect to FIG. 1 and / or FIG. 2. As another embodiment, a computer-readable storage medium may be provided that includes instructions that, when executed by a computer, cause the computer to perform the steps of the method described in FIG. 1 and / or FIG. 2. Thus, the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "module" or "system." Furthermore, the embodiments of the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0093] Although described in connection with an exemplary computing system environment, embodiments of the disclosure are operational with numerous other general purpose or specialized computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with aspects of the disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0094] Any combination of one or more computer usable or computer readable media may be utilized. For example, the computer readable media may include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read only memory (ROM) device, an erasable programmable read only memory (EPROM or flash memory) device, a portable compact disk read only memory (CDROM), an optical storage device, and a magnetic storage device. Computer program code for carrying out the operations of the embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0095] The flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions. These computer program instructions may be stored on a computer-readable medium that can instruct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored on the computer-readable medium generate an article of manufacture including instruction means that implement the functions / operations specified in the flowcharts and / or block diagrams.

[0096] Below is a list of embodiments of the present disclosure.

[0097] 1. A computer-implemented method for performing concept description mining, comprising: obtaining a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface, UI, element corresponding to a type of UI element represented by the concept; generating at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, the local description of the concept comprising description components specific to a respective UI, and the global description of the concept comprising description components that are not specific to a UI; storing at least one of the generated local and global descriptions so as to be associated with a concept identifier corresponding to the concept; The method according to claim 1,

[0098] 2. The method of embodiment 1, wherein the concept identifier uniquely identifies the concept within a concept collection, the concept collection being an ontology of multiple different types of UI elements.

[0099] 3. The method of embodiment 2, wherein the multiple reference descriptions include at least a local historical reference description and a non-local reference description, the local historical reference description being specific to the current UI, and the non-local reference description being not specific to the current UI.

[0100] 4. The method according to any one of the preceding embodiments, wherein generating the local description includes updating an existing local description of the concept and / or generating the global description includes updating an existing global description of the concept.

[0101] 5. A method according to any one of the preceding embodiments, wherein the description indicates at least one of a location and attributes of each UI element or each concept.

[0102] 6. The method of any one of the preceding embodiments, wherein each of the plurality of criteria descriptions is in the form of one of Cascading Style Sheets, CSS, Selectors, Extensible Markup Language Path Language, XPath, Expressions, Images and / or Audio Descriptions.

[0103] 7. The method of any one of the above embodiments, further comprising a step of obtaining a corresponding UI representation for each of a plurality of criteria descriptions, wherein the step of generating a local description of the concept or a global description of the concept is further based on a correlation between the criteria descriptions and the corresponding UI representation.

[0104] 8. The method of embodiment 7, wherein the UI representation comprises at least one of a structured representation of the respective UI and an unstructured representation of the respective UI.

[0105] 9. The method of embodiment 8, wherein the structured representation of the UI includes a document object model, DOM, tree, and / or the unstructured representation of the UI includes screenshots of the respective UI.

[0106] 10. Before obtaining a representation of the corresponding UI of each reference description, further comprising the step of generating an updated UI representation by removing information content from the original UI representation, and the obtained representation is the updated representation; The method according to any one of embodiments 7 to 9.

[0107] 11. The method of embodiment 10, wherein the original UI representation includes a DOM tree, and the information content includes text nodes and / or image nodes in the DOM tree.

[0108] 12. A method according to any one of the preceding embodiments, wherein generating a local description for the concept includes determining a common description component from a plurality of local historical reference descriptions, and storing the generated local description includes storing the determined common description component as at least a part of the local description.

[0109] 13. A method according to any one of the preceding embodiments, wherein generating a global description for the concept includes determining common description components from a plurality of non-local reference descriptions, and storing the generated global description includes storing the determined common description components as at least a part of the global description.

[0110] 14. The method of embodiment 12 or embodiment 13, wherein determining the common description component is performed using a trained machine learning model.

[0111] 15. The method includes determining whether a predetermined trigger condition is met prior to generating at least one of a local description and a global description for the concept; generating at least one of the local description and the global description is performed only if it is determined that a predetermined trigger condition is satisfied; 13. The method according to any one of the preceding embodiments.

[0112] 16. The method of embodiment 15, wherein the predetermined trigger condition is associated with at least one of the following: a given time interval; storing a new concept instance of the concept, the concept instance including a position of each UI element and a representation of each UI; Indicates user input.

[0113] 17. The method further comprises: obtaining user input, the user input corresponding to an assignment of the concept identifier to a first UI element in a respective UI; storing a concept instance for the concept identifier, the concept instance comprising the respective UI representation, a description of the first UI element, and the concept identifier; having The storage of a concept instance is a predetermined trigger condition. The method of embodiment 15 or embodiment 16.

[0114] 18. The method further comprises obtaining user input to accept or reject at least one of the generated local description and the generated global description; storing the generated local description and / or the generated global description is performed only by obtaining a user input accepting the respective generated local description or the generated global description; The method of any one of the preceding embodiments.

[0115] 19. A computer-implemented method for performing a re-identification check of a user interface, UI, element in a current UI, comprising: obtaining, for the UI element, at least one of a first stored local description and a first stored global description based on a concept identifier representing the UI element; querying a representation of the current UI element using at least one of the stored first local description and the stored first global description to determine whether the respective descriptions can be used to re-identify the UI element; The method according to claim 1,

[0116] 20. The method of embodiment 19, wherein the UI element is part of an action of an automated workflow, and prior to retrieving the at least one of the stored first local description and the stored first global description, the method further includes obtaining the automated workflow, the automated workflow including at least an action involving the UI element.

[0117] 21. The method of embodiment 20, wherein obtaining the automated workflow includes receiving a user input corresponding to the action in the automated workflow.

[0118] 22. A method according to any one of embodiments 19 to 21, wherein performing a re-identification check of the UI element is part of executing the acquired automated workflow.

[0119] 23. A method according to any one of embodiments 19 to 22, wherein both a stored first local description and a stored first global description are available, and the method includes retrieving at least the first local description based on the concept identifier, and querying the current UI presentation using the first local description.

[0120] 24. A method described in any one of embodiments 19 to 23, further comprising storing a representation of the current UI and the first local description and / or the first global description in association with the concept identifier if it is determined that the first local description and / or the first global description can be used to re-identify the UI element.

[0121] 25. If it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, obtaining at least one of a stored second local description and a stored second global description based on the concept identifier, the second local description being different from the first local description and the second global description being different from the second global description; querying a representation of the current UI using at least one of the second local description and the second global description to determine whether the respective description can be used to re-identify the UI element; The method according to any one of embodiments 19 to 23, comprising:

[0122] 26. A method described in any one of embodiments 19 to 23, further comprising, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, requesting user input to indicate the UI element within the current UI.

[0123] 27. A method described in any one of embodiments 19 to 23, further comprising performing concept description mining according to any one of embodiments 1 to 18 if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description.

[0124] 28. A method according to any one of embodiments 19 to 27, wherein the first local description and / or the first global description are generated according to any one of embodiments 1 to 18.

[0125] 29. A system for performing description mining for concepts, comprising: an acquiring unit configured to acquire a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface, UI, element corresponding to a type of UI element represented by the concept; a generation unit configured to generate at least one of a local description of the concept and a global description of the concept based on the plurality of reference descriptions, the local description of the concept comprising description components specific to a respective UI, and the global description of the concept comprising description components that are not specific to a UI; a storage unit configured to store the at least one of the generated local and global descriptions so as to be associated with a concept identifier corresponding to the concept; A system comprising:

[0126] 32. A system for performing a re-identification check of a user interface, UI, element in a current UI, comprising: an obtaining unit configured to retrieve, for the UI element, at least one of the stored first local description and the stored first global description based on a concept identifier representing the UI element; a query unit configured to query a representation of the current UI using at least one of the stored first local description and the stored first global description to determine whether the respective descriptions can be used to re-identify the UI element; A system comprising:

[0127] 33. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method of any one of embodiments 1 to 28.

Claims

1. A computer-implemented method (100) for performing concept description mining, comprising: obtaining (S110) a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface, UI, element corresponding to a type of UI element represented by the concept; generating (S120) at least one of a local description of the concept or a global description of the concept based on the plurality of reference descriptions, the local description of the concept comprising description components specific to a respective UI, and the global description of the concept comprising description components that are not specific to a UI; storing (S130) at least one of the generated local and global descriptions in association with a concept identifier corresponding to the concept; The method according to claim 1,

2. The method of claim 1 , wherein the concept identifier uniquely identifies the concept within a concept set, the concept set being an ontology of a plurality of different types of UI elements.

3. 3. The method of claim 1, wherein each of the plurality of criteria descriptions is in the form of one of a Cascading Style Sheet (CSS) selector, an Extensible Markup Language Path Language (XPath) description, an image and / or an audio description.

4. The method further comprises obtaining, for each of the plurality of criteria descriptions, a corresponding UI representation; generating a local description of the concept or a global description of the concept based on a correlation between the reference description and a corresponding UI representation; 4. The method according to any one of claims 1 to 3.

5. The method further comprises generating an updated UI representation by removing information content from the original UI representation before obtaining the corresponding UI description of each reference description; the obtained representation is an updated representation. The method of claim 4.

6. 6. The method of claim 1, wherein generating a local description of the concept comprises determining a common description component from a plurality of local historical reference descriptions, and / or generating a global description of the concept comprises determining a common description component from a plurality of non-local reference descriptions.

7. The method further comprises, prior to generating at least one of a local description or a global description of the concept, determining whether a predetermined trigger condition is met; the step of generating at least one of the local or global description is performed only if it is determined that the predetermined trigger condition is satisfied.

7. The method according to any one of claims 1 to 6.

8. The method further comprises: obtaining user input, the user input corresponding to assigning the concept identifier to a first UI element in a respective UI; storing a concept instance of the concept identifier, the concept instance comprising the representation of the respective UI, a description of the first UI element, and the concept identifier; having storing the concept instance is a predetermined trigger condition; The method of claim 7.

9. The method further comprises obtaining user input to accept or reject at least one of the generated local description and the generated global description; the step of storing the generated local description and / or the generated global description is performed only by obtaining a user input accepting the generated local description or the generated global description, respectively; 9. The method according to any one of claims 1 to 8.

10. A computer-implemented method (200) for performing a re-identification check of a user interface (UI) element in a current UI, comprising: obtaining, for the UI element, at least one of a first stored local description or a first stored global description based on a concept identifier representing the UI element (S210); Querying a current UI element using the at least one stored first local description and the stored first global description to determine whether the respective descriptions can be used to re-identify the UI element (S220); The method according to claim 1,

11. The method of claim 10, further comprising storing the current UI representation and the first local description and / or the first global description so as to be associated with the concept identifier if it is determined that the first local description and / or the first global description can be used to re-identify the UI element.

12. The method further comprises, if it is determined that the UI element cannot be re-identified using the first local description and / or the first global description, obtaining at least one of a second stored local description and a second stored global description based on the concept identifier, the second local description being different from the first local description and the second global description being different from the second global description; querying the current UI representation using at least one of the second local description and the second global description to determine whether the respective descriptions can be used to re-identify the UI element; The method of claim 10, comprising:

13. The method further comprises, if it is determined that the first local description and / or the first global description cannot be used to re-identify the UI element, requesting user input indicating the UI element within the current UI; performing description mining of concepts according to any one of claims 1 to 9; Executing at least one of the following: The method of claim 10.

14. A system (300) for performing concept description mining, comprising: an acquisition unit (310) configured to acquire a plurality of criteria descriptions, each of the plurality of criteria descriptions being associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; a generating unit (320) configured to generate at least one of a local description of the concept or a global description of the concept based on the plurality of reference descriptions, the local description of the concept including description components specific to a respective UI, and the global description of the concept including description components that are not specific to a UI; a storage unit (330) configured to store the at least one generated local description and global description in association with a concept identifier corresponding to the concept; A system comprising:

15. 1. A system (400) for performing a re-identification check of a user interface (UI) element in a current UI, comprising: an acquiring unit (410) configured to acquire, for the UI element, at least one of a first stored local description or a first stored global description based on a concept identifier representing the UI element; a query unit (420) configured to query a current UI representation using the at least one stored first local description or the stored first global description to determine whether the respective description can be used to re-identify the UI element; A system comprising:

16. A computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 13.

Images