Tagging user action data in user environment

Abstract

The invention relates to a computer-implemented method for tagging user action (UAC) data in a user environment (ENV), in particular wherein the user environment (ENV) is a computer (CMP)-implemented user interaction (UIA) environment (ENV), the method is characterized by comprising the following steps: a) providing a pre-trained user action (UAC) pattern recognition model (PRM), the pattern recognition model (PRM) being designed in such a way that the pattern recognition model (PRM) recognizes a user action (UAC) pattern (APT) known to the pattern recognition model (PRM) from a sequence of user action (UAC) parts comprising the user action (UAC) pattern (APT) known to the pattern recognition model (PRM); and outputting a user action (UAC) mode (APT) or a flag associated with the user action (UAC) mode (APT), the invention relates to a method for identifying user actions (UAC) in a user action (UAC) sequence (SQC), b) recording of user actions (UAC), c) analyzing the recorded user actions (UAC) by a user action (UAC) pattern recognition model (PRM) for a user action (UAC) pattern (APT) known by the pattern recognition model (PRM), d) if a known user action (UAC) pattern (APT) is not detected in the user action (UAC) sequence (SQC), generating a user interaction (UIA) as a user interaction (UIA) requesting information about a previous user action (UAC) sequence (SQC), the present invention relates to a method for training a pattern recognition model (PRM) using a user action (UAC) sequence, e) tagging past user action (UAC) sequences (SQC) using information received from an information request, f) storing the tagged user action (UAC) sequences (SQC) in a user action (UAC) sequence (SQC) database (DBS), g) training the pattern recognition model (PRM) using the user action (UAC) sequences (SQC) and corresponding tags (LBL) of the user action (UAC) sequence (SQC) database (DBS).

Description

Technical Field

[0001] This invention relates to a computer-implemented method for tagging user action data in a user environment, particularly wherein the user environment is a computer-implemented user interactive environment. Furthermore, this invention relates to a system for tagging user action data in a computer-implemented user interactive environment. Background Technology

[0002] Learning how to operate a new machine or use new software is a tedious task for both experienced personnel and new hires. While training provides an initial understanding, expert knowledge and domain expertise are not easily transferred. This results in lengthy learning times and makes the use of machines and software prone to errors. User-assistive technologies can provide ways to shorten learning times and offer contextual support. In many cases, rule-based assistance is impossible because the rules are unknown or too difficult to implement. Therefore, modern assistive technologies are often data-driven. In an industrial context, assistive technologies face a number of exacerbated challenges: scarce user data, highly variable user backgrounds, and the critical importance of recommendations, necessitating a high level of accuracy.

[0003] US 6564213B1 and US 8,645,825 B1 relate to known user-assisted techniques known as “autocomplete” or “predictive text.” Autocomplete is a feature that suggests words or phrases to a user as they type, based on previously entered or frequently used phrases. This feature is commonly used in search engines, text messaging applications, and other applications in which users type text.

[0004] In user-assistance technologies, autocomplete can be used to suggest commands or actions based on the user's current context or history of previous actions. For example, an AI assistant can suggest common commands or actions based on the user's previous interactions with the system or on the user's current task or goal.

[0005] Another form of command prediction in user-assisted technology is known as "intent recognition," as exemplified by US2012 / 0016678 A1. Intent recognition involves analyzing user input to determine the user's desired action or request. This can be used to suggest relevant commands or actions to the user, or to automate common tasks based on user input.

[0006] Overall, command prediction is a powerful tool among user-assisted technologies because it can help users navigate complex systems more quickly and efficiently.

[0007] When applying machine learning assistants to best support users, a key factor appears to be the training data for the assistant.

[0008] Training data can be collected from manuals (at a low level) or by recording user actions (to enable high-level training for the assistant).

[0009] Currently, if fully available, typical user action data logging yields unlabeled sequences of user interactions. This data is stored locally as log files or collected via telemetry business intelligence. Such data is diluted and therefore largely useless, as only a tiny fraction of user interactions represent meaningful sequences of steps. Furthermore, there is no inference available to understand why users performed the recorded actions or in what context.

[0010] Siemens NX CAD is a computer-aided design (CAD) software developed by Siemens Digital Industries Software. It is used to design and develop products in various industries, such as automotive, aerospace, and industrial machinery.

[0011] In Siemens NX CAD, "NX" stands for "Unigraphics NX," which was the software's original name before it was acquired by Siemens.

[0012] "Command prediction in Siemens NX CAD" is an example of command prediction technology in user assistance. Siemens NX CAD includes a command prediction feature that suggests the next command or function to the user based on the user's current context and previous actions.

[0013] The command prediction feature in Siemens NX CAD uses machine learning algorithms to analyze user input and predict the next command or function the user is likely to use.

[0014] This auxiliary technology used in industrial settings typically has two drawbacks: 1. It is recommended to have a very limited field of view, such as only the next step, rather than the entire sequence. This drawback may be caused by insufficient training.

[0015] 2. The base model is not self-learning. Summary of the Invention

[0016] The purpose of this invention is to improve auxiliary technology.

[0017] The present invention solves the problem by means of a method of the type initially mentioned, including the additional features of claim 1. Furthermore, the present invention proposes a system for solving the problems described above.

[0018] One line of reasoning in this invention is that the shortcomings described above may be due to insufficient training of the auxiliary model. The auxiliary model has not been trained on meaningful sequences or even subsequences of user actions.

[0019] Another fundamental understanding of the invention is that the quality of the training and training data necessary for the assistance system to provide more professional advice requires providing labeled user action data sequences or subsequences.

[0020] Conventionally, labeling is not feasible because the workload of labeling user action data used to train assistance systems grows exponentially.

[0021] Another fundamental understanding of this invention is that conventional assistance systems suffer from the problem that the model is trained once and then used.

[0022] Another fundamental understanding of this invention is that training data can be varied more efficiently over time. Users can be tracked by the system, but simply tracking users over time does not increase the number of labeled data points. There appears to be a conventional lack of tracking labels and context.

[0023] This invention addresses the problem of improving known computer implementations of methods, enabling further optimization with respect to the specified challenge.

[0024] A computer-implemented method for tagging user action data in a user environment is proposed, wherein the user environment is a computer-implemented user interactive environment.

[0025] The key point is that this method includes the following steps: a) Provide a pre-trained user action pattern recognition model; The pattern recognition model is designed to identify the known user action patterns from a sequence of user action parts that include user action patterns known to the pattern recognition model, and output the user action patterns or tags associated with the user action patterns. b) Record user actions, c) User actions recorded by the user action pattern recognition model based on the user action patterns known to the model. d) If no known user action pattern is detected in the user action sequence, generate a user interaction that requests information about past user action sequences. e) Use the information received from the information request to mark past user action sequences. f) Store the tagged user action sequences in the user action sequence database. g) Train the pattern recognition model using user action sequences and corresponding labels in a user action sequence database.

[0026] In the context of this invention, user action data tagging is the process of adding descriptive labels or annotations to the data to make it easier to understand and analyze. This involves assigning tags or labels to user action data. Tags are used to identify the data and categorize it into different classes, making searching, sorting, and analysis easier. In the context of the method according to the invention, user action data tagging is a key step in training a pattern recognition model, preferably a machine learning model or an artificial intelligence model. Tagging helps in training to recognize patterns and make accurate identifications.

[0027] User action records can include: 1. Login attempts and user authentication activities. 2. User input, such as data entered into forms or search queries. 3. Preferably, cursor movement in an application environment combined with additional cursor actions. 4. View and navigation behavior, 5. Any changes made to settings or configurations.

[0028] Basically, user action logging or tracking can refer to all the inputs a user makes within an application.

[0029] These are just a few examples, and the specific types of user actions recorded will depend on the specific program or application used and the type of human-computer interface used. It is also worth noting that some programs may offer users the option to disable logging or completely opt out of data collection, while others may need to log for security or other operational reasons.

[0030] According to one proposed implementation, user interaction is accomplished using a natural language processing model.

[0031] According to one proposed implementation, a user action reduction module is applied to a recorded sequence of user actions, wherein the user action reduction module is designed to identify and eliminate potential user actions in the sequence of user actions that have no impact on the user's environment.

[0032] Another equally important teaching of claim 4 relates to a system for tagging user action data in a computer-implemented user interactive environment.

[0033] Another equally important teaching relates to a system for tagging user action data in a computer-implemented user interactive environment, in particular, wherein the user environment is a computer-implemented user environment.

[0034] According to one proposed implementation method - At least one computer with a processor; - At least one memory coupled to at least one processor, the memory storing a set of instructions to be executed by the processor. The instruction set, when executed by the processor, causes the system to perform the steps according to any one of claims 1 to 3.

[0035] All instructions given for computer-implemented methods are fully applicable.

[0036] Another equally important teaching concerns a system that includes: - At least one computer with a processor; - At least one memory, coupled to the at least one processor, stores a set of instructions to be executed by the processor. The instruction set, when executed by the processor, causes the system to perform the steps according to any one of claims 1 to 4.

[0037] All instructions given for computer-implemented methods are fully applicable.

[0038] According to one embodiment, a system for providing suggestions to a user in a computer-implemented user interactive environment is proposed, the system comprising: - At least one computer with a processor; - At least one memory, coupled to the at least one processor, stores a set of instructions to be executed by the processor. The instruction set, when executed by the processor, causes the system to perform the steps according to claim 5.

[0039] All instructions given for computer-implemented methods are fully applicable.

[0040] Furthermore, according to the present invention, a computer system for preparing to execute the method is provided.

[0041] The computer system referred to above can be a single computer or a computer system consisting of multiple computers connected to each other (such as through a World Wide Web or another network configuration). Preferably, the simulation can be distributed within the computer system, such that the computational work is distributed among several computers and processors.

[0042] This invention proposes a method and system for tracking user interactions during the operation of machinery or equipment, or while using software. These application possibilities are summarized by the term "user environment." The system records user actions or interactions in a "user action sequence database," which may also be referred to as an "interaction sequence database." Preferably, a specific history of user interactions (e.g., the last 100 interactions) can be continuously analyzed in the system's memory using a pattern recognition algorithm, which is used to identify repetitive or meaningful sequences. Once a pattern is identified, a user-assistance method (claim 5) or corresponding system triggers an interaction with the user and requests user input, such as via a chatbot or even more preferably via voice input (=> tagging). In some cases, the type of interaction (interaction model) used for tagging depends on the detected pattern. In this case, a user interaction generation component defines the tagging workflow, preferably using a template / rule-based method.

[0043] In the labeling step, user input for a specific pattern is preferably analyzed using Natural Language Processing (NLP), and semantic information can be stored as labels in the user's step sequence, thus forming high-quality data points. The NLP algorithm can rely on a pre-trained language model, and the labeled database can preferably contain all previously used labels. This method ensures the consistency and quality of the input labels.

[0044] A key objective is to minimize user workload while receiving high-quality input (i.e., the context and reasoning behind user behavior). The method and system acquire crucial information about why customers repeat certain actions or why users perform certain steps. This information is critical for training pattern recognition models at a higher functional level, which not only identifies patterns but also the corresponding action sequences. Attached Figure Description

[0045] Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 A block diagram of the method according to the present invention is shown.

[0046] The illustrations in the accompanying drawings are schematic. It should be understood that similar or identical elements may be provided with the same reference numerals in different drawings. Detailed Implementation

[0047] Figure 1 A block diagram of a computer-implemented method according to the present invention for marking user action UAC data in a user interactive UIA environment (ENV) implemented in a computer CMP is shown. Furthermore, Figure 1 The diagram illustrates the user-aided method and the system SYS, which respectively apply the marking method.

[0048] The flowchart illustrates the sequence of steps a) through g).

[0049] During the first step a), a pre-trained User Action UAC Pattern Recognition Model (PRM) is provided. This PRM is designed and trained to recognize User Action UAC Patterns (APTs) from partial sequences of User Action UACs, provided that the corresponding pattern is known to the PRM. The PRM outputs a stored User Action UAC Pattern (APT) or an identifier or tag (LBL) associated with the User Action UAC Pattern (APT).

[0050] In the second step b), the method provides the recording or tracking of user actions (UAC), where user actions are performed by the user (USR) using the environment (ENV).

[0051] Subsequently, during step c), the User Action UAC Pattern Recognition Model (PRM) analyzes these recorded User Action UACs against the User Action UAC Pattern (APT) known to the PRM. Depending on the analysis results, a User Interaction (UIA) is generated in step d). If no known User Action UAC Pattern (APT) is detected in the User Action UAC Sequence (SQC) during the analysis, the User Interaction (UIA) requests information INF about previous User Action UAC Sequences (SQC). This request is made using the Natural Language Processing (NPL) model.

[0052] If no known user action UAC pattern APT is detected, monitor the user action UAC of a predefined number of LMTs in the sequence SQC before starting to request information INF about the previous sequence SQC for user interaction UIA.

[0053] If at least one known user action UAC pattern APT [YES] is identified through the above analysis of the pattern recognition model PRM, the method provides to generate a user interaction UIA via a human-machine interface HMI, which proposes at least one subsequent user action UAC.

[0054] Upon receiving information about an unknown user action (UAC), this information is used to mark a previous user action UAC sequence (SQC) in step e) using the information INF received from the information INF request.

[0055] Before storing the labeled sequence SQC, for example, immediately after recording or during analysis, the User Action UAC Reduction Module (RDM) can optionally be applied to the recorded User Action UAC sequence SQC, wherein the User Action UAC Reduction Module (RDM) is designed to identify and eliminate potential User Action UACs in the User Action UAC sequence SQC that do not affect the User Environment (ENV).

[0056] Subsequently, in step f), the labeled user action UAC sequence SQC is stored in the user action UAC sequence SQC database DBS, and the entries in the database are used to train the pattern recognition model PRM (step g).

[0057] A system SYS (which may be part of an auxiliary system SYS) for marking user action UAC data in a user interactive UIA environment ENV implemented in a computer CMP (here, the term "system" is not distinguished because its corresponding hardware components may be the same) includes: a computer CMP having a processor CPU; and a memory MEM coupled to at least one processor CPU, wherein the memory MEM stores an instruction set SOI to be executed by the processor CPU, wherein the instruction set SOI, when executed by the processor CPU, causes the system to perform the method steps according to the invention.

[0058] Although the invention has been described in detail with reference to preferred embodiments, it should be understood that the invention is not limited to the disclosed examples, and that many additional modifications and variations can be made by those skilled in the art without departing from the scope of the invention.

Claims

1. A computer-implemented method for tagging user action (UAC) data in a user environment (ENV), particularly, wherein, The user environment (ENV) is a user interactive (UIA) environment (ENV) implemented by a computer (CMP), characterized by including the following steps: a) Provide a pre-trained User Action (UAC) pattern recognition model (PRM); - Wherein, the pattern recognition model (PRM) is designed such that the pattern recognition model (PRM) recognizes the user action (UAC) pattern (APT) known by the pattern recognition model (PRM) from a sequence of user action (UAC) partial sequences including the user action (UAC) pattern (APT) known by the pattern recognition model (PRM), and outputs the stored user action (UAC) pattern (APT) or an identifier or tag (LBL) associated with the user action (UAC) pattern (APT). b) Record user actions (UAC). c) The user actions (UACs) recorded by the User Action (UAC) Pattern Recognition Model (PRM) for the User Action (UAC) Pattern (APT) known to the PRM. d) If no known User Action (UAC) pattern (APT) is detected in the User Action (UAC) sequence (SQC), generate a User Interaction (UIA) via the Human-Machine Interface (HMI) as a User Interaction (UIA) requesting Information (INF) about previous User Action (UAC) sequences (SQC). e) Use the information (INF) received from the information (INF) request to mark the previous user action (UAC) sequence (SQC). f) Store the tagged User Action (UAC) sequences (SQCs) in the User Action (UAC) Sequence (SQC) Database (DBS). g) The pattern recognition model (PRM) is trained using the User Action (UAC) sequence (SQC) and the corresponding label (LBL) of the User Action (UAC) sequence (SQC) database (DBS).

2. The computer-implemented method according to claim 1, wherein, The user interaction (UIA) is accomplished using a natural language processing (NPL) model.

3. The computer-implemented method according to claim 1 or 2, wherein, A User Action (UAC) Reduction Module (RDM) is applied to a recorded User Action (UAC) Sequence (SQC), wherein the User Action (UAC) Reduction Module (RDM) is designed to identify and eliminate potential User Actions (UACs) in the User Action (UAC) Sequence (SQC) that do not affect the User Environment (ENV).

4. The computer-implemented method according to at least one of claims 1 or 2, wherein, If no known User Action (UAC) pattern (APT) is detected, a predefined number of User Actions (UACs) are monitored in the sequence of quality control (SQC) before the User Interaction (UIA) begins.

5. A computer-implemented user-assisted method comprising a method for marking user action (UAC) data according to at least one of the preceding claims, the user-assisted method being designed to provide suggestions to a user of a computer-implemented user interactive environment (UIA) (ENV), the user-assisted method comprising the following steps: - If at least one known User Action (UAC) pattern (APT) is detected, a User Interaction (UIA) is generated via Human-Machine Interface (HMI) as a User Interaction (UIA) proposing at least one next User Action (UAC).

6. A system for marking user action (UAC) data in a computer-implemented (CMP) user interactive (UIA) environment (ENV), particularly, wherein, The user environment (ENV) is a computer-implemented user environment (ENV) that includes: - At least one computer (CMP) with a processor (CPU). - At least one memory (MEM) is coupled to at least one processor (CPU), the memory (MEM) storing a set of instructions (SOI) to be executed by the processor (CPU). The instruction set (SOI), when executed by the processor (CPU), causes the system to perform the steps according to any one of claims 1 to 4.

7. A user assistance system for providing suggestions to a user of a computer-implemented (CMP) user interactive (UIA) environment (ENV), comprising: - At least one computer (CMP) with a processor (CPU). - At least one memory (MEM) is coupled to at least one processor (CPU), the memory (MEM) storing a set of instructions (SOI) to be executed by the processor (CPU). The instruction set (SOI), when executed by the processor (CPU), causes the system to perform the steps according to claim 5.

Images