Use of machine learning and free text data to detect and report events related to the use of software applications.

Abstract

A method and system for performing actions for events related to application usage is provided. [Solution] A computing system identifies free text to be evaluated for at least one of a plurality of events related to application use, applies the free text to a machine learning (ML) architecture, determines a value indicating the likelihood of occurrence of the event related to application use based on the application of the free text to the ML architecture, and provides a generative ML model with model input based on the free text and the value to obtain data for an electronic document characterizing the event related to application use.

Description

Technical Field

[0001] (Cross - reference to related applications) This invention claims priority to U.S. Non - Provisional Patent Application No. 18 / 806,230, filed on August 15, 2024, entitled "USING MACHINE LEARNING AND FREE TEXT DATA TO DETECT AND REPORT EVENTS ASSOCIATED WITH USE OF SOFTWARE APPLICATIONS", which is hereby incorporated by reference in its entirety.

Background Art

[0002] During the use of a software application, unexpected or undesirable events may occur. These events can be caused by a number of factors. For example, an individual who is not familiar with software technology may be unable to log in correctly to their account. These events have an adverse effect on the overall performance of the application. These events also impair the user experience of using the application, resulting in a decrease in the interaction rate. By constantly preventing the full and proper use of the application, users may become dissatisfied and may even stop using the application. In addition to events that affect the performance of the application, there are events that affect the health of the user when the software application is a digital therapy. If the causes of these events remain unaddressed, the symptoms and underlying health conditions of individual users may not be addressed or treated.

[0003] In the medical field, there are monitoring and reporting requirements for certain types of events that should be described as adverse events. These requirements are imposed by various entities, including regulatory authorities (e.g., the U.S. Food and Drug Administration (FDA)), healthcare institutions, and device manufacturers. These requirements mandate that companies continuously record and report certain types of events. Monitoring and reporting adverse events encountered with digital therapies is unique and differs from that of conventional pharmaceuticals. Because digital therapies are used to address a user's disease, condition, or symptoms, an adverse event could be a software-related issue, such as a patient using a migraine digital therapy where the unpleasant symptoms are not a side effect of the treatment but rather user frustration caused by confusion while operating the software technology, and the patient is unable to log into their account and describe that this experience is worsening their migraine.

[0004] Unlike pharmaceuticals, monitoring, let alone reporting, these kinds of events is difficult. One reason for this is that users of digital therapies are not confined to a specific location (e.g., a hospital or medical institution), but can access their treatment from outside these environments, for example, from home, or as part of their daily work. Access to digital therapies from outside a specific location is a major advantage of digital therapies over traditional medicine. However, it is still more difficult to confirm whether or not a user is experiencing an adverse event, and therefore, tracking and monitoring adverse events that users are experiencing is difficult. In the case of pharmaceuticals, when a user is physically face-to-face with a healthcare professional, the healthcare professional can check whether or not the person is experiencing an adverse event (e.g., a side effect or harm).

[0005] One approach to detecting an event could rely on users directly submitting feedback to the application administrator (e.g., phone calls to customer service agents, emails to customer service addresses, chats with customer service chatbots, or voicemails received on customer service lines outside of business hours). In this case, the administrator would carefully review the submissions to determine whether the user actually experienced the event. However, the process of reviewing incoming submissions is manual, time-consuming, and may not be adequately addressed, especially if there are a large number of users using the application and these submissions are generated and recorded from multiple different sources. [Overview of the project]

[0006] This disclosure describes a system and method for using machine learning to detect and process events related to a user's experience with an application, such as a digital therapy application, as described in user feedback. There are several advantages to using the machine learning techniques detailed in this disclosure to detect events in messages about application usage. Specifically, user messages are free text and are recorded through a wide range of channels, such as email, messages, call center transcripts, chatbot messages, after-hours voicemail, or in-app input. The machine learning models described in this disclosure can rapidly analyze this large volume of free text, classify events by type and severity, and select actions to resolve the events.

[0007] In digital therapy applications, machine learning techniques are used to detect events, enabling more efficient delivery of therapeutic interventions to the user. The machine learning architecture described herein has the ability to respond to these events in near real-time upon detection, enabling immediate notification to administrators, faster resolution of events, and more prompt delivery of treatments to users, thereby reducing the likelihood of interruptions to users receiving treatment caused by such events. Furthermore, reducing such events improves the user experience, reducing frustration in digital therapy applications and increasing user satisfaction. As a result, compliance rates are higher, health outcomes are improved, and digital therapy interventions are more effective.

[0008] In addition, machine learning techniques enable better and more consistent compliance with monitoring and reporting requirements for events occurring during the use of digital therapy applications. When an event is detected, information about that event, including initial free text, is fed into a generative model, and an analysis report of the event is generated. The analysis report includes the severity level of the event and a natural language description of possible remedy measures. The analysis results generated by the generative model and the free text from the user may be stored and maintained for record-keeping and monitoring purposes. If the event is identified as being of a reportable type (e.g., a serious adverse event), the analysis report itself may be used as a submission to the relevant entity to meet reporting requirements.

[0009] As a result, the time between receiving a message about the application and detecting any event related to the use of the application is significantly reduced. The service, using the machine learning architecture detailed in this disclosure, can continuously monitor all incoming messages, including free text, and automatically generate event documentation in natural language in near real-time. This is particularly beneficial when events occur outside of business hours when actual customer service personnel cannot respond. The analysis report provides an overview of the event related to the use of the application, along with recommendations for actions to address the causes contributing to the event. As a result, application administrators no longer need to manually monitor, improving the efficiency and accuracy of identifying potential risks and reducing the time required to respond when taking action to address these events. In this way, any event detected from free text can be addressed seamlessly by the service, from the initial identification of a potential problem to the final resolution of the root cause, with minimal human input required.

[0010] Multiple aspects of this disclosure relate to systems, methods, and computer-readable media for performing actions for events related to the use of an application. One or more processors can identify free text related to an application that should be evaluated for at least one of a plurality of events related to the use of the application. One or more processors can apply the free text to a machine learning (ML) architecture. The machine learning (ML) architecture can be trained using a plurality of sample texts that represent at least one of a plurality of events related to the use of the application. Based on the application of the free text to the ML architecture, one or more processors can determine values ​​that indicate the likelihood of an event related to the use of the application occurring. One or more processors can provide a generating ML model with model inputs based on the free text and values ​​to obtain data for electronic documents that characterize events related to the use of the application. One or more processors can use the data for electronic documents to perform actions.

[0011] In some embodiments, one or more processors can apply free text to an ML architecture that includes a natural language processing (NLP) model configured to access multiple information resources related to an application. Based on the application of the free text to the ML architecture, one or more processors can identify one of the multiple information resources related to the application using at least a portion of the free text. One or more processors can obtain data by providing model input to a generated ML model based on the one information resource identified using at least a portion of the free text from the multiple information resources related to the application.

[0012] In some embodiments, the ML architecture may include a classifier model constructed using multiple sample texts. Each of the multiple sample texts may be labeled with an indication of the presence or absence of each event related to the use of the application. The multiple events related to the use of the application include at least one of adverse events, serious adverse events, incidents, serious incidents, software bugs, user complaints, or convenience issues. In some embodiments, one or more processors may classify an event as at least one of adverse events, serious adverse events, incidents, serious incidents, software bugs, user complaints, or convenience issues based on the application of free text to the ML architecture. In some embodiments, one or more processors may determine that an event meets reporting criteria for providing at least a portion of an electronic document to a remote device. In response to the determination that an event meets reporting criteria, one or more processors may transmit at least a portion of the electronic document to the remote device.

[0013] In some embodiments, a generative ML model may be trained using at least one corpus including sample inputs and sample outputs. The sample inputs can identify at least one of (i) sample information resources associated with at least one of a plurality of events, or (ii) sample values ​​indicating the possibility of at least one event, and the sample outputs can identify at least one of (i) a diagnostic result for at least one event, (ii) a sample action for at least one event, or (iii) an analysis result for at least one event. In some embodiments, one or more processors can select an action from a plurality of actions according to the data. The plurality of actions may include at least one of (i) terminating the use of an application on a user device associated with a user, (ii) restricting the operation of an application associated with an event, (iii) sending a notification to the user device for presentation to the user, (iv) providing an electronic document to an administrator device, or (v) storing an electronic document.

[0014] In some embodiments, one or more processors can determine if a value indicating the likelihood of an event meets a threshold. In response to this determination, one or more processors can provide model inputs to a generating ML model. In some embodiments, one or more processors can receive feedback via an interface that identifies updated values ​​indicating the updated likelihood of events related to the application. One or more processors can update at least one of several weights in the ML architecture based on this feedback. In some embodiments, one or more processors can generate model inputs that include free text and value-based contextual information according to a template.

[0015] In some embodiments, one or more processors can obtain free text related to an application from at least one of the following: (i) email, (ii) text message, (iii) voice transcript, (iv) chatbot message, (v) e-mail, or (vi) communication platform message. In some embodiments, one or more processors can build an event listener on the application to monitor free text generated by the application's users. One or more processors can obtain the free text through the application programming interface (API) of the application's event listener.

[0016] In some embodiments, one or more processors can generate data elements that identify (i) information resources related to the application, (ii) values, and (iii) timestamps related to a message containing free text. In some embodiments, one or more processors can generate an electronic document containing one or more recommendations for an event based on the provision of model inputs to a generating ML model. In some embodiments, the application includes a digital therapy application for addressing a condition. The application may include a digital therapy application. In parallel with the use of the digital therapy application, an effective amount of medicine to address the condition may be administered to the user. [Brief explanation of the drawing]

[0017] The aforementioned and other purposes, aspects, features, and advantages of this disclosure will be made clearer and better understood by reading the following description in conjunction with the accompanying drawings. [Figure 1] This is a block diagram of a system for performing actions for application usage-related events detected from free text data, according to an exemplary embodiment. [Figure 2]This is a block diagram of the process for training a machine learning (ML) architecture and generative model in a system for performing actions for events, according to an exemplary embodiment. [Figure 3] This is a block diagram of a process for detecting an event from free text in a system for performing actions for an event, according to an exemplary embodiment. [Figure 4] This is a block diagram of a process for generating an electronic document relating to an event in a system for performing actions for an event, according to one exemplary embodiment. [Figure 5] This is a block diagram of a process for enforcing a policy to perform an action in a system for performing an action for an event, according to an exemplary embodiment. [Figure 6] An exemplary user interface for presenting user messages and analysis result reports regarding an event is illustrated in a system for performing actions for an event, according to one exemplary embodiment. [Figure 7] This is a flowchart illustrating a method for performing actions for application usage-related events detected from free text data, according to an exemplary embodiment. [Figure 8] This is a block diagram of a server system and a client computer system according to an exemplary embodiment. [Modes for carrying out the invention]

[0018] The following sections of the specification and their respective contents may be helpful for reading the descriptions of the various embodiments below.

[0019] Section A describes the systems and methods for taking action for events related to application usage detected from free text data.

[0020] Section B describes networks and computing environments that may be useful for implementing embodiments of the present disclosure. A. Systems and Methods for Performing Actions for Events Related to the Use of Applications Detected from Free-Text Data

[0021] The present disclosure presents systems and methods for performing actions for events related to the use of applications detected from free-text data. The events may correspond to unexpected or undesirable events that occur with the use of an application. The events may have an adverse impact on the overall performance of the application, the computing device on which the application is running, and even remote servers interfacing with the application. For example, a malfunction of a particular function of an application may result in excessive consumption of resources (e.g., processors and memory) on a computing device, and as a result, the operation of the computing device may become very slow and the response time may become long. These events also impair the user experience in the application. To mitigate the adverse effects resulting from these events, the detection of these events can be used.

[0022] To address these and other technical challenges, an application monitoring service may obtain messages from multiple sources, and a user may provide feedback about the use of an application and use machine learning techniques to classify these messages to detect the occurrence of events. These messages may include free text aggregated from various channels such as emails, messages, call center transcripts, chatbot messages, or in-app inputs. This may enable the user to input free text via more channels to provide feedback about the application. The machine learning techniques described in the present disclosure can process the free text to detect such events associated with the use of an application.

[0023] By aggregating these messages, the application monitoring service can process free text using machine learning architectures and generative models. The machine learning architecture may include a natural language processing (NLP) model and a classifier model. The NLP model may be used to access application-specific information resources (such as specific user interface elements), such as materials related to events (e.g., medical conditions and computer performance) and standard operating procedures (SOPs) of the application. The NLP model may incorporate retrieval-augmented generation (RAG) techniques using free text and query information resources related to the content of the free text.

[0024] In addition, the classifier model may be trained using examples of sample text labeled in the display of events. The examples may also be application-specific and may be previous user submissions for the same application related to events (e.g., performance problems with the application or worsening symptoms of the user's medical condition). Both the NLP model and the classifier model may be used to process free text and detect the display of any event of the application. The classifier model may use additional examples of sample text considered to indicate a harmful event and may be constantly iteratively updated via interactive learning. The classifier model can use the collected free text to determine the likelihood of the occurrence of a given event and classify which events occurred with the use of the application.

[0025] The application monitoring service can use the outputs of NLP and classifier models to generate data elements to be used as input prompts for a generative machine learning model. These data elements may include informational resources retrieved by the NLP model, the likelihood of an event occurring, and message data such as free-text portions and reception timestamps. The generative model may be trained using a large sample corpus to produce analytical reports and documentation about events. The application monitoring service can generate event reports by providing input prompts. The report may include diagnostic or analytical results about the event (e.g., a potential performance issue or a user's medical condition) and actions to address the event. In response to generating a report, the application monitoring service can take actions according to the contents of the report. Actions may include, for example, disabling a specific function of the application causing the event, notifying the application administrator, or sending the report to a third-party organization.

[0026] In this way, application monitoring services can process free text to detect events such as those associated with application usage. Furthermore, application monitoring services can take action to mitigate or counter detected events that other approaches might fail to detect. Detecting events and taking action can improve application performance, the computing devices running the application, and the user experience. In the case of digital therapy, these can also improve compliance rates, enhance health outcomes, and make digital therapy interventions more effective.

[0027] In addition, when an event is detected, information about the event, including the initial free text, may be supplied by the application monitoring service to the generative model to generate documentation about the event. By utilizing the outputs from both the NLP model and the classifier model, the likelihood of hallucination occurring on the generative model side when creating output about the event can be reduced. Hallucination refers to the generative model producing output that is contrary to the facts, logically inconsistent, or outright fabricated. Such errors significantly reduce the reliability and usefulness of documentation from the generative model used to evaluate events. A model architecture including both an NLP model and a classifier model can therefore also improve the quality of documentation output from the generative model when evaluating free text messages and selecting actions to address any displayed events.

[0028] Referring here to Figure 1, a block diagram of system 100 for taking action for application usage-related events detected from free text data is illustrated. Schematically, system 100 may include at least one application monitoring service 105, at least one user device 110, at least one administrator device 115, at least one support device 120, at least one remote device 125, etc., all interconnected via at least one network 130. The user device 110 may include at least one application 135. The application monitoring service 105 may include at least one model trainer 140, at least one message aggregator 145, at least one event evaluator 150, at least one report generator 155, at least one policy enforcer 160, at least one machine learning (ML) architecture 165, and at least one generative model 170. The ML architecture 165 may include at least one natural language processing (NLP) model 175 and at least one classifier model 180, etc. The application monitoring service 105 may include or have access to at least one database 185. The functionality of application 135 on user device 110 may be partially executed on application monitoring service 105, or vice versa.

[0029] More specifically, the application monitoring service 105 may be any computing device having one or more processors coupled to memory and software and capable of performing the various processes and tasks described herein. The application monitoring service 105 may be associated with an entity that is implementing or managing instances of application 135 running on one or more user devices 110. The application monitoring service 105 may communicate with user devices 110, administrator devices 115, support devices 120, and remote devices 125, etc. The application monitoring service 105 may be located in, situated in, or associated with at least one computer system. The computer system may correspond to a data center, branch office, or site where one or more computers corresponding to the application monitoring service 105 are located.

[0030] The application monitoring service 105 may include one or more subsystems, modules, or components for performing the various processes and tasks described herein. The model trainer 140 may initialize, train, build, and update the ML architecture 165 and generative model 170 on the application monitoring service 105 using training data. The message aggregator 145 may search, identify, or retrieve messages containing free text related to application 135 from various sources. The event evaluator 150 may use the ML architecture 165 to process the free text and detect the occurrence of events related to the use of application 135. The report generator 155 may use the generative model 170, which uses the output from the ML architecture 165, to create an electronic document about the detected events. The policy enforcer 160 may take action according to the data in the electronic document.

[0031] The ML architecture 165 may include one or more machine learning (ML) models for processing input in the form of free text and generating various outputs. The NLP model 175 may include or execute any number of NLP algorithms for processing free text. In some embodiments, the NLP model 175 may be used to perform or execute Search-Enhanced Generation (RAG) for the generation model 170. Generally, the NLP model 175 may be used to retrieve one or more informational resources from a database 185 using free text. The NLP model 175 may have at least one input and at least one output. The input may include free text or data derived from free text (e.g., tokens or embedded representations generated via tokenization). The output may include at least one informational resource identified using the input free text. The informational resource may include text determined to be related to the input free text. The NLP model 175 may use any number of algorithms to identify relevant information resources, such as term frequency-inverse document frequency (TF-IDF), vector space model (VSM), or latent semantic analysis (LSA), best matching (BM) ranking function.

[0032] The classifier model 180 may be used to process free text to determine a value indicating the likelihood of at least one event occurring in connection with the use of the application, and to classify the type of event. The architecture of the classifier model 180 may include deep learning neural networks (e.g., convolutional neural network architecture, residual network, or transformer-utilizing architecture), regression models (e.g., linear or logistic regression models), random forests, support vector machines (SVMs), clustering algorithms (e.g., k-nearest neighbors), or naive Bayes models. The classifier model 180 may be trained using supervised learning, unsupervised learning, or semi-supervised learning. In general, the classifier model 180 may have at least one input and at least one output. The input and output may be related via a set of weights. The input may include free text or data derived from free text (e.g., tokens or embedding representations generated via tokenization). The output may include a value indicating the likelihood or type of event in connection with the use of the application.

[0033] The generative model 170 may receive input and output content in one or more modalities (e.g., in the form of text strings, audio content, images, videos, or multimedia content). The input may include output from the ML architecture 165, such as the probability of an event occurring and identified information resources, and at least a portion of free text. The generative model 170 may be a machine learning model based on a transformer model (e.g., a generative pre-trained model, or a bidirectional encoder representation from a transformer). The generative model 170 may be a large language model (LLM), a text-image model, a text-speech model, or a text-video model, etc. In some embodiments, the generative model 170 may be part of the application monitoring service 105 (e.g., as shown). In some embodiments, the generative model 170 may be a separate server from the application monitoring service 105 that communicates with the application monitoring service 105 via the network 130.

[0034] The generative model 170 may include a set of weights provided across a set of layers based on a transformer architecture. In this architecture, the generative model 170 may include at least one tokenization layer (which may also be referred to herein as a tokenizer) interconnected (e.g., via forward, reverse, or skip connections), at least one input embedding layer, at least one position encoder, at least one encoder stack, at least one decoder stack, and at least one output layer, etc. In some embodiments, the transformer layer may lack an encoder stack (e.g., in the case of a decoder-only architecture) or a decoder stack (e.g., in the case of an encoder-only model architecture). The tokenization layer may transform a raw input in the form of a set of strings into a corresponding set of word vectors (which may also be referred to herein as tokens, embedding representations, or vectors) in an n-dimensional feature space. The input embedding layer may use the word vectors to generate a set of embedding representations. Each embedding representation may be a lower-dimensional representation of the corresponding word vector and may capture semantic syntactic information of the string associated with the word vector. The position encoder may generate the position encoding for each input embedding representation as a function of the position of the corresponding word vector, or by extending the strings within the input set of strings.

[0035] Next, in the generative model 170, the encoder stack may include a set of encoders. Each encoder may include at least one attention layer and at least one feedforward layer, etc. The attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each input embedding representation to indicate the degree of attention the embedding representation deserves, and may generate a weighted sum of the set of input embedding representations. The feedforward layer may apply a linear transformation using nonlinear activation (e.g., a normalized linear unit (ReLU)) to the output of the attention layer. The output may be fed to another encoder in the encoder stack within the transformer layer. If the encoder is the last encoder in the encoder stack, the output may be fed to the decoder stack.

[0036] A decoder stack may include at least one attention layer, at least one encoder-decoder attention layer, and at least one feedforward layer, etc. In a decoder stack, an attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each output embedding representation (e.g., an embedding representation generated from the target or expected output). An encoder-decoder attention layer may combine inputs from the attention layers in the decoder stack with outputs from one of the encoders in the encoder stack and calculate an attention score from the combined inputs. A feedforward layer may apply a linear transformation using nonlinear activation (e.g., a normalized linear unit (ReLU)) to the output of the encoder-decoder attention layer. The output of the decoder may be fed to another decoder in the decoder stack. If the decoder is the last decoder in the decoder stack, the output may be fed to an output layer.

[0037] The output layer of the generative model 170 may include at least one linear layer and at least one activation layer, etc. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to compute a token score. The activation layer may apply an activation function (e.g., softmax, sigmoid, or normalized linear unit) to the output of the linear function to transform the token score into a probability (or distribution). The probability may represent the likelihood of an output token occurring given an input token. The output layer may use the probability to select an output token (e.g., at least a portion of output text, image, audio, video, or multimedia content with the highest probability). This may be repeated over a set of input tokens, and the resulting set of output tokens may be used to form the output of the entire generative model 170. Although this disclosure has mainly described transformer models, the application monitoring service 105 may use other machine learning models to generate and output content.

[0038] The user device 110 (which may also be referred to in this disclosure as an end-user computing device) may be any computing device having one or more processors coupled to memory and software and capable of performing various processes and tasks described in this disclosure. The user device 110 may communicate with the application monitoring service 105, the administrator device 115, the support device 120, the remote device 125, and the database 185 via the network 130. The user device 110 may be operated by or associated with an end user using an application 135 on the user device 110. The user device 110 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., a smartwatch or smart glasses), or laptop computer. The user device 110 may be used to access the application 135. In some embodiments, the application 135 may be downloaded and installed on the user device 110 (e.g., via a digital distribution platform). In some embodiments, the application 135 may be a web application having resources accessible via the network 130.

[0039] An application 135 running on the user device 110 may be a digital therapy application and may provide sessions (which may also be referred to in this disclosure as therapy sessions) to address one or more of the user's conditions (or symptoms). The user's conditions may include, for example, chronic pain (e.g., related to or including arthritis, migraines, fibromyalgia, lower back pain, Lyme disease, endometriosis, repetitive strain injury, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin lesions (e.g., atopic dermatitis, psoriasis, dermatophyte, and eczema), cognitive impairment (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, borderline personality disorder, and attention deficit hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, smoking disorder, or hallucinogenic drug disorder), and other illnesses (e.g., narcolepsy and cancer).

[0040] The end user may be taking or prescribed an effective amount of medication to address the condition, at least in part, in parallel with the use of Application 135 (for example, over any number of sessions). For example, if the medication is for pain, the end user may be taking acetaminophen, nonsteroidal anti-inflammatory compositions, antidepressants, anticonvulsants, or other compositions. In the case of skin lesions, the end user may be taking steroids, antihistamines, or topical disinfectants. In the case of cognitive impairment, the end user may be taking cholinesterase inhibitors or memantine. In the case of mental conditions, the end user may be taking antidepressants, mood stabilizers, antipsychotics, tranquilizers, or stimulants. In the case of substance use disorders, the end user may be taking naltrexone, disulfiram, acamprosate, or nicotine replacement therapy. Application 135 may enhance the potency of any medication the user is taking to address the condition. In this disclosure, Application 135 is primarily described as a digital therapy application, but Application 135 may be any type of application, such as a word processor, spreadsheet editor, web browser, video game, social media application, multimedia player, messaging application, or mobile application.

[0041] The administrator device 115 (which may also be referred to in this disclosure as the administrator computing device) may be any computing device having one or more processors coupled to memory and software and capable of performing the various processes and tasks described in this disclosure. The administrator device 115 may communicate with the application monitoring service 105, user device 110, support device 120, remote device 125, and database 185 via the network 130. The administrator device 115 may be associated with an entity that monitors the operation, updates, or development of application 135. The administrator device 115 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch or smart glasses), or laptop computer. In some embodiments, the administrator device 115 may be separate from the application monitoring service 105 (e.g., as shown). In some embodiments, the administrator device 115 may be part of the application monitoring service 105.

[0042] The support device 120 (which may also be referred to in this disclosure as a call center computing device) may be any computing device having one or more processors coupled to memory and software and capable of performing the various processes and tasks described in this disclosure. The support device 120 may communicate with the application monitoring service 105, user device 110, administrator device 115, remote device 125, and database 185 via the network 130. The support device 120 may be accompanied by entities responsible for responding to end-user inquiries regarding application 135 or for providing treatment regimens to end users. The support device 120 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch or smart glasses), or laptop computer.

[0043] The remote device 125 may be any computing device having one or more processors coupled to memory and software and capable of performing the various processes and tasks described herein. The support device 120 may communicate with the application monitoring service 105, the user device 110, the administrator device 115, the remote device 125, and the database 185 via the network 130. The remote device 125 may be accompanied by entities that should report events related to the use of application 135. These entities may include, for example, regulatory authorities (e.g., the U.S. Food and Drug Administration (FDA), the European Medicines Agency (EMA), the UK Medicines and Healthcare Products Regulatory Agency (MHRA), the Pharmaceuticals and Medical Devices Agency (PMDA) of Japan, or the National Medical Products Administration of China (NMPA)), healthcare institutions (e.g., hospitals or private practices), device manufacturers, or pharmaceutical entities. The remote device 125 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch or smart glasses), or laptop computer. In some embodiments, the remote device 125 may be located in, situated in, or associated with at least one computer system. The computer system may correspond to a data center, branch office, or site where one or more computers corresponding to the remote device 125 are located.

[0044] Database 185 may store and maintain various resources and data related to the application monitoring service 105 and the application 135. Database 185 may include a database management system (DBMS) for arranging and organizing the data maintained thereon. Database 185 may communicate with the application monitoring service 105, user device 110, administrator device 115, support device 120, and remote device 125 via network 130. While performing various operations, the application monitoring service 105 and the application 135 may access database 185 and retrieve specific data from it. The application monitoring service 105 and the application 135 may also write data to database 185 in response to the performance of such operations.

[0045] Figure 2 illustrates a block diagram of a process 200 for training a machine learning (ML) architecture and a generative model in a system 100 for performing actions for an event. Process 200 may include, or correspond to, operations for initializing, training, and building the ML architecture 165 and the generative model 170 in system 100. In process 200, a model trainer 140 on the application monitoring service 105 may initialize, train, or build the ML architecture 165 (e.g., the illustrated classifier model 180). The ML architecture 165 may be built in particular to process text related to a given application 135. To train the classifier model 180, the model trainer 140 may search, retrieve, or identify sample datasets 205A-N (hereinafter collectively referred to as sample datasets 205) from the database 185. The sample datasets 205 may be used to train the classifier model 180.

[0046] Each sample dataset 205 may identify or include at least one sample text 210 labeled with at least one display 215, etc. The sample text 210 may include free text related to the use of the sample application. The sample application may correspond to application 135 or another application in a similar field (e.g., a digital therapy application from the same developer). The sample text 210 may be collected from previous submissions regarding user complaints or problems about the sample application. The sample text 210 may include, for example, emails, text messages (e.g., Short Message Service (SMS) or Multimedia Message Service (MMS) messages), voice transcripts (e.g., Automated Speech Recognition (ARS) transcriptions of call center calls), chatbot messages (e.g., via an in-app help interface with a chatbot), e-posts (e.g., social media postings, app review posts, or comments), or communication platform messages (e.g., messages via a messaging application).

[0047] Display 215 may identify the presence or absence of each event related to the use of the Application. Events may include, for example, adverse events (e.g., unexpected or undesirable experiences related to the use of the Application that affect the user's medical condition), serious adverse events (e.g., serious harm or life-threatening events), software bugs (e.g., occurrence of exceptions, slow response times, heavy performance), user complaints (e.g., user unfamiliarity with the technology, user settings in an undesirable interface design), or convenience issues (e.g., unresponsive functionality, lack of functionality, or suggestions for improvement). In some embodiments, events may be defined in relation to a different environment or situation. For example, events may include, for example, adverse events (e.g., unexpected or undesirable experiences related to the use of the Application that affect the user's medical condition during a clinical trial), serious adverse events (e.g., serious harm or life-threatening events during a clinical trial), incidents (e.g., unexpected or undesirable experiences related to the use of the Application that affect the user's medical condition outside of a clinical trial), or serious incidents (e.g., serious harm or life-threatening events outside of a clinical trial). Display 215 may define, specify, or identify the type of event present (or absent) in the relevant sample text 210. For example, Display 215 may specify that the sample text 210 indicates a software bug, but does not indicate an adverse event affecting the user's medical condition.

[0048] In response to the identification, the model trainer 140 may provide, supply, or apply the sample text 210 to the classifier model 180 of the ML architecture 165. In some embodiments, the model trainer 140 may generate input data for provision to the classifier model 180 using the sample text 210. The input data may include tokens (also referred to as word vectors or embedding representations) in an n-dimensional space for processing by the ML architecture 165. The model trainer 140 may process the sample text 210 (or derived input data) using a set of weights of the classifier model 180. In response to the processing, the model trainer 140 may generate at least one output value 220 for at least one corresponding event. The output value 220 may identify or indicate the likelihood of the corresponding event occurring in relation to the use of a sample application using the sample text 210. In some embodiments, the model trainer 140 may generate a set of output values ​​220 for a corresponding set of event types. In some embodiments, the model trainer 140 may determine the type of event in the sample text 210.

[0049] The model trainer 140 may calculate, generate, or determine at least one loss metric based on a comparison between the output value 220 of the sample text 210 of the sample dataset 205 and the display 215. In some embodiments, the model trainer 140 may determine the loss metric by comparing the type with the display 215. The loss metric may indicate the degree of deviation of the output value 220 from the display 215. The loss metric may be generated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean absolute error (MAE), mean squared error (MSE), quadratic loss, cross-entropy error, and Huber loss. Generally, the more inaccurate the output value 220 is compared to the display 215, the higher the loss metric may be. Conversely, the more accurate the output value 220 is compared to the display 215, the lower the loss metric may be.

[0050] The model trainer 140 may modify, change, or update at least one parameter of the classifier model 180 using a loss metric. The update of the parameters of the classifier model 180 may be performed according to an objective function. The objective function may define one or more percentages for which the parameters of the classifier model 180 should be updated. The objective function may follow stochastic gradient descent and may include, for example, adaptive moment estimation (Adam), implicit update (ISGD), and adaptive gradient algorithm (AdaGrad). The model trainer 140 may iteratively update the ML architecture 165 by iterating through a set of sample datasets 205. The update of the parameters of the classifier model 180 may be repeated until convergence occurs. In response to the completion of training, the model trainer 140 may store and maintain the set of parameters of the classifier model 180.

[0051] In addition, the model trainer 140 may initialize, train, or construct a generative model 170. In some embodiments, the model trainer 140 may fine-tune, modify, or update a previously trained generative model 170. The generative model 170 may be constructed to process text particularly relevant to a given application 135. For training, the model trainer 140 may search, retrieve, or identify a set of corpora 225A-N (hereinafter collectively referred to as corpora 225) from the database 185. In some embodiments, at least one of the corpora 225 may be a generalized dataset. For example, the generalized text for corpus 225 may be obtained from a large set of unstructured text without being concentrated in a particular knowledge domain. In some embodiments, at least one corpus 225 may be a knowledge domain-specific dataset. Each corpus 225 may identify or include at least one sample input 230 and at least one sample output 235, etc.

[0052] The sample input 230 may include or identify at least one sample data element to be used as part of the input to the generative model 170. The sample data element may include or identify at least one sample value and at least one sample information resource, etc. The sample value may indicate the likelihood of occurrence of at least one corresponding event determined from the sample free text. The sample information resource may include an information resource identified as being related to the free text. The sample information resource may include, for example, text sentences identifying user interface elements in a sample application (e.g., application 135), standard operating procedures (SOPs) for using the application, and literature on medical conditions that digital therapy addresses. In some embodiments, the sample data element of the sample input 230 may include at least one sample event type and sample message data. The sample event type may identify the type of event to which the sample free text is classified. The message data may include at least a portion of the sample free text itself and various types of information about the sample free text. The information may include a timestamp of receipt, the user's location, device type, or the user's state, etc.

[0053] Sample output 235 may include or identify predicted output data 240 from the generative model 170 when the relevant sample input 230 is applied. Sample output 235 may include a natural language description characterizing the event shown in sample input 230, and recommended actions to address the event. Sample output 235 may include or identify, for example, a diagnostic result for at least one corresponding event related to the use of the sample application, an analysis result of the event, sample actions to address the event, and one or more recommendations for the application's user. The diagnostic result may define or identify the occurrence, cause, and impact of the event in the sample application, user, and user device, etc. The analysis result may identify or include a detailed documented report of the event, such as the nature and circumstances of the event, the type of the event, the source of free text related to the event, the state of the sample application, the state of the user, the state of the user device, and a statistical descriptor of the event.

[0054] Next, in sample output 235, the sample action may identify or include corrective actions to address at least one event. The sample action may include, for example, ending the user's use of the sample application on the user device, modifying or restricting the application's behavior (e.g., a feature or function) related to the event, sending a notification to the user device or a support device, sending a report to an administrator device, sending a notification to a remote device, or storing a report related to the event. The sample recommendation may identify or include steps that the user should take to address the event. For example, the sample recommendation may include how to use a specific feature of the application (e.g., logging in or opening an icon), or a guide on how to complete a set of lessons for the user if the event is a usability issue with a feature or lesson.

[0055] In response to the identification, the model trainer 140 may construct or train a generative model 170 using a set of corpora 225. In some embodiments, the model trainer 140 may initialize the generative model 170. For example, the model trainer 140 may create an instance of the generative model 170 by applying random values ​​to the weights in the layers. In some embodiments, the model trainer 140 may use a set of corpora 225 to fine-tune a pre-trained generative model 170 (e.g., a ChatGPT model, a LLAMA model, and a stable diffusion model). For training or fine-tuning, the model trainer 140 may define, select, or identify at least a portion of each corpus 225 as a source set (e.g., at least a portion of the sample inputs 230) and at least a portion of each corpus 225 as a destination set (e.g., at least a portion of the sample outputs 235). In some embodiments, the model trainer 140 may select or identify source sets and destination sets using mappings within the corpus 225. The source sets may be used as input to the generative model 170 to generate outputs to be compared with the destination sets. Multiple parts of each corpus 225 may overlap at least partially and may correspond to subsets of text strings across sample inputs 230 and sample outputs 235.

[0056] For each corpus 225, the model trainer 140 may supply or apply strings from the source set of corpus 225 to the generative model 170. During application, the model trainer 140 may process the input strings according to a set of layers in the generative model 170. As described above, the generative model 170 may include a tokenization layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer, etc. The model trainer 140 may use the tokenization layer of the generative model 170 to process the input strings (words or phrases in alphanumeric form) of the source set to generate a set of word vectors for the input set. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a single embedding table). The model trainer 140 may apply the set of word vectors to the input embedding layer to generate a corresponding set of embedding representations. The model trainer 140 may identify the position of each string in the set of strings of the source set. In response to the identification, the model trainer 140 may apply a position encoder to the position of each string to generate a position encoding for each embedding representation corresponding to the string by extending the embedding representation.

[0057] The model trainer 140 may apply the set of embedding representations, along with the corresponding set of positional encodings generated from the input set of the corpus 225, to the encoder stack of the generation model 170. During application, the model trainer 140 may process the set of embedding representations, along with the corresponding set of positional encodings, according to the layers within each encoder in the encoder block (e.g., an attention layer and a feedforward layer). In response to the processing, the model trainer 140 may generate another set of embedding representations and feedforward them to the encoders in the encoder stack. The model trainer 140 may then supply the output of the encoder stack to the decoder stack.

[0058] In addition, the model trainer 140 may process the destination set using a separate tokenization layer of the generative model 170 to generate a set of word vectors for the destination set. The destination set may be of the same modality as the source set of corpus 225, or of a different modality than the source set of corpus 225. Each word or code vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a single embedding table). The model trainer 140 may apply the set of words or code vectors to the input embedding layer to generate a corresponding set of embedding representations. The model trainer 140 may identify the position of each string in the set of strings of the target set. In response to the identification, the model trainer 140 may apply a position encoder to the position of each string to generate a position encoding for each embedding representation corresponding to the string by extending the embedding representation.

[0059] The model trainer 140 may apply a set of embedding representations to the decoder stack of the generating model 170, along with a corresponding set of positional encodings generated from the destination set of the corpus 225. The model trainer 140 may also combine the outputs of the encoder stack during processing via the decoder stack. During application, the model trainer 140 may process the set of embedding representations along with the corresponding set of positional encodings according to the layers within each decoder in the decoder block (e.g., attention layer, encoder-decoder attention layer, feedforward layer). The model trainer 140 may combine the outputs from the encoders with the inputs of the encoder-decoder attention layer in the decoder block. In response to processing, the model trainer 140 may generate an output set of embedding representations to be fedforward to the output layer.

[0060] Next, the model trainer 140 may supply the output from the decoder block to the output layer of the generating transformer layer. During supply, the model trainer 140 may process the embedding representation from the decoder block according to the linear and activation layers of the output layer. In response to the processing, the model trainer 140 may calculate the probability of each embedding representation. The probability may represent the likelihood of the output occurring given the input tokens. Based on the probabilities, the model trainer 140 may select the output token with the highest probability (e.g., at least a portion of the sample output 235) to form, create, or generate the output data 240. The output data 240 may include sample diagnostic results, actions, analysis results, or recommendations, or any combination thereof. The output data 240 may be of the same modality as the set of subjects in the corpus 225. Although the description has mainly focused on the transformer model architecture, other architectures may be used in the generating model 170 to output content.

[0061] In response to generation, the model trainer 140 may compare the output data 240 from the generation model 170 with the destination set of the corpus 225 used to generate the output data 240. The comparison may be between the probabilities (or destinations) of various tokens in the content of the output data 240 and the probabilities of tokens in the target set of the corpus 225. For example, the model trainer 140 may determine the difference between the probability distribution of the output data 240 and the target set of the corpus 225 to be compared. The probability distribution may identify the probability of each candidate token in the output data 240 or the token in the target set of the corpus 225. Based on the comparison, the model trainer 140 may calculate, determine, or generate a loss metric. The loss metric may indicate the degree of deviation of the output data 240 from the expected output defined by the target set of the corpus 225 used to generate the output data 240. The loss metric may be calculated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), quadratic loss, cross-entropy error, and Huber loss.

[0062] In some embodiments, the model trainer 140 may determine a loss metric for the output data 240 based on data retrieved from the database 185. In making this determination, the model trainer 140 may calculate similarity by comparing the content of the output data 240 with a destination set (e.g., at least a portion of the sample output 235). Similarity may measure, correspond to, or indicate, for example, a level of code similarity (e.g., using a knowledge map in the case of comparing the sample output 235 with the output data 240). Generally, a higher loss metric means the generated output may deviate more significantly from the expected output corresponding to the destination set derived from the corpus 225. Conversely, a lower loss metric means the generated output may deviate less significantly from the expected output derived from the destinations. The loss metric may be calculated to train the generative model 170 to produce output content with a higher probability of accurately generating the output.

[0063] The model trainer 140 may update one or more weights in the set of layers of the generative model 170 using a loss metric. The weight update may be performed according to a backpropagation and optimization function (which may also be referred to in this disclosure as an objective function) using one or more parameters (e.g., learning rate, momentum, weighted decay, and number of iterations). The optimization function may define one or more parameters for which the weights of the generative model 170 should be updated. The optimization function may follow a stochastic gradient descent method and may include, for example, adaptive moment estimation (Adam), implicit update (ISGD), and adaptive gradient algorithm (AdaGrad). The model trainer 140 may repeatedly train the generative model 170 until it converges. In response to convergence, the model trainer 140 may store and maintain a set of weights for the set of layers of the generative model 170 for use in the estimation phase.

[0064] Figure 3 is a block diagram of a process 300 for detecting events from free text in a system 100 for performing actions for events. Process 300 may include, or correspond to, operations in system 100 for evaluating free text in messages from users associated with application use. In process 300, a message aggregator 145 running on the application monitoring service 105 may search, receive, or identify at least one free text 305 related to application 135. The free text 305 may be identified for evaluating one or more events related to the use of application 135. The free text 305 may include, or have, a column of alphanumeric characters, independently of a predefined data structure.

[0065] Free text 305 may be obtained from a variety of sources, such as email (e.g., received at a customer support center), text messages (e.g., Short Message Service (SMS) or Multimedia Message Service (MMS) messages), voice transcripts (e.g., Automated Speech Recognition (ARS) transcripts of call center calls), chatbot messages (e.g., via an in-app help interface with a chatbot), electronic posts (e.g., social media postings, app review posts, or comments), or communication platform messages (e.g., messages via a messaging application). Free text 305 may correspond to alphanumeric characters that do not have a predefined data structure (e.g., pairs of field values). Free text 305 may include alphanumeric characters in an unstructured format. In some embodiments, free text 305 may correspond to text from elements within a structured dataset without defining a format. For example, free text 305 may correspond to a portion of text within an inline frame on a web page using Hypertext Markup Language (HTML).

[0066] In some embodiments, the message aggregator 145 may identify or retrieve free text 305 from defined data sources via the network 130. The data sources may include, for example, a website for mobile application reviews (e.g., by user-created electronic posts) or a distribution service for downloading applications (including application 135). Each data source may be identified via an address (e.g., a Uniform Resource Locator (URL)). The address may be provided by the system administrator of application 135 (e.g., via administrator device 115). The message aggregator 145 may use the address to access the corresponding data source and retrieve or identify the free text 305. For example, the message aggregator 145 may access a website or distribution service hosting mobile application reviews and retrieve electronic posts containing user reviews of application 135.

[0067] In some embodiments, the message aggregator 145 may identify the free text 305 from the application 135 on the user device 110. The message aggregator 145 may build at least one event listener on the application 135 to monitor the generation of the free text 305 by the user of the application 135 on the user device 110. For example, the event listener may receive text entered by the user into the user interface for feedback about the application 135. In response to the reception, the event listener on the application 135 may provide the input to the message aggregator 145 as free text 305. The message aggregator 145 may look up, fetch, or retrieve the free text 305 via the application programming interface (API) of the event listener on the application 135. The API may allow various functions of the application 135 to be called from outside the application 135. In some embodiments, the message aggregator 145 may receive the free text 305 from the support device 120 (or other device). For example, the free text 305 may correspond to a call transcript generated using automatic speech recognition (ASR) while an employee using the support device 120 is on a phone call with a user of application 135.

[0068] In response to the identification, the message aggregator 145 may determine whether the free text 305 is related to the use of application 135. To make this determination, the message aggregator 145 may use a natural language processing (NLP) algorithm (e.g., keyword extraction, information extraction, or text mining) to extract or identify one or more keywords from the free text 305 related to the use of application 135. For example, the message aggregator 145 may use an NLP algorithm to detect keywords that relate to a state or symptoms associated with a state that should be addressed by application 135. If at least one keyword is detected, the message aggregator 145 may determine that the free text 305 is related to the use of application 135. In addition, the message aggregator 145 may forward the free text 305 to the event evaluator 150 for further processing. Alternatively, if no keywords are detected, the message aggregator 145 may exclude the free text 305 from further processing.

[0069] An event evaluator 150 running on the application monitoring service 105 may provide, supply, or apply free text 305 to an ML architecture 165, which includes an NLP model 175 and a classifier model 180. In some embodiments, the event evaluator 150 may generate input data using the free text 305 for provision to the ML architecture 165. The input data may have tokens (which may also be called word vectors or vectors) in an n-dimensional space for processing by the ML architecture 165. When applying the free text 305 to the NLP model 175, the event evaluator 150 may create, determine, or generate at least one query 310 to search for a set of information resources 315A to N (hereinafter collectively referred to as information resources 315) on a database 185 (or another data source). The query 310 may be generated using at least a portion of the free text 305. In some embodiments, the event evaluator 150 may use query expansion (QE) algorithms such as vector space expansion, latent semantic analysis, knowledge graphs, statistical expansion, or synonym expansion.

[0070] The event evaluator 150 may use the query 310 and the NLP model 175 to discover, select, or identify at least one information resource 315' from a set of information resources 315. The set of information resources 315 may be related to the application 135. Each information resource 315 may contain content (e.g., in the form of text, tokens, vectors, or embedding representations) related to various aspects of the use of the application 135. For example, an information resource 315 may include sentences on individual user interfaces available through the application 135, standard operating procedures (SOPs) for operating the application 135 through various devices, conditions to be addressed through the application 135, or previous submissions by other users related to the use of the application 135. The NLP model 175 may use various functions, such as term frequency-inverse document frequency (TF-IDF), vector space model (VSM), latent semantic analysis (LSA), or best matching (BM) ranking functions, to select at least one information resource 315' related to the query 310. Information resource 315' may be used to perform Search Enhancement Generation (RAG) on the generation model 170.

[0071] In addition, based on the application of the free text 305 to the classifier model 180 of the ML architecture 165, the event evaluator 150 may calculate, generate, or determine at least one value 320. In the application, the event evaluator 150 may process the free text 305 according to the set of weights of the classifier model 180 to generate the value 320. The value 320 may identify, define, or indicate the likelihood of at least one event occurring related to the use of application 135. In some embodiments, the value 320 may identify, define, or indicate the risk of the event related to the use of application 135. The value 320 can be a number in any range, such as 0 to 1, 0 to 100, -1 to 1, or -100 to 100. Generally, a larger value 320 indicates a higher likelihood that the corresponding event is occurring while using application 135. Conversely, a smaller value 320 indicates a lower likelihood that the corresponding event is occurring while using application 135. In some embodiments, the event evaluator 150 may determine a set of values ​​320 corresponding to the event type in response to the application of free text 305 to the classifier model 180.

[0072] In some embodiments, the event evaluator 150 may generate, determine, or classify an event type 325 of an event based on the application of free text 305 to a classifier model 180. The event type 325 may identify the event as at least one of the following: an adverse event (e.g., an unexpected or undesirable experience related to the use of an application that affects a user's medical condition), a serious adverse event (e.g., a serious harm or life-threatening event), a software bug (e.g., an exception, slow response time, or heavy performance), a user complaint (e.g., a user being unfamiliar with the technology, or user settings in an undesirable interface design), or a convenience issue (e.g., a feature being unresponsive, lacking a feature, or a suggestion for improvement). In some embodiments, the event may be defined in relation to a different environment or situation. For example, events may include, for instance, adverse events (e.g., unexpected or undesirable experiences related to the use of an application that affect a user's medical condition during a clinical trial), serious adverse events (e.g., serious harm or life-threatening events during a clinical trial), incidents (e.g., unexpected or undesirable experiences related to the use of an application that affect a user's medical condition outside of a clinical trial), or serious incidents (e.g., serious harm or life-threatening events outside of a clinical trial). The event evaluator 150 may classify event types 325 using a set of values ​​320 generated for the event type set. The event evaluator 150 may select or identify event types 325 corresponding to the highest value 320. For example, if the value 320 for an adverse event with a particular symptom is the highest, the event evaluator 150 may classify event type 325 as an adverse event with a defined symptom related to the use of application 135.

[0073] The event evaluator 150 may use the output from the ML architecture 165 to create, construct, or generate at least one data element 335. The data element 335 may identify or include one or more of the following: an information resource 315', a value 320, an event type 325, and message data 330. The message data 330 may include, for example, at least a portion of the free text 305 and metadata associated with the free text 305. The metadata may include, for example, a timestamp associated with the message containing the free text 305 (e.g., a generation or reception timestamp), user identification information, application instance identification information, device identification information, device location, or a state to be addressed by the application 135. The data element 335 may be a data structure (e.g., a table, matrix, linked list, tree, array, or class) for including one or more of the following: an information resource 315', a value 320, an event type 325, and message data 330. The data element 335 may be used as input to the generative model 170.

[0074] Figure 4 is a block diagram of a process 400 in a system 100 for performing actions for events, which generates an electronic document about an event. Process 400 may include or correspond to an operation in system 100 for evaluating free text in user messages associated with application use. In process 400, a report generator 155 running on an application monitoring service 105 may identify or determine whether a value 320 indicating the likelihood of a corresponding event meets a threshold. The threshold may define a value that the report generator 155 should provide as input to a generation model 170 in order to create a report about the detected event. In some embodiments, the report generator 155 may repeatedly compare a set of values ​​320 corresponding to the event type with the threshold.

[0075] If value 320 does not meet the threshold (for example, is below the threshold), report generator 155 may identify or determine that no event has occurred. If none of the set of values ​​320 meet the threshold, report generator 155 may determine that no event has occurred. Report generator 155 may also refrain from further processing of the free text 305 or data element 335. Report generator 155 may generate a display indicating that no event related to the use of application 135 exists in the free text 305. Report generator 155 may transmit, send, or provide the display along with the free text 305 to administrator device 115. On the other hand, if value 320 meets the threshold (for example, is greater than or equal to the threshold), report generator 155 may determine that a corresponding event related to the use of application 135 has occurred. If at least one of the set of values ​​320 meets the threshold, report generator 155 may determine that a corresponding event has occurred. The report generator 155 may also identify event types 325 corresponding to values ​​320 that meet the threshold. The report generator 155 may continue processing the free text 305 and data elements 335.

[0076] The report generator 155 may create, prepare, or generate at least one model input 405 (which may also be referred to in this disclosure as a prompt) to be used as input to the generation model 170 based on at least a portion of the data element 335. The generation of the model input 405 may be performed in response to a determination that the value 320 meets a threshold. The model input 405 may be based on one or more of the following: free text 305, information resource 315', value 320, event type 325, and message data 330. In some embodiments, the model input 405 may be based on at least a portion of the free text 305 and the value 320. In some embodiments, the report generator 155 may prepare the model input 405 to include contextual information according to a template. The template may include a set of predefined strings and a set of placeholders. The set of predefined strings may include a command or instruction that requests the generation model 170 to produce a specific type of output, such as the text string "Prepare a report about this [event type] using information...". A set of placeholders may be used to include information from data element 335. Contextual information may be derived from one or more of the following: free text 305, information resource 315', value 320, event type 325, and message data 330. Contextual information may include, for example, further information about the user, application 135, and the condition to be addressed.

[0077] The report generator 155 may supply, apply, or provide the model input 405 to the generating model 170. During application, the report generator 155 may process the model input 405 using a set of layers within the generating model 170. As described above, the generating model 170 may include a tokenization layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer, etc. The report generator 155 may use the tokenization layer of the generating model 170 to process the input string (code in alphanumeric form) of the model input 405 to generate a set of word vectors (which may also be referred to in this disclosure as word tokens or tokens) for the input set. Each word vector may be a vector representation of at least one corresponding input (e.g., a portion of data element 335) in an n-dimensional feature space (using a word embedding table).

[0078] The report generator 155 may apply a set of word vectors to the input embedding layer to generate a corresponding set of embedding representations. The report generator 155 may identify the position of each string in the set of strings of the model input 405. In response to the identification, the report generator 155 may apply a position encoder to the position of each string to generate a position encoding for each embedding representation corresponding to the string by extending the embedding representation. The report generator 155 may apply the set of embedding representations, along with the corresponding set of position encodings generated from the model input 405, to the encoder stack of the generated model 170. During application, the report generator 155 may process the set of embedding representations, along with the corresponding set of position encodings, according to the layers in each encoder in the encoder stack (e.g., an attention layer and a feedforward layer). In response to the processing, the report generator 155 may generate another set of embedding representations and feedforward them to the encoders in the encoder stack. The report generator 155 may then supply the output of the encoder stack to the decoder stack.

[0079] In addition, the report generator 155 may take an input (which may be referred to in this disclosure as a start token) using another tokenization layer of the generative model 170 and generate one corresponding word vector. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (for example, using a word embedding table). The report generator 155 may apply the set of word vectors to an input embedding layer to generate a corresponding set of embedding representations. The report generator 155 may identify the position of each string in the set of strings of the set in question. In response to the identification, the report generator 155 may apply a position encoder to the position of each string to generate a position encoding for each embedding representation corresponding to the string by extending the embedding representation.

[0080] The report generator 155 may apply a set of embedding representations together with a corresponding set of positional encodings generated from the decoder stack of the generation model 170. The report generator 155 may also combine the outputs of the encoder stack when processing via the decoder stack. When applying, the report generator 155 may process the set of embedding representations together with a corresponding set of positional encodings according to the layers within each decoder in the decoder block (e.g., attention layer, encoder-decoder attention layer, feedforward layer). The report generator 155 may combine the output from the encoder with the input of the encoder-decoder attention layer in the decoder block. In response to processing, the report generator 155 may generate an output set of embedding representations that should be fed forward to the output layer.

[0081] Next, the report generator 155 may supply the output from the decoder block to the output layer of the generating transformer layer. When supplying, the report generator 155 may process the embedded representation from the decoder block according to the linear and activation layers of the output layer. In response to the processing, the report generator 155 may calculate the probability of each embedded representation. The probability may represent the likelihood of an output occurring given the input tokens. Based on the probabilities, the report generator 155 may select the output tokens (e.g., the diagnostic result, analysis result, action, or recommendation with the highest probability) to form, create, or generate a portion of the output. The report generator 155 may repeat the above processing using the layers of the generating model 170 to form the entire output.

[0082] In response to the provision of model input 405 to the generation model 170, the report generator 155 may create, generate, or retrieve data for at least one electronic document 410 (which may also be referred to in this disclosure as a report or documentation). The electronic document 410 may characterize events related to the use of application 135. The electronic document 410 may include or identify at least one action 415 and one or more event information 420, etc. The electronic document 410 may include a natural language text description of the action 415 and the event information 420. Action 415 may be selected from one or more of the following: for example, terminating the use of application 135 on user device 110; modifying or restricting behavior (e.g., features or functions) within application 135 related to the event; sending a notification to user device 110 or support device 120; sending an electronic document 410 to administrator device 115; sending an electronic document 410 to remote device 125; or storing the electronic document 410 related to the event in database 185. In some embodiments, the electronic document 410 may include or identify one or more recommendations for the event. The recommendations may include steps that a user of application 135 should take to address the event. For example, the recommendations in electronic document 410 may include a set of steps for a user to find a specific user interface element (e.g., a button) within the graphical user interface of application 135.

[0083] Event information 420 may include information derived from parts of data element 335, such as value 320, event type 325, and message data 330. Event information 420 may identify or include diagnostic results of an event related to the use of application 135. Diagnostic results may define or identify the occurrence, cause, or impact of the event in application 135, user, and user device 110. For example, the diagnostic results of event information 420 for electronic document 410 may include text beginning with "The cause of the application slowdown is estimated to be due to the type of smartphone being used to play the auditory stimuli (probability 0.8)." Event information 420 may identify or include analysis results of an event related to the use of application 135. Analysis results may identify or include a detailed documented report of the event, such as the nature and circumstances of the event, the type of event, the source of free text related to the event, the state of the application, the state of the user device 110, and a statistical descriptor of the event. For example, the analysis result of event information 420 in electronic document 410 may include text that begins with, "The event occurred at approximately 3:30:45 PM (Eastern Standard Time) on March 15, 2025." At this point, the user is experiencing nausea while using the application.

[0084] Figure 5 is a block diagram of a process 500 that enforces a policy to take action in system 100 for taking action for an event. Process 500 may include or correspond to an operation in system 100 for taking action using data from a report. In process 500, a policy enforcer 160 running on application monitoring service 105 may perform, execute, or implement at least one action using data for electronic document 410. The policy enforcer 160 may perform action 415 according to the specifications in electronic document 410. In some embodiments, the policy enforcer 160 may identify or select the action 415 to be performed from a set of actions according to the data in electronic document 410.

[0085] To make a selection, the policy enforcer 160 may parse the electronic document 410 and extract or identify an action 415 from the contents of the electronic document 410. The policy enforcer 160 may use natural language processing (NLP) algorithms to parse regular expression templates, named entities, lexical analyzers, or pattern matching, etc. For example, the policy enforcer 160 may use a regular expression template to identify a fixed string "recommended action" and then identify a specific action 415 such as "lower application brightness". In response to the selection, the policy enforcer 160 may execute the action 415. The execution of action 415 may be automatic (e.g., without approval) or in response to approval (e.g., acceptance by the system administrator of application 135).

[0086] If action 415 is to terminate the use of application 135, policy enforcer 160 may generate at least one instruction 505 commanding that application 135 be terminated on user device 110. Instruction 505 may specify that user device 110 restrict or disable the execution of application 135 on user device 110, or uninstall application 135 from user device 110. Instruction 505 may identify an instance of application 135, user device 110, or the user of user device 110, etc. In response to generation, policy enforcer 160 may send, transmit, or provide instruction 505 to user device 110. In response to receipt, user device 110 may stop, terminate, or end the use of application 135 on user device 110.

[0087] If action 415 is to modify a specific function of application 135, policy enforcer 160 may generate an instruction 505 that commands application 135 to change, alter, or modify the function as specified by action 415. Instruction 505 may also specify that application 135 change the behavior of a specific function of application 135. The modification may include, for example, changing the order in which user interface (or screen) elements are presented, moving the position of user interface elements within the user interface, or changing the timing at which a specific function is performed by application 135. In response to generation, policy enforcer 160 may send, transmit, or provide instruction 505 to user device 110. In response to receipt, user device 110 may modify the function of application 135 in accordance with instruction 505.

[0088] If action 415 is to restrict a particular function of application 135, policy enforcer 160 may generate an instruction 505 that commands the application 135 to disable, deactivate, or restrict the function as specified by action 415. Instruction 505 may specify that application 135 should disable, prevent, or restrict the execution of a particular function. For example, instruction 505 may disable the presentation of a particular lesson through application 135, deactivate the function to retrieve specific information, or restrict access to a function during a given time (e.g., afternoon). In response to generation, policy enforcer 160 may send, transmit, or provide instruction 505 to user device 110. In response to receipt, user device 110 may restrict the function of application 135 in accordance with instruction 505.

[0089] If action 415 is to send a notification to be presented to the user, the policy enforcer 160 may generate an instruction 505 that commands the presentation of at least a portion of the electronic document 410. The instruction 505 may include one or more recommendations, diagnostic information, or analysis information from the electronic document 410. The instruction 505 may specify whether the notification should be received and presented to a user of application 135, a user of user device 110, or a user of support device 120 of application 135. In response to generation, the policy enforcer 160 may send, transmit, or provide the instruction 505 to user device 110 (or support device 120) in accordance with the instruction 505. In response to receipt, user device 110 (or support device 120) may display, render, or present the notification. For example, user device 110 may present a message box within the user interface of application 135 to provide a step-by-step guide through the lessons presented via application 135.

[0090] If action 415 is to provide notification 510 to administrator device 115, policy enforcer 160 may generate at least one notification. Notification 510 may include at least a portion of an electronic document 410, such as action 415, event information 420 (e.g., diagnostic or analysis results), or recommendations. Notification 510 may also include the original free text 305 and at least a portion of data elements 335, such as a value 320, event type 325, and message data 330. In response to generation, policy enforcer 160 may send, transmit, or provide notification 510 to administrator device 115. In response to receipt, administrator device 115 may render, display, or present notification 510. Using the information in notification 510, a system administrator may determine what corrective actions should be taken to address the occurrence of an event across one or more instances of application 135. An example of displaying notification 510 via a user interface on administrator device 115 is shown in Figure 6.

[0091] If action 415 is to provide a report to the remote device 125, the policy enforcer 160 may generate at least one report 515. The report 515 may be notified to or made known to an entity other than the entity that is performing or managing the operation of application 135. The report 515 may include at least a portion of the electronic document 410, such as event information 420 (e.g., diagnostic or analysis results). In some embodiments, the policy enforcer 160 may determine whether to send the report 515 to the remote device 125 based on reporting criteria. The reporting criteria may identify or define the conditions under which the report 515 should be provided to the remote device 125 (using at least a portion of the electronic document 410). For example, the reporting criteria may specify that if the event type 325 of an event is a critical adverse event, the corresponding report 515 should be sent to the remote device 125.

[0092] In making a decision, the policy enforcer 160 may identify or determine whether the event (or event type 325 or other data) meets the reporting criteria. If the event meets the reporting criteria, the policy enforcer 160 may transmit, provide, or send the report 515 to the remote device 125. Otherwise, if the event does not meet the reporting criteria, the policy enforcer 160 may refrain from sending the report 515 to the remote device 125. In some embodiments, the policy enforcer 160 may send the report 515 to the remote device 125 independently of the reporting criteria. For example, if the specified action 415 is to send a report to the remote device 125, the policy enforcer 160 may generate and send the report 515 to the remote device 125. In response to receipt, the remote device 125 may store and maintain the report 515 on a data repository. The remote device 125 may also display, render, or present the report 515 via a user interface.

[0093] If action 415 is to store a report, the policy enforcer 160 may store and maintain the electronic document 410 on the database 185. The electronic document 410 may be stored in relation to a user, user device 110, application 135, or free text 305, etc. The electronic document 410 may be maintained on the database 185 using any type of data structure such as a table, matrix, array, linked list, tree, or heap. The policy enforcer 160 may store the electronic document 410 on the database 185 for record-keeping purposes. The electronic document 410 may be retrieved later (e.g., by a system administrator) for the diagnosis or analysis of application 135. In some embodiments, the policy enforcer 160 may store and maintain the electronic document 410 for later transmission to a remote device 125. For example, the policy enforcer 160 may store electronic documents 410 for various events and, in response to a request, transmit the electronic documents 410 in bulk to the remote device 125 as part of a reporting procedure.

[0094] In some embodiments, the policy enforcer 160 may search for, identify, or receive feedback 520 related to the electronic document 410. The feedback 520 may be received via a user interface. For example, the user interface may be used to present a notification 510 (including a portion of the electronic document 410) and to receive input for the feedback 520 on the administrator device 115. In some embodiments, the feedback 520 may define or identify an updated value indicating the likelihood of an event occurring related to the use of application 135. In some embodiments, the feedback 520 may identify a corrected event type of the event related to the use of application 135.

[0095] Based on feedback 520, the policy enforcer 160 may modify, change, or update one or more of the set of weights in the ML architecture 165. Weight updates may be similar to weight updates as part of training in process 200 detailed in this disclosure. For example, the policy enforcer 160 (or model trainer 140) may calculate, generate, or determine at least one loss metric based on a comparison of the output value 320 with feedback 520. The loss metric may indicate the degree of deviation of the value 320 from feedback 520. The loss metric may be generated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), mean absolute error (MAE), quadratic loss, cross-entropy error, and Huber loss. Using the loss metric, the model trainer 140 may modify, change, or update at least one parameter of the classifier model 180.

[0096] In this manner, the application monitoring service 105 can process free text 305 to detect events occurring during application use. The application monitoring service 105 can indirectly detect further events affecting application performance and user experience through the free text 305. Furthermore, the application monitoring service 105 can perform actions 415 to counter or address detected events that might not be detectable by other approaches. By using the NLP model 175 and classifier model 180 of the ML architecture 165 in combination with the generative model 170, the likelihood of hallucination occurring in the electronic document 410 characterizing the events can be reduced, improving the quality and reliability of the documentation of these detected events. The detection of such events and the implementation of actions 415 can significantly improve the performance of application 135, the user devices 110 running on application 135, and the user experience. In the case of digital therapy, both of these can also improve compliance rates, improve health outcomes, and make digital therapy interventions more effective.

[0097] Figure 6 illustrates an exemplary user interface 600 for presenting user messages and analysis reports about an event in a system for taking action for an event. User interface 600 may correspond to a graphical user interface presented via an administrator device 115. User interface 600 may include any number of user interface elements, such as a message element 605, a report element 610, a rejection element 615, and an approval element 620. Message element 605 may include free text entered by user "XYZ". The free text may be received as part of an email to a support desk and may describe the user's experience with a specific interface element in application 135 (e.g., "animated bird icon") (e.g., "felt nauseous").

[0098] The report element 610 may display a portion of the electronic document 410 generated by the generation model 170. The report element 610 may include a list of candidate event type categories and the probability of each occurring. Each candidate event type 325 (e.g., a user interface problem, a network connectivity problem, or a worsening medical condition) may correspond to one of the event type categories based on the output of the ML architecture 165. Each probability may correspond to a value indicating likelihood based on the output of the ML architecture 165. The report element 610 may also include one or more actions to be taken to address or mitigate an event related to the use of application 135. In addition, on the user interface 600, a rejection element 615 may be used by the system administrator to refrain from taking action on a detected event. In response to interaction with the rejection element 615, the policy enforcer 160 may refrain from taking action. An approval element 620 may be used by the system administrator to take recommended action. In response to interaction with the approval element 620, the policy enforcer 160 may take action.

[0099] Figure 7 is a flowchart of Method 700 for performing actions for events related to application usage detected from free text data. Method 700 may be implemented or executed using any of the components detailed in this disclosure, such as Application Monitoring Service 105 or System 800. In Method 700, a computing system may receive free text related to the user of the application (705). The computing system may apply a machine learning (ML) architecture to the free text (710). The computing system may determine the probability value of an event based on the application of the ML architecture to the free text (715). The computing system may use at least a portion of the free text to identify an information resource (720). The computing system may determine whether the value meets a threshold (725). If the value does not meet a threshold (e.g., is below the threshold), the computing system may determine the absence of an event (730). On the other hand, if the value meets a threshold (e.g., is greater than or equal to the threshold), the computing system may provide input to a generative model based on the value and the information resource (735). The computing system may generate a report in response to the application of input to the generative model (740). The computing system may use the data from the report to take action (745). B. Network and computing environment

[0100] Various operations described in this disclosure may be implemented on a computer system. Figure 8 shows a schematic block diagram of a typical server system 800, a client computing system 814, and a network 826 that can be used to implement some embodiments of this disclosure. In various embodiments, the server system 800 or a similar system may implement the services or servers or parts thereof described in this disclosure. The client computing system 814 or a similar system may implement the clients described in this disclosure. System 100 described in this disclosure may be similar to the server system 800. The server system 800 may have a modular design incorporating a plurality of modules 802 (e.g., blades in a blade server embodiment). Two modules 802 are shown, but any number may be provided. Each module 802 may include a processing unit 804 and local storage 806.

[0101] The processing unit 804 may include a single processor having one or more cores, or multiple processors. In some embodiments, the processing unit 804 may include a general-purpose primary processor and one or more dedicated coprocessors, such as a graphics processor or a digital signal processor. In some embodiments, some or all of the processing unit 804 may be embodied using customized circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). In some embodiments, such an integrated circuit executes instructions stored in the circuitry itself. In other embodiments, the processing unit 804 may execute instructions stored in local storage 806. Any combination of any type of processors may be included in the processing unit 804.

[0102] The local storage 806 may include volatile storage media (e.g., DRAM, SRAM, SDRAM, etc.) and / or non-volatile media (e.g., magnetic disks or optical disks, flash memory, etc.). The storage media incorporated into the local storage 806 may be fixed, removable, or updatable, as needed. The local storage 806 may be physically or logically divided into various subunits such as system memory, read-only memory (ROM), and a permanent storage device. The system memory may be a read-write memory device or a volatile read-write memory such as dynamic random access memory. The system memory may store some or all of the instructions and data required by the processing unit 804 at runtime. The ROM may store static data and instructions required by the processing unit 804. The permanent storage device may be a non-volatile read-write memory device capable of storing instructions and data even when the module 802 is not powered on. As used in this disclosure, the term “storage medium” includes any medium in which data can be stored indefinitely (without being overwritten, subjected to electrical interference, power outages, etc.), but does not include carrier waves or transient electronic signals propagated wirelessly or via wired communications.

[0103] In some embodiments, local storage 806 may store one or more software programs to be executed by processing unit 804, such as operating systems and / or programs that embody various server functions, such as functions of system 100 or any other system described herein, or functions of any other server related to system 100 or any other system described herein.

[0104] "Software" generally refers to a sequence of instructions that, when executed by the processing unit 804, cause the server system 800 (or a part thereof) to perform various operations, and thus define one or more specific machine embodiments that perform and execute the operations of the software program. The instructions may be stored as firmware resident in read-only memory and / or as program code stored in a non-volatile storage medium that can be loaded into volatile working memory for execution by the processing unit 804. The software may be embodied as a single program or as a collection of separate programs or program modules that interact as needed. The processing unit 804 may obtain program instructions to be executed and data to be processed in order to perform the various operations described above from local storage 806 (or the non-local storage described below).

[0105] In some server systems 800, multiple modules 802 may be interconnected via a bus or other interconnection 808 to form a local area network that facilitates communication between the modules 802 and other components of the server system 800. The interconnection 808 may be implemented using various technologies, including server racks, hubs, routers, and the like.

[0106] The wide area network (WAN) interface 810 may provide data communication capabilities between the local area network (for example, via interconnect 808) and a network 826 such as the Internet. Other technologies, including wired technology (e.g., Ethernet, IEEE 802.3 standard) and / or wireless technology (e.g., Wi-Fi, IEEE 802.11 standard), may be used to connect the server system 800 to the network 826 in a communicative manner.

[0107] In some embodiments, local storage 806 is intended to provide working memory for processing unit 804 to provide rapid access to programs and / or data to be processed, while reducing traffic on interconnect 808. One or more mass storage subsystems 812 connectable to interconnect 808 may provide storage for larger amounts of data on the local area network. The mass storage subsystem 812 may be based on magnetic data storage media, optical data storage media, semiconductor data storage media, or other data storage media. Direct-attached storage, storage area networks, network-attached storage, etc., may be used. Any data store or other collection of data described in this disclosure as created, used, or maintained by a service or server may be stored within the mass storage subsystem 812. In some embodiments, additional data storage resources may be accessible via WAN interface 810 (potentially with higher latency).

[0108] The server system 800 may operate in response to requests received via the WAN interface 810. For example, one of the modules 802 may perform a monitoring function in response to a received request and assign individual tasks to other modules 802. Work allocation techniques may be used. As the request is processed, the results may be returned to the requester via the WAN interface 810. Such operations may be largely automated. Furthermore, in some embodiments, the WAN interface 810 may interconnect multiple server systems 800 to provide a scalable system capable of managing a large volume of activity. Other techniques for managing server systems and server farms (collections of collaborating server systems), including dynamic resource allocation and reallocation, may be used.

[0109] The server system 800 may interact with various user-owned or user-operated devices via a wide-area network such as the Internet. An example of a user-operated device is illustrated in Figure 8 as a client computing system 814. The client computing system 814 may be embodied as a consumer device such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch, glasses), desktop computer, or laptop computer. For example, the client computing system 814 may communicate via a WAN interface 810. The client computing system 814 may include computer components such as a processing unit 816, a storage device 818, a network interface 820, a user input device 822, and a user output device 824. The client computing system 814 may be a computing device embodied in various form factors such as a desktop computer, laptop computer, tablet computer, smartphone, other mobile computing device, or wearable computing device.

[0110] The processing unit 816 and the storage device 818 may be the same as the processing unit 804 and local storage 806 described above. A suitable device may be selected based on the requirements imposed on the client computing system 814. For example, the client computing system 814 may be implemented as a "thin" client with limited processing power, or as a high-performance computing device. The client computing system 814 may be provided with program code executable by the processing unit 816 to enable various interactions with the server system 800.

[0111] The network interface 820 may provide a connection to a network 826, such as a wide area network (e.g., the Internet), to which the WAN interface 810 of the server system 800 is also connected. In various embodiments, the network interface 820 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0112] The user input device 822 may include any device (or more devices) through which the user can supply signals to the client computing system 814. The client computing system 814 may interpret these signals as representing specific user requests or information. In various embodiments, the user input device 822 may include at least one of the following: a keyboard, touchpad, touchscreen, mouse, or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, etc.

[0113] The user output device 824 may include any device through which the client computing system 814 can provide information to the user. For example, the user output device 824 may include a display-to-display image generated by or delivered to the client computing system 814. The display may include various image generation technologies, such as liquid crystal displays (LCDs), light-emitting diodes (LEDs) including organic light-emitting diodes (OLEDs), projection systems, and cathode ray tubes (CRTs), together with supporting electronics (e.g., digital-to-analog converters, analog-to-digital converters, signal processors, etc.). Some embodiments may include a device such as a touchscreen that functions as both an input and output device. In some embodiments, other user output devices 824 may be provided in addition to, or instead of, the display. Examples include indicator lights, speakers, haptic "display" devices, and printers.

[0114] Some embodiments include electronic components such as microprocessors, storage, and memory that store computer program instructions in a computer-readable storage medium. Many of the features described herein can be implemented as processes identified as a set of program instructions coded on a computer-readable storage medium. When one or more processing units execute these program instructions, these program instructions cause the processing units to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as that created by a compiler, as well as files containing higher-level code that is executed using an interpreter by a computer, electronic component, or microprocessor. Processing units 804 and 816 may, by preferred programming, provide functions for server system 800 and client computing system 814, including any or other functions described herein as being executed by a server or user.

[0115] It will be understood that the server system 800 and client computing system 814 are illustrative and subject to change and modification. Computer systems used in combination with embodiments of the present disclosure may have other capabilities not specifically described herein. Furthermore, it should be understood that although the server system 800 and client computing system 814 are described with reference to certain blocks, these blocks are defined for illustrative purposes only and are not intended to imply any specific physical arrangement of components. For example, different blocks may, but are not required, reside in the same facility, in the same server rack, or on the same motherboard. Furthermore, the blocks do not need to correspond to physically discrete components. Blocks may be configured to perform various operations, for example, by programming a processor or by providing appropriate control circuits, and the various blocks may be reconfigurable or not, depending on how the initial configuration was obtained. Embodiments of the present disclosure may be realized in various devices, including electronic devices embodied using various combinations of circuitry and software.

[0116] While this disclosure has been described in relation to specific embodiments, those skilled in the art will understand that numerous modifications are possible. Embodiments of this disclosure may be implemented using a variety of computer systems and communication technologies, including but not limited to the specific examples described herein. Embodiments of this disclosure may be implemented using any combination of dedicated components, and / or programmable processors, and / or other programmable devices. The various processes described herein may be performed on the same processor or on any combination of different processors. Components are configured to perform specific operations, and such configurations may be achieved, for example, by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or by any combination thereof. Furthermore, while the embodiments described above may refer to specific hardware and software components, those skilled in the art will understand that different combinations of hardware components or software components may also be used, and that specific operations described as being embodied in hardware may be embodied in software, or vice versa.

[0117] Computer programs incorporating various features of this disclosure may be coded and stored on various computer-readable storage media. Suitable media include optical storage media such as magnetic disks or tapes, compact discs (CDs) or digital versatile discs (DVDs), flash memory, and other non-temporary media. The computer-readable media on which the program code is coded may be packaged together with a compatible electronic device, or the program code may be provided separately from the electronic device (for example, by internet download or as separately packaged computer-readable storage media).

[0118] Therefore, although this disclosure has been described in relation to specific embodiments, it will be understood that this disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

1. A method for performing actions for events related to the use of an application, One or more processors identify free text related to an application, which should be evaluated for at least one of a plurality of events related to the use of the application. The one or more processors apply the free text to a machine learning (ML) architecture which is trained using a plurality of sample texts that represent at least one of the plurality of events related to the use of the application, The one or more processors determine a value indicating the likelihood of events related to the use of the application occurring, based on the application of the free text to the ML architecture. The one or more processors provide the generated ML model with model inputs based on the free text and the values ​​to obtain data for an electronic document that characterizes the events related to the use of the application, One or more processors perform an action using the data for the electronic document, Methods that include...

2. The method according to claim 1, wherein applying the free text to the ML architecture further comprises applying the free text to the ML architecture which includes a natural language processing (NLP) model configured to access a plurality of information resources related to the application.

3. The method of claim 2, wherein determining the value further includes identifying one of the plurality of information resources related to the application using at least a portion of the free text, based on the application of the free text to the ML architecture.

4. The method of claim 2, further comprising providing the model input to the generated ML model based on one information resource identified using at least a portion of the free text from the plurality of information resources related to the application, and obtaining the data.

5. The method according to claim 1, wherein the ML architecture further comprises a classifier model constructed using the plurality of sample texts, each of which is labeled with an indication of the presence or absence of the respective event related to the use of the application.

6. The method according to claim 1, wherein the plurality of events relating to the use of the application includes at least one of adverse events, serious adverse events, incidents, serious incidents, software bugs, user complaints, or convenience issues.

7. The method according to claim 1, further comprising determining the value by classifying the event as at least one of a harmful event, a serious harmful event, an incident, a serious incident, a software bug, a user complaint, or a convenience issue, based on the application of the free text to the ML architecture.

8. Performing the aforementioned action means Determining that the aforementioned event meets the reporting criteria for providing at least a portion of the electronic document to a remote device, In response to the determination that the aforementioned event meets the reporting criteria, transmit at least a portion of the electronic document to the remote device, The method according to claim 1, further comprising:

9. The method according to claim 1, wherein the generating ML model is trained using at least one corpus including sample inputs and sample outputs, the sample inputs identify at least one of (i) a sample information resource relating to at least one of the plurality of events, or (ii) a sample value indicating the possibility of the at least one event, and the sample outputs identify at least one of (i) a diagnostic result for the at least one event, (ii) a sample action for the at least one event, or (iii) an analysis result for the at least one event.

10. The method according to claim 1, wherein the execution of the action for the event further includes selecting the action from a plurality of actions according to the data, the plurality of actions including at least one of (i) ending the use of the application on a user device relating to the user, (ii) restricting the operation of the application relating to the event, (iii) sending a notification for presentation to the user to the user device, (iv) providing the electronic document to an administrator device, or (v) storing the electronic document.

11. The process further includes determining, by one or more processors, that the value indicating the probability of the event satisfies a threshold, Providing the model input further includes providing the model input to the generating ML model in response to a determination that the value satisfies the threshold. The method according to claim 1.

12. The one or more processors receive feedback via an interface that identifies updated values ​​indicating the possibility of updated events related to the application, The one or more processors update at least one of the multiple weights of the ML architecture based on the feedback, The method according to claim 1, further comprising:

13. The method according to claim 1, further comprising using one or more processors to generate the model input to include the free text and contextual information based on the values, according to a template.

14. The method according to claim 1, further comprising identifying the free text by obtaining the free text relating to the application from at least one of (i) email, (ii) text message, (iii) voice transcript, (iv) chatbot message, (v) e-post, or (vi) communication platform message.

15. Identifying the aforementioned free text means This involves constructing an event listener on the application to monitor the free text generated by the user of the application, The free text is obtained via the application programming interface (API) of the event listener of the aforementioned application, The method according to claim 1, further comprising:

16. The method according to claim 1, further comprising determining the value to generate a data element that identifies (i) an information resource related to the application, (ii) the value, and a timestamp related to a message including the free text.

17. The method according to claim 1, further comprising providing the model input to the generated ML model, and generating the electronic document containing one or more recommendations for the event.

18. The method according to claim 1, wherein the application includes a digital therapy application, and in parallel with the use of the digital therapy application, an effective amount of medicine to address the condition is administered to the user.

19. A system for events related to the use of an application, One or more processors coupled to memory, Free text related to an application, which should be evaluated for at least one of a plurality of events related to the use of the application, The free text is applied to a machine learning (ML) architecture, which is trained using a plurality of sample texts that represent at least one of the plurality of events related to the use of the application, Based on the application of the free text to the ML architecture, a value indicating the likelihood of occurrence of events related to the use of the application is determined. The generated ML model is provided with model inputs based on the free text and the values ​​to obtain data for an electronic document that characterizes the events related to the use of the application. Use the data for the electronic document to perform an action. One or more processors configured in this way A system equipped with these features.

20. The system according to claim 19, wherein one or more processors are configured to apply the free text to the ML architecture, which includes a natural language processing (NLP) model configured to access a plurality of information resources related to the application.

21. The system according to claim 20, wherein one or more processors are configured to identify one of the plurality of information resources related to the application using at least a portion of the free text, based on the application of the free text to the ML architecture.

22. The system according to claim 20, wherein one or more processors are configured to provide the model input to the generated ML model based on one information resource identified using at least a portion of the free text from the plurality of information resources related to the application, and to obtain the data.

23. The system according to claim 19, wherein the ML architecture further comprises a classifier model constructed using the plurality of sample texts, each of which is labeled with an indication of the presence or absence of the respective event related to the use of the application.

24. The system according to claim 19, wherein one or more processors are configured to classify the events as at least one of adverse events, serious adverse events, incidents, serious incidents, software bugs, user complaints, or convenience issues, based on the application of the free text to the ML architecture.

25. The one or more processors described above are: If it is determined that the aforementioned event meets the reporting criteria for providing at least a portion of the electronic document to a remote device, In response to the determination that the aforementioned event meets the reporting criteria, transmit at least a portion of the electronic document to the remote device. It is structured in such a way. The system according to claim 19.

26. The system according to claim 19, wherein the generating ML model is trained using at least one corpus including sample inputs and sample outputs, the sample inputs identify at least one of (i) a sample information resource relating to at least one of the plurality of events, or (ii) a sample value indicating the possibility of the at least one event, and the sample outputs identify at least one of (i) a diagnostic result for the at least one event, (ii) a sample action for the at least one event, or (iii) an analysis result for the at least one event.

27. The system according to claim 19, wherein one or more processors are configured to select an action from a plurality of actions according to the data, the plurality of actions include at least one of (i) ending the use of the application on a user device relating to a user, (ii) restricting the operation of the application relating to the event, (iii) sending a notification for presentation to the user to the user device, (iv) providing the electronic document to an administrator device, or (v) storing the electronic document.

28. The one or more processors are further configured to determine whether the value indicating the probability of the event satisfies a threshold, The one or more processors are configured to provide the model input to the generated ML model in response to a determination that the value satisfies the threshold. The system according to claim 19.

29. The system according to claim 19, wherein one or more processors are configured to obtain the free text related to the application from at least one of (i) email, (ii) text message, (iii) voice transcript, (iv) chatbot message, (v) e-post, or (vi) communication platform message.

30. The system according to claim 19, wherein the application includes a digital therapy application, and in parallel with the use of the digital therapy application, an effective amount of medicine to address the condition is administered to the user.

Images