Using machine learning and free text data to detect and report events related to use of a software application

Abstract

To provide a method and system for performing an action for an event related to the use of an application.SOLUTION: The computing system identifies free text to be evaluated for at least one of a plurality of events related to use of an application, applies the free text to a machine learning (ML) architecture, determines a value indicating a likelihood of occurrence of the event related to use of the application based on the application of the free text to the ML architecture, and provides, to a generative ML model, a model input based on the free text and the value to obtain data for an electronic document that characterizes the event related to use of the application.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Non-Provisional Patent Application No. 18 / 806,230, filed August 15, 2024, entitled "USING MACHINE LEARNING AND FREE TEXT DATA TO DETECT AND REPORT EVENTS ASSOCIATED WITH USE OF SOFTWARE APPLICATIONS," which is incorporated by reference in its entirety. [Background technology]

[0002] While using a software application, unexpected or unwanted events may occur. These events can result from a number of causes. For example, an individual who is not familiar with software technology may be unable to properly log into their account. These events negatively impact the overall performance of the application. These events also detract from the user's experience using the application, resulting in lower interaction rates. Consistent interference with the full and proper use of the application may frustrate users and cause them to abandon the application. In addition to events affecting application performance, there are events that affect the user's health when the software application is a digital therapeutic. If the causes of these events go unaddressed, individual users' symptoms or underlying health conditions may remain unaddressed or untreated.

[0003] In healthcare, there are monitoring and reporting requirements for some types of events described as adverse events. These requirements are imposed by various entities, such as regulatory authorities (e.g., the U.S. Food and Drug Administration (FDA)), medical institutions, and device manufacturers. These requirements obligate companies to continuously record and report certain types of events. Monitoring and reporting of adverse events encountered with digital therapeutics is unique and different from monitoring and reporting of adverse events encountered with traditional pharmaceuticals. Because digital therapeutics are used to address a user's disease, condition, or symptoms, an adverse event could include, for example, a patient using a migraine digital therapeutic and being unable to log in to their account and describe this experience as worsening their migraines, when the unpleasant symptoms are not a side effect of the treatment but rather the user's frustration due to confusion in operating the software technology.

[0004] Unlike pharmaceuticals, these types of events are difficult to monitor, let alone report. One reason is that users of digital therapeutics are not tied to a specific location (e.g., a hospital or medical facility) but can access the treatment outside of these environments, e.g., from home or as part of their daily routine. Being able to access digital therapeutics from outside of a specific location is a major advantage of digital therapeutics over traditional medicine. However, it is still more difficult to determine whether a user is experiencing an adverse event, making tracking and monitoring the adverse events a user is experiencing difficult. With pharmaceuticals, when a user is physically present with a healthcare professional, the healthcare professional can determine whether the person is experiencing an adverse event (e.g., a side effect or adverse effect).

[0005] One approach to detecting an event may rely on users submitting feedback (e.g., a call to a customer service agent, an email to a customer service address, a chat with a customer service chatbot, or a voicemail left on a customer service line outside of business hours) directly to an administrator of the application, who then carefully reviews the submission to determine whether the user truly experienced the event. However, the process of reviewing incoming submissions is manual, time-consuming, and potentially unresponsive, especially when there are a large number of users using the application and these submissions are generated and recorded by multiple different sources. Summary of the Invention

[0006] This disclosure describes systems and methods for using machine learning to detect and process events described in user feedback about their experience with an application, such as a digital therapeutic application. Utilizing the machine learning techniques detailed in this disclosure to detect events in messages about application usage has several advantages. Specifically, user messages are free text, recorded across a variety of channels, such as emails, text messages, call center transcripts, chatbot messages, after-hours voicemails, or in-app entries. The machine learning models described in this disclosure can rapidly analyze this large amount of free text, classify events by type and severity, and select actions to resolve the events.

[0007] Digital therapeutic applications use machine learning techniques to detect events so that therapeutic interventions can be delivered more efficiently to their users. The machine learning architecture described in this disclosure has the ability to detect events and address them in near real time, enabling immediate notification to administrators, faster resolution of the events, and more rapid delivery of treatment to users, reducing the likelihood that such events will disrupt the user receiving treatment. Furthermore, reducing these types of events improves the user experience, reducing frustration and increasing user satisfaction with digital therapeutic applications. This results in higher adherence, improved health outcomes, and better success with digital therapeutic interventions.

[0008] Additionally, machine learning techniques enable better and consistent compliance with monitoring and reporting requirements for events occurring during use of digital therapeutic applications. When an event is detected, information about the event, including initial free text, is fed into a generative model, which generates an analysis report for the event. The analysis report includes a severity classification for the event and a natural language description of possible remedies. The analysis results generated by the generative model and the free text from the user may be stored and maintained for recording and monitoring purposes. If the event is identified as being of a reportable type (e.g., a serious adverse event), the analysis report itself may be used as a submittal to relevant entities to fulfill reporting requirements.

[0009] As a result, the time between receiving a message about an application and detecting any events related to the use of the application is significantly reduced. Using the machine learning architecture detailed in this disclosure, a service can continuously monitor all incoming messages, including free text, and automatically generate event documentation in near real time using natural language. This is particularly beneficial when events occur outside of business hours when live customer service personnel are unavailable. The analysis report provides an overview of events related to the use of the application, along with recommended actions to address contributing causes. This eliminates the need for manual monitoring by application administrators, improves the efficiency and accuracy of identifying potential risks, and reduces response times when taking steps to address these events. In this way, any events detected from free text can be addressed by the service seamlessly, from initial identification of a potential problem to ultimate resolution of the root cause, with little or no human input.

[0010] Aspects of the present disclosure relate to a system, method, and computer-readable medium for performing an action for an event related to application use. One or more processors can identify free text related to the application, the free text to be evaluated for at least one of a plurality of events related to application use. The one or more processors can apply the free text to a machine learning (ML) architecture. The machine learning (ML) architecture can be trained using a plurality of sample texts that indicate at least one of a plurality of events related to application use. The one or more processors can determine a value that indicates a likelihood of occurrence of the event related to application use based on applying the free text to the ML architecture. The one or more processors can provide model input based on the free text and the value to a generative ML model to obtain data for an electronic document that characterizes the event related to application use. The one or more processors can perform the action using the data for the electronic document.

[0011] In some implementations, the one or more processors can apply the free text to an ML architecture including a natural language processing (NLP) model configured to access a plurality of information resources associated with the application. The one or more processors can identify an information resource of the plurality of information resources associated with the application using at least a portion of the free text based on applying the free text to the ML architecture. The one or more processors can provide model input to a generative ML model based on the one information resource identified using at least a portion of the free text from the plurality of information resources associated with the application to obtain data.

[0012] In some embodiments, the ML architecture may include a classifier model constructed using a plurality of sample texts. Each of the plurality of sample texts may be labeled with a respective indication of the presence or absence of a respective event associated with use of the application. The plurality of events associated with use of the application may include at least one of an adverse event, a serious adverse event, an incident, a serious incident, a software bug, a user complaint, or a usability issue. In some embodiments, the one or more processors may classify the event as at least one of an adverse event, a serious adverse event, an incident, a serious incident, a software bug, a user complaint, or a usability issue based on application of the free text to the ML architecture. In some embodiments, the one or more processors may determine that the event meets reporting criteria for providing at least a portion of the electronic document to a remote device. In response to determining that the event meets the reporting criteria, the one or more processors may transmit at least a portion of the electronic document to the remote device.

[0013] In some embodiments, the generative ML model can be trained using at least one corpus including sample inputs and sample outputs. The sample inputs can identify at least one of (i) a sample information resource associated with at least one of the plurality of events, or (ii) a sample value indicating a likelihood of the at least one event, and the sample outputs can identify at least one of (i) a diagnostic result for the at least one event, (ii) a sample action for the at least one event, or (iii) an analytical result for the at least one event. In some embodiments, the one or more processors can select an action from a plurality of actions according to the data. The plurality of actions can include at least one of (i) terminating use of an application on a user device associated with the user, (ii) restricting operation of the application related to the event, (iii) sending a notification to the user device for presentation to the user, (iv) providing an electronic document to an administrator device, or (v) storing the electronic document.

[0014] In some embodiments, the one or more processors may determine that a value indicating the likelihood of an event meets a threshold. The one or more processors may provide model inputs to the generative ML model in response to determining that the value meets the threshold. In some embodiments, the one or more processors may receive feedback via an interface identifying an updated value indicating an updated likelihood of the event associated with the application. The one or more processors may update at least one of a plurality of weights of the ML architecture based on the feedback. In some embodiments, the one or more processors may generate the model inputs to include free text and value-based context information according to a template.

[0015] In some embodiments, the one or more processors may obtain free text associated with the application from at least one of (i) email, (ii) text message, (iii) voice transcript, (iv) chatbot message, (v) electronic post, or (vi) communication platform message. In some embodiments, the one or more processors may build an event listener on the application to monitor free text generated by a user of the application. The one or more processors may obtain the free text via an application programming interface (API) of the event listener of the application.

[0016] In some embodiments, the one or more processors can generate data elements identifying (i) an information resource associated with the application, (ii) a value, and (iii) a timestamp associated with a message including free text. In some embodiments, the one or more processors can generate an electronic document including one or more recommendations for an event based on providing the model inputs to the generative ML model. In some embodiments, the application includes a digital therapeutic application for addressing a condition. The application may include a digital therapeutic application. Concurrent with use of the digital therapeutic application, an effective amount of medication for addressing the condition may be administered to the user. [Brief explanation of the drawings]

[0017] The above and other objects, aspects, features, and advantages of the present disclosure will become more apparent and be better understood from the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a block diagram of a system for performing actions for application usage related events detected from free text data, according to an exemplary implementation. [Figure 2]FIG. 1 is a block diagram of a machine learning (ML) architecture and a process for training a generative model in a system for performing actions for events, according to an illustrative embodiment. [Figure 3] FIG. 2 is a block diagram of a process for detecting events from free text in a system for performing actions for events, according to an exemplary embodiment. [Figure 4] FIG. 2 is a block diagram of a process for generating an electronic document for an event in a system for performing an action for an event, according to an exemplary embodiment. [Figure 5] FIG. 2 is a block diagram of a process for enforcing policies that perform actions in a system for performing actions for events, according to an illustrative embodiment. [Figure 6] 1 illustrates an exemplary user interface for presenting a user message and analysis result report for an event in a system for performing an action for an event, according to an exemplary embodiment. [Figure 7] FIG. 1 is a flow diagram of a method for performing actions for events related to application usage detected from free text data, according to an example implementation. [Figure 8] FIG. 1 is a block diagram of a server system and a client computer system according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] For purposes of reading the following description of the various embodiments, the following listed sections of the specification and their respective contents may be helpful:

[0019] Section A describes systems and methods for performing actions for application usage related events detected from free text data.

[0020] Section B describes network and computing environments that may be useful for implementing embodiments of the present disclosure. A. Systems and methods for performing actions for application usage related events detected from free text data

[0021] This disclosure presents a system and method for taking action for events related to application usage detected from free text data. The events may correspond to unexpected or undesirable occurrences occurring with application usage. The events may adversely affect the overall performance of the application, the computing device running the application, and even remote servers interfacing with the application. For example, a malfunction in a particular function of an application may result in excessive consumption of resources (e.g., processor and memory) on the computing device, causing the computing device to operate very slowly and experience long response times. These events also impair the user experience with the application. Detection of these events can be used to mitigate the adverse effects resulting from these events.

[0022] To address these and other technical challenges, an application monitoring service may obtain messages from multiple sources, and users may provide feedback about their application usage, and may use machine learning techniques to classify these messages and detect the occurrence of events. These messages may include free text aggregated from various channels, such as emails, messages, call center transcripts, chatbot messages, or in-app inputs. This may allow users to enter free text through more channels to provide feedback about the application. The machine learning techniques described in this disclosure can process the free text to detect events such as these associated with application usage.

[0023] By aggregating these messages, the application monitoring service can process the free text using machine learning architectures and generative models. The machine learning architecture may include natural language processing (NLP) models and classifier models. The NLP models may be used to access application-specific information resources (e.g., specific user interface elements), such as documentation about the event (e.g., medical conditions and computer performance) and standard operating procedures (SOPs) for the application. The NLP models may use the free text to incorporate retrieval-augmented generation (RAG) techniques and query information resources related to the content of the free text.

[0024] Additionally, the classifier model may be trained using sample text examples labeled with indications of an event. The examples may also be application-specific or may be previous user submissions for the same application regarding an event (e.g., a performance issue with the application or a worsening symptom of the user's medical condition). Both the NLP model and the classifier model may be used to process the free text and detect indications of any event in the application. The classifier model may be constantly updated iteratively through interactive learning using additional examples of sample text deemed to be indicative of an adverse event. The classifier model can use the collected free text to determine the likelihood of a given event occurring and classify which events occurred with use of the application.

[0025] The application monitoring service can use the output of the NLP model and the classifier model to generate data elements to be used as input prompts for the generative machine learning model. The data elements may include the information resources searched by the NLP model, the likelihood of the event occurring, and data about the message, such as portions of free text and timestamps of receipt. The generative model may be trained using a large number of sample corpora to produce analytical reports and documentation about the event. The application monitoring service can generate a report of the event by providing input prompts. The report may include a diagnosis or analysis of the event (e.g., a potential computing performance issue or a user's medical condition) and actions to address the event. In response to generating the report, the application monitoring service can take actions according to the contents of the report. Actions may include, for example, disabling certain features of the application causing the event, notifying an administrator of the application, or sending the report to a third party.

[0026] In this way, the application monitoring service can process the free text to detect events such as these associated with application use. The application monitoring service can also take action to mitigate or combat detected events that would otherwise go undetected. Detecting events and taking action can improve the performance of the application, the computing device on which the application is running, and the user experience. In the case of digital therapeutics, this can also improve adherence, improve health outcomes, and lead to better outcomes from digital therapeutic interventions.

[0027] Additionally, when an event is detected, information about the event, including the initial free text, may be provided by the application monitoring service to the generative model to generate documentation about the event. Using outputs from both the NLP model and the classifier model may reduce the likelihood of hallucination on the part of the generative model when generating output about the event. Hallucination refers to the generation of output by the generative model that is factually incorrect, logically inconsistent, or even outright fabricated. Such errors significantly reduce the reliability and usefulness of documentation from the generative model used to evaluate an event. A model architecture including an NLP model and a classifier model may therefore also improve the quality of documentation output from the generative model when evaluating free text messages and selecting actions to address any displayed events.

[0028] Referring now to FIG. 1 , a block diagram of a system 100 for performing actions for events related to application usage detected from free text data is illustrated. Generally, the system 100 may include at least one application monitoring service 105, at least one user device 110, at least one administrator device 115, at least one support device 120, at least one remote device 125, etc., coupled to each other via at least one network 130. The user device 110 may include at least one application 135. The application monitoring service 105 may include at least one model trainer 140, at least one message aggregator 145, at least one event evaluator 150, at least one report generator 155, at least one policy enforcer 160, at least one machine learning (ML) architecture 165, and at least one generative model 170. The ML architecture 165 may include at least one natural language processing (NLP) model 175, at least one classifier model 180, etc. The application monitoring service 105 may include or have access to at least one database 185. The functionality of the application 135 on the user device 110 may be partially performed on the application monitoring service 105, or vice versa.

[0029] More specifically, the application monitoring service 105 may be any computing device comprising one or more processors coupled to memory and software capable of executing the various processes and tasks described in this disclosure. The application monitoring service 105 may be associated with an entity implementing or managing instances of an application 135 running on one or more user devices 110. The application monitoring service 105 may communicate with user devices 110, administrator devices 115, support devices 120, remote devices 125, and the like. The application monitoring service 105 may be located, located, or associated with at least one computer system. The computer system may correspond to a data center, branch office, or site where one or more computers corresponding to the application monitoring service 105 are located.

[0030] The application monitoring service 105 may include one or more subsystems, modules, or components for performing the various processes and tasks described in this disclosure. The model trainer 140 may initialize, train, build, and update the ML architecture 165 and generative model 170 on the application monitoring service 105 using training data. The message aggregator 145 may search, identify, or retrieve messages containing free text related to the application 135 from various sources. The event evaluator 150 may process the free text using the ML architecture 165 to detect the occurrence of events related to the use of the application 135. The report generator 155 may create electronic documentation of the detected events using the generative model 170 that uses output from the ML architecture 165. The policy enforcer 160 may take action according to the data in the electronic documentation.

[0031] ML architecture 165 may include one or more machine learning (ML) models for processing input in the form of free text and generating various outputs. NLP model 175 may include or execute any number of NLP algorithms for processing free text. In some implementations, NLP model 175 may be used to implement or perform search expansion generation (RAG) for generative model 170. In general, NLP model 175 may be used to search one or more of a set of information resources using free text from database 185. NLP model 175 may have at least one input and at least one output. The input may include free text or data derived from the free text (e.g., tokens or embedded representations generated via tokenization). The output may include at least one information resource identified using the input free text. The information resource may include text determined to be related to the input free text. The NLP model 175 may use any number of algorithms to identify relevant information resources, such as term frequency-inverse document frequency (TF-IDF), vector space model (VSM), or latent semantic analysis (LSA), best matching (BM) ranking functions, etc.

[0032] The classifier model 180 may be used to process free text to determine a value indicating the likelihood of occurrence of at least one event associated with application usage and to classify the type of event. The architecture of the classifier model 180 may include a deep learning neural network (e.g., a convolutional neural network architecture, a residual network, or a transformer-based architecture), a regression model (e.g., a linear or logistic regression model), a random forest, a support vector machine (SVM), a clustering algorithm (e.g., k-nearest neighbors), or a naive Bayesian model. The classifier model 180 may be trained using supervised learning, unsupervised learning, or semi-supervised learning. In general, the classifier model 180 may have at least one input and at least one output. The input and output may be related via a set of weights. The input may include free text or data derived from the free text (e.g., tokens or embedded representations generated through tokenization). The output may include a value indicating the likelihood of an event associated with application usage or the type of event.

[0033] Generative model 170 may receive input and output content in one or more modalities (e.g., in the form of text strings, audio content, images, video, or multimedia content). Input may include outputs from ML architecture 165, such as event occurrence likelihood values ​​and identified information resources, and at least a portion of free text. Generative model 170 may be a machine learning model based on a Transformer model (e.g., a generative pre-trained model or a bidirectional encoder representation from a Transformer). Generative model 170 may be a large language model (LLM), a text-to-image model, a text-to-speech model, a text-to-video model, or the like. In some implementations, generative model 170 may be part of application monitoring service 105 (e.g., as shown). In some implementations, generative model 170 may be a separate server from application monitoring service 105 that communicates with application monitoring service 105 via network 130.

[0034] The generative model 170 may include a set of weights across a set of layers based on a Transformer architecture. In this architecture, the generative model 170 may include at least one tokenization layer (which may also be referred to as a tokenizer in this disclosure), at least one input embedding layer, at least one positional encoder, at least one encoder stack, at least one decoder stack, and at least one output layer, all interconnected (e.g., via forward, backward, or jump connections). In some implementations, the Transformer layer may lack an encoder stack (e.g., in the case of a decoder-only architecture) or a decoder stack (e.g., in the case of an encoder-only model architecture). The tokenization layer may convert raw input in the form of a set of strings into a corresponding set of word vectors (which may also be referred to as tokens, embeddings, or vectors in this disclosure) in an n-dimensional feature space. The input embedding layer may use the word vectors to generate a set of embeddings. Each embedding may be a lower-dimensional representation of the corresponding word vector and may capture semantic-syntactic information of the string associated with the word vector. The positional encoder may generate a positional encoding for each input embedded representation as a function of the position of the corresponding word vector or by expanding strings in the input set of strings.

[0035] Then, in the generative model 170, an encoder stack may include a set of encoders. Each encoder may include at least one attention layer and at least one feedforward layer, etc. The attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each input embedding to indicate the degree to which the embedding should pay attention and generate a weighted sum of the set of input embeddings. The feedforward layer may apply a linear transformation using a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the attention layer. The output may be fed to another encoder in the encoder stack in the transformer layer. If the encoder is the last encoder in the encoder stack, the output may be fed to the decoder stack.

[0036] The decoder stack may include at least one attention layer, at least one encoder-decoder attention layer, and at least one feedforward layer. In the decoder stack, the attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each output embedding representation (e.g., an embedded representation generated from a target or expected output). The encoder-decoder attention layer may combine inputs from an attention layer in the decoder stack with outputs from one of the encoders in the encoder stack and calculate an attention score from the combined inputs. The feedforward layer may apply a linear transformation using a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the encoder-decoder attention layer. The output of the decoder may be supplied to another decoder in the decoder stack. If the decoder is the last decoder in the decoder stack, the output may be supplied to an output layer.

[0037] The output layer of the generative model 170 may include at least one linear layer and at least one activation layer, etc. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to calculate token scores. The activation layer may apply an activation function (e.g., softmax, sigmoid, or rectified linear unit) to the output of the linear function to convert the token scores into probabilities (or distributions). The probabilities may represent the likelihood of an output token occurring given an input token. The output layer may use the probabilities to select an output token (e.g., at least a portion of output text, image, audio, video, or multimedia content with the highest probability). This may be repeated over a set of input tokens, and the resulting set of output tokens may be used to form the output of the generative model 170 as a whole. While this disclosure has primarily described Transformer models, the application monitoring service 105 may use other machine learning models to generate and output content.

[0038] The user device 110 (which may also be referred to in this disclosure as an end-user computing device) may be any computing device including one or more processors coupled to memory and software capable of executing the various processes and tasks described in this disclosure. The user device 110 may communicate with the application monitoring service 105, administrator device 115, support device 120, remote device 125, and database 185 via network 130. The user device 110 may be operated by or associated with an end user using an application 135 on the user device 110. The user device 110 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch or smart glasses), or laptop computer. The user device 110 may be used to access the application 135. In some implementations, the application 135 may be downloaded and installed on the user device 110 (e.g., via a digital distribution platform). In some implementations, the application 135 may be a web application with resources accessible via network 130.

[0039] The application 135 executing on the user device 110 may be a digital therapeutic application and may provide sessions (which may also be referred to in this disclosure as therapy sessions) to address one or more conditions (or symptoms) of the user. The user's conditions may include, for example, chronic pain (e.g., related to or including arthritis, migraines, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive strain injury, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin conditions (e.g., atopic dermatitis, psoriasis, excoriation disorder, and eczema), cognitive impairment (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, borderline personality disorder, and attention deficit hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, smoking disorder, or hallucinogen disorder), and other illnesses (e.g., narcolepsy and cancer).

[0040] The end user may be taking or be prescribed an effective amount of medication to address a condition at least partially in conjunction with use of the application 135 (e.g., over any number of sessions). For example, if the medication is for pain, the end user may be taking acetaminophen, a nonsteroidal anti-inflammatory composition, an antidepressant, an anticonvulsant, or other composition. For skin lesions, the end user may be taking a steroid, an antihistamine, or a topical antiseptic. For cognitive impairment, the end user may be taking a cholinesterase inhibitor or memantine. For psychiatric conditions, the end user may be taking an antidepressant, a mood stabilizer, an antipsychotic, a tranquilizer, or a stimulant. For substance use disorders, the end user may be taking naltrexone, disulfiram, acamprosate, or nicotine replacement therapy. The application 135 may increase the efficacy of the medication the user is taking to address the condition. Although this disclosure describes application 135 primarily as a digital therapeutic application, application 135 may be any type of application, such as a word processor, a spreadsheet editor, a web browser, a video game, a social media application, a multimedia player, a messaging application, or a mobile application.

[0041] The administrator device 115 (which may also be referred to in this disclosure as an administrator computing device) may be any computing device comprising one or more processors coupled to memory and software capable of executing the various processes and tasks described in this disclosure. The administrator device 115 may communicate with the application monitoring service 105, the user devices 110, the support devices 120, the remote devices 125, and the database 185 via the network 130. The administrator device 115 may be associated with an entity that monitors the operation, updates, or development of the application 135. The administrator device 115 may be a smartphone, other mobile phone, a tablet computer, a wearable computing device (e.g., a smartwatch or smart glasses), or a laptop computer. In some implementations, the administrator device 115 may be separate from the application monitoring service 105 (e.g., as shown). In some implementations, the administrator device 115 may be part of the application monitoring service 105.

[0042] Support device 120 (which may also be referred to in this disclosure as a call center computing device) may be any computing device comprising one or more processors coupled with memory and software capable of executing the various processes and tasks described in this disclosure. Support device 120 may communicate with application monitoring service 105, user devices 110, administrator devices 115, remote devices 125, and database 185 via network 130. Support device 120 may involve an entity responsible for responding to end-user inquiries regarding application 135 or prescribing treatment regimens for end-users. Support device 120 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch or smart glasses), or laptop computer.

[0043] The remote device 125 may be any computing device including one or more processors coupled with memory and software capable of executing the various processes and tasks described herein. The support device 120 may communicate with the application monitoring service 105, the user device 110, the administrator device 115, the remote device 125, and the database 185 via the network 130. The remote device 125 may also include entities to which events related to the use of the application 135 should be reported. The entities may include, for example, regulatory authorities (e.g., the U.S. Food and Drug Administration (FDA), the European Medicines Agency (EMA), the UK Medicines and Healthcare products Regulatory Agency (MHRA), the Japanese Pharmaceuticals and Medical Devices Agency (PMDA), or the China National Medical Products Administration (NMPA)), medical institutions (e.g., hospitals or private practitioners), device manufacturers, or pharmaceutical entities. The remote device 125 may be a smartphone, other mobile phone, a tablet computer, a wearable computing device (e.g., a smartwatch or smart glasses), or a laptop computer. In some implementations, remote device 125 may be located at, located in, or associated with at least one computer system, which may correspond to a data center, branch office, or site where one or more computers corresponding to remote device 125 are located.

[0044] The database 185 may store and maintain various resources and data related to the application monitoring service 105 and the applications 135. The database 185 may include a database management system (DBMS) for arranging and organizing the data maintained thereon. The database 185 may communicate with the application monitoring service 105, the user devices 110, the administrator devices 115, the support devices 120, and the remote devices 125 via the network 130. While performing various operations, the application monitoring service 105 and the applications 135 may access the database 185 to retrieve certain data therefrom. The application monitoring service 105 and the applications 135 may also write data to the database 185 in response to performing such operations.

[0045] FIG. 2 illustrates a block diagram of a process 200 for training a machine learning (ML) architecture and a generative model in system 100 for performing actions for an event. Process 200 may include or correspond to operations for initializing, training, and constructing ML architecture 165 and generative model 170 in system 100. In process 200, model trainer 140 on application monitoring service 105 may initialize, train, or construct ML architecture 165 (e.g., illustrated classifier model 180). ML architecture 165 may be constructed specifically to process text associated with a given application 135. To train classifier model 180, model trainer 140 may search, obtain, or identify sample datasets 205A-N (hereinafter collectively referred to as sample datasets 205) from database 185. The sample datasets 205 may be used to train classifier model 180.

[0046] Each sample dataset 205 may identify or include at least one sample text 210 labeled with at least one indication 215 or the like. The sample text 210 may include free text related to the use of the sample application. The sample application may correspond to application 135 or another application in a similar field (e.g., a digital therapeutics application from the same developer). The sample text 210 may be collected from previous submissions of user complaints or issues with the sample application. The sample text 210 may be, for example, an email, a text message (e.g., a short message service (SMS) or multimedia message service (MMS) message), an audio transcript (e.g., an automatic speech recognition (ARS) transcription of a call center call), a chatbot message (e.g., via an in-app help interface with a chatbot), an electronic post (e.g., a social media posting, app review post, or comment), or a communication platform message (e.g., a message via a messaging application).

[0047] The display 215 may identify the presence or absence of events associated with use of the application. Events may include, for example, adverse events (e.g., an unexpected or undesirable experience associated with use of the application that affects a user's medical condition), serious adverse events (e.g., a serious harm or life-threatening event), software bugs (e.g., an exception occurs, slow response time, heavy performance), user complaints (e.g., a user being unfamiliar with the technology, a user setting up an undesirable interface design), or usability issues (e.g., unresponsiveness of a feature, missing functionality, or suggested improvements). In some embodiments, an event may be defined relative to another environment or situation. For example, an event may include, for example, an adverse event (e.g., an unexpected or undesirable experience associated with use of the application that affects a user's medical condition during a clinical trial), a serious adverse event (e.g., a serious harm or life-threatening event during a clinical trial), an incident (e.g., an unexpected or undesirable experience associated with use of the application that affects a user's medical condition outside of a clinical trial), or a serious incident (e.g., a serious harm or life-threatening event outside of a clinical trial). The indication 215 may define, identify, or identify the type of event that is present (or not) in the associated sample text 210. For example, the indication 215 may identify that the sample text 210 indicates a software bug, but does not indicate an adverse event that affects the user's medical condition.

[0048] In response to the identification, the model trainer 140 may provide, feed, or apply the sample text 210 to the classifier model 180 of the ML architecture 165. In some implementations, the model trainer 140 may use the sample text 210 to generate input data to provide to the classifier model 180. The input data may include tokens (which may also be referred to as word vectors or embeddings) in an n-dimensional space for processing by the ML architecture 165. The model trainer 140 may process the sample text 210 (or derived input data) using a set of weights for the classifier model 180. In response to the processing, the model trainer 140 may generate at least one output value 220 for at least one corresponding event. The output value 220 may identify or indicate the likelihood of occurrence of the corresponding event associated with use of the sample application with the sample text 210. In some implementations, the model trainer 140 may generate a set of output values ​​220 for a corresponding set of event types. In some implementations, the model trainer 140 may determine the type of events in the sample text 210 .

[0049] The model trainer 140 may calculate, generate, or determine at least one loss metric based on a comparison of the output values ​​220 of the sample text 210 in the sample dataset 205 to the representation 215. In some implementations, the model trainer 140 may compare the type to the representation 215 to determine the loss metric. The loss metric may indicate the degree of deviation of the output values ​​220 from the representation 215. The loss metric may be generated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean absolute error (MAE), mean squared error (MSE), quadratic loss, cross-entropy error, and Huber loss. In general, the more inaccurate the output values ​​220 are relative to the representation 215, the higher the loss metric may be. Conversely, the more accurate the output values ​​220 are relative to the representation 215, the lower the loss metric may be.

[0050] The model trainer 140 may use the loss metric to modify, change, or update at least one parameter of the classifier model 180. The updating of the parameters of the classifier model 180 may be performed according to an objective function. The objective function may define one or more rates at which the parameters of the classifier model 180 should be updated. The objective function may follow a stochastic gradient descent method, including, for example, adaptive moment estimation (Adam), implicit gradient updating (ISGD), and adaptive gradient algorithms (AdaGrad). The model trainer 140 may iteratively process a set of sample datasets 205 to iteratively update the ML architecture 165. The updating of the parameters of the classifier model 180 may be repeated until convergence. In response to completing training, the model trainer 140 may store and maintain the set of parameters of the classifier model 180.

[0051] Additionally, model trainer 140 may initialize, train, or build generative models 170. In some implementations, model trainer 140 may fine-tune, modify, or update a previously trained generative model 170. Generative models 170 may be built specifically to process text associated with a given application 135. To train, model trainer 140 may search, obtain, or identify a set of corpora 225A-N (hereinafter collectively referred to as corpora 225) from database 185. In some implementations, at least one of corpora 225 may be a generalized dataset. For example, generalized text for corpus 225 may be obtained from a large set of unstructured text without focusing on a specific knowledge domain. In some implementations, at least one corpus 225 may be a knowledge domain-specific dataset. Each corpus 225 may identify or include at least one sample input 230 and at least one sample output 235, etc.

[0052] The sample input 230 may include or identify at least one sample data element to be used as part of the input to the generative model 170. The sample data element may include or identify at least one sample value and at least one sample information resource, etc. The sample value may indicate a degree of likelihood of occurrence of at least one corresponding event determined from the sample free text. The sample information resource may include an information resource identified as being related to the free text. The sample information resource may be, for example, a text sentence identifying a user interface element within a sample application (e.g., application 135), standard operating procedures (SOPs) for using the application, literature on a medical condition that the digital therapeutic is to address, etc. In some embodiments, the sample data elements of the sample input 230 may include at least one sample event type and sample message data. The sample event type may identify the type of event into which the sample free text is classified. The message data may include or identify at least a portion of the sample free text itself and various types of information about the sample free text. The information may include a timestamp of receipt, a user's location, a device type, a user's status, etc.

[0053] The sample output 235 may include or identify expected output data 240 from the generative model 170 when the associated sample input 230 is applied. The sample output 235 may include a natural language description characterizing the event depicted in the sample input 230 and recommended actions to address the event. The sample output 235 may include or identify, for example, a diagnostic result for at least one corresponding event associated with use of the sample application, an analysis result for the event, a sample action for addressing the event, and one or more recommendations for a user of the application. The diagnostic result may define or identify the occurrence, cause, and impact of the event on the sample application, the user, the user device, etc. The analysis result may identify or include a report documenting the event in detail, such as the nature and circumstances of the event, the classification type of the event, the source of free text associated with the event, the state of the sample application, the state of the user, the state of the user device, and statistical descriptors of the event.

[0054] In the sample output 235, the sample action may then identify or include a corrective action to address at least one event. The sample action may include, for example, terminating the user's use of the sample application on the user device, modifying or restricting an operation (e.g., a feature or function) within the application related to the event, sending a notification to the user device or a support device, sending a report to an administrator device, sending a notification to a remote device, or storing a report related to the event. The sample recommendation may identify or include steps the user should take to address the event. For example, the sample recommendation may include guidance on how to use a particular feature of the application (e.g., logging in or opening an icon), or how to complete a set of lessons for the user if the event is a usability issue with the feature or lesson.

[0055] In response to the identification, model trainer 140 may build or train generative model 170 using the set of corpora 225. In some implementations, model trainer 140 may initialize generative model 170. For example, model trainer 140 may create an instance of generative model 170 by applying random values ​​to weights in layers. In some implementations, model trainer 140 may fine-tune a pre-trained generative model 170 (e.g., a ChatGPT model, a LLAMA model, and a stable diffusion model) using the set of corpora 225. For training or fine-tuning, model trainer 140 may define, select, or identify at least a portion of each corpus 225 as a source set (e.g., at least a portion of sample inputs 230) and at least a portion of each corpus 225 as a destination set (e.g., at least a portion of sample outputs 235). In some implementations, model trainer 140 may select or identify source and destination sets using mappings in corpora 225. The source sets may be used as inputs to generative model 170 to generate outputs to be compared with the destination sets. Portions of each corpus 225 may at least partially overlap and may correspond to subsets of text strings across sample inputs 230 and sample outputs 235.

[0056] For each corpus 225, model trainer 140 may feed or apply the source set strings from corpus 225 to generative model 170. During application, model trainer 140 may process input strings according to a set of layers in generative model 170. As described above, generative model 170 may include a tokenization layer, an input embedding layer, a positional encoder, an encoder stack, a decoder stack, and an output layer, among others. Model trainer 140 may process the source set input strings (alphanumeric words or phrases) using the tokenization layer of generative model 170 to generate a set of word vectors for the input set. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a simple embedding table). Model trainer 140 may apply the set of word vectors to an input embedding layer to generate a corresponding set of embedding representations. Model trainer 140 may identify the position of each string within the set of strings in the source set. In response to the identification, model trainer 140 may apply a positional encoder to the position of each string to generate a positional encoding for each embedded representation corresponding to the string by expanding the embedded representation.

[0057] Model trainer 140 may apply the set of embedding representations to the encoder stack of generative model 170, along with a corresponding set of positional encodings generated from the input set of corpus 225. During application, model trainer 140 may process the set of embedding representations along with the corresponding set of positional encodings according to the layers (e.g., attention layers and feedforward layers) within each encoder in the encoder block. In response to the processing, model trainer 140 may generate another set of embedding representations to feed forward to the encoders in the encoder stack. Model trainer 140 may then provide the output of the encoder stack to the decoder stack.

[0058] Together, model trainer 140 may process the destination set using a separate tokenization layer of generative model 170 to generate a set of word vectors for the destination set. The destination set may be of the same modality as the source set of corpus 225 or may be of a different modality from the source set of corpus 225. Each word or code vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a simple embedding table). Model trainer 140 may apply the set of words or code vectors to an input embedding layer to generate a corresponding set of embedded representations. Model trainer 140 may identify the position of each string within the set of strings of the target set. In response to the identification, model trainer 140 may apply a positional encoder to the position of each string to generate a positional encoding for each embedded representation corresponding to the string by expanding the embedded representation.

[0059] Model trainer 140 may apply the set of embedding representations to a decoder stack of generative model 170 along with a corresponding set of positional encodings generated from a destination set of corpus 225. Model trainer 140 may also combine the outputs of the encoder stacks during processing through the decoder stack. During application, model trainer 140 may process the set of embedding representations along with the corresponding sets of positional encodings according to the layers (e.g., attention layer, encoder-decoder attention layer, feedforward layer) within each decoder in the decoder block. Model trainer 140 may combine the output from the encoder with the input of the encoder-decoder attention layer in the decoder block. In response to processing, model trainer 140 may generate an output set of embedding representations to be fed forward to the output layer.

[0060] The model trainer 140 may then feed the output from the decoder block to the output layer of the generative transformer layer. To do so, the model trainer 140 may process the embeddings from the decoder block according to the linear and activation layers of the output layer. In response to the processing, the model trainer 140 may calculate a probability for each embedding. The probability may represent the likelihood of an output occurring given an input token. Based on the probabilities, the model trainer 140 may select the most probable output tokens (e.g., at least a portion of the sample outputs 235) to form, create, or generate output data 240. The output data 240 may include sample diagnoses, actions, analysis results, or recommendations, or any combination thereof. The output data 240 may be of the same modality as the set of subjects in the corpus 225. While primarily described with respect to a transformer model architecture, other architectures may be used to output content for the generative model 170.

[0061] In response to the generation, the model trainer 140 may compare the output data 240 from the generative model 170 with the destination set of the corpus 225 used to generate the output data 240. The comparison may be made between the probabilities (or destinations) of various tokens of content from the output data 240 and the probabilities of tokens in the target set of the corpus 225. For example, the model trainer 140 may determine the discrepancy between the probability distribution of the output data 240 and the target set of the corpus 225 to which it is compared. The probability distribution may identify the probability of each candidate token in the output data 240 or token in the target set of the corpus 225. Based on the comparison, the model trainer 140 may calculate, determine, or generate a loss metric. The loss metric may indicate the degree to which the output data 240 deviates from the expected output defined by the target set of the corpus 225 used to generate the output data 240. The loss metric may be calculated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), quadratic loss, cross entropy error, and Huber loss.

[0062] In some implementations, model trainer 140 may determine a loss metric for output data 240 based on data retrieved from database 185. In determining, model trainer 140 may compare the content of output data 240 to a destination set (e.g., at least a portion of sample output 235) to calculate a similarity. The similarity may measure, correspond to, or indicate, for example, a level of code similarity (e.g., using a knowledge map in the case of a comparison between sample output 235 and output data 240). Generally, the higher the loss metric, the more likely the generated output will deviate from the expected output corresponding to the destination set derived from corpus 225. Conversely, the lower the loss metric, the less likely the generated output will deviate from the expected output derived from the destination. A loss metric may be calculated to train generative model 170 to generate output content with a higher probability of accurate generation of the output.

[0063] The model trainer 140 may use the loss metric to update one or more weights in the set of layers of the generative model 170. The weight updates may be performed according to backpropagation and an optimization function (which may also be referred to as an objective function in this disclosure) using one or more parameters (e.g., a learning rate, momentum, weight decay, and number of iterations). The optimization function may define one or more parameters for which the weights of the generative model 170 should be updated. The optimization function may follow a stochastic gradient descent method, and may include, for example, adaptive moment estimation (Adam), implicit update (ISGD), and adaptive gradient algorithm (AdaGrad). The model trainer 140 may iteratively train the generative model 170 until convergence. In response to convergence, the model trainer 140 may store and maintain a set of weights for the set of layers of the generative model 170 for use in the estimation stage.

[0064] 3 is a block diagram of a process 300 for detecting events from free text in the system 100 for performing actions for the events. The process 300 may include or correspond to operations in the system 100 for evaluating free text in messages from users associated with application usage. In the process 300, a message aggregator 145 running on the application monitoring service 105 may search for, receive, or identify at least one piece of free text 305 associated with an application 135. The free text 305 may be identified for evaluation of one or more events associated with the use of the application 135. The free text 305 may include or have a string of alphanumeric characters, independent of a predefined data structure.

[0065] Free text 305 may be obtained from a variety of sources, such as emails (e.g., received at a customer service desk), text messages (e.g., short message service (SMS) or multimedia message service (MMS) messages), voice transcripts (e.g., automatic speech recognition (ARS) transcriptions of call center calls), chatbot messages (e.g., via an in-app help interface with a chatbot), electronic posts (e.g., social media postings, app review posts, or comments), or communication platform messages (e.g., messages via a messaging application). Free text 305 may correspond to alphanumeric characters without a predefined data structure (e.g., field-value pairs). Free text 305 may include alphanumeric characters in an unstructured format. In some implementations, free text 305 may correspond to text from an element in a structured data set without a defined format. For example, free text 305 may correspond to a portion of text in an inline frame in a web page using Hypertext Markup Language (HTML).

[0066] In some implementations, the message aggregator 145 may identify or search for the free text 305 from defined data sources via the network 130. The data sources may include, for example, a website for mobile application reviews (e.g., with electronic posts created by users) or a distribution service for downloading applications (including the application 135). Each data source may be identified via an address (e.g., a uniform resource locator (URL)). The address may be provided by a system administrator of the application 135 (e.g., via the administrator device 115). The message aggregator 145 may use the address to access the corresponding data source and search for or identify the free text 305. For example, the message aggregator 145 may access a website or distribution service that hosts mobile application reviews and search for electronic posts containing reviews by users of the application 135.

[0067] In some implementations, the message aggregator 145 may identify free text 305 from an application 135 on the user device 110. The message aggregator 145 may establish at least one event listener on the application 135 to monitor the generation of free text 305 by a user of the application 135 on the user device 110. For example, the event listener may receive text entered by the user into a user interface for feedback about the application 135. In response to receipt, the event listener on the application 135 may provide the input as free text 305 to the message aggregator 145. The message aggregator 145 may search, fetch, or retrieve the free text 305 via an application programming interface (API) of the event listener of the application 135. The API may allow various functions of the application 135 to be invoked from outside the application 135. In some implementations, the message aggregator 145 may receive the free text 305 from the support device 120 (or other device). For example, the free text 305 may correspond to a call transcript generated using automatic speech recognition (ASR) during a telephone conversation between an employee using the support device 120 and a user of the application 135.

[0068] In response to the identification, the message aggregator 145 may determine whether the free text 305 is related to the use of the application 135. To make the determination, the message aggregator 145 may use a natural language processing (NLP) algorithm (e.g., keyword extraction, information extraction, or text mining) to extract or identify one or more keywords from the free text 305 related to the use of the application 135. For example, the message aggregator 145 may use an NLP algorithm to detect keywords that target a condition to be addressed by the application 135, symptoms related to the condition, or the like. If at least one keyword is detected, the message aggregator 145 may determine that the free text 305 is related to the use of the application 135. Additionally, the message aggregator 145 may forward the free text 305 to the event evaluator 150 for further processing. Alternatively, if no keywords are detected, the message aggregator 145 may exclude the free text 305 from further processing.

[0069] An event evaluator 150 running on the application monitoring service 105 may provide, feed, or apply free text 305 to an ML architecture 165, including an NLP model 175 and a classifier model 180. In some implementations, the event evaluator 150 may generate input data using the free text 305 for provision to the ML architecture 165. The input data may comprise tokens (which may also be referred to as word vectors or vectors) in an n-dimensional space for processing by the ML architecture 165. In applying the free text 305 to the NLP model 175, the event evaluator 150 may formulate, determine, or generate at least one query 310 to search a set of information resources 315A-N (hereinafter collectively referred to as information resources 315) on a database 185 (or another data source). The query 310 may be generated using at least a portion of the free text 305. In some implementations, the event evaluator 150 may use query expansion (QE) algorithms such as vector space expansion, latent semantic analysis, knowledge graphs, statistical expansion, or synonym expansion.

[0070] The event evaluator 150 may use the query 310 and the NLP model 175 to discover, select, or identify at least one information resource 315′ from a set of information resources 315. The set of information resources 315 may be associated with the application 135. Each information resource 315 may include content (e.g., in the form of text, tokens, vectors, or embedded representations) related to various aspects of using the application 135. For example, the information resources 315 may include individual user interface statements available through the application 135, standard operating procedures (SOPs) for operating the application 135 through various devices, conditions to be addressed through the application 135, or previous submissions by other users related to using the application 135. The NLP model 175 may select at least one information resource 315′ related to the query 310 using various functions, such as term frequency-inverse document frequency (TF-IDF), vector space model (VSM), or latent semantic analysis (LSA), or a best-match (BM) ranking function. The information resource 315 ′ may be used to perform search expansion generation (RAG) on the generative model 170 .

[0071] In conjunction with this, based on the application of the free text 305 to the classifier model 180 of the ML architecture 165, the event evaluator 150 may calculate, generate, or determine at least one value 320. During application, the event evaluator 150 may process the free text 305 according to the set of weights of the classifier model 180 to generate the value 320. The value 320 may identify, define, or indicate the likelihood of occurrence of at least one event associated with use of the application 135. In some implementations, the value 320 may identify, define, or indicate the risk of the event associated with use of the application 135. The value 320 may be a numeric value in any range, such as 0 to 1, 0 to 100, −1 to 1, or −100 to 100. Generally, the larger the value 320, the more likely the corresponding event has occurred while using the application 135. Conversely, the smaller the value 320, the less likely the corresponding event has occurred while using the application 135. In some implementations, the event evaluator 150 may determine a set of values ​​320 for a corresponding set of event types in response to applying the free text 305 to the classifier model 180 .

[0072] In some implementations, the event evaluator 150 may generate, determine, or classify an event type 325 of the event based on application of the free text 305 to the classifier model 180. The event type 325 may identify the event as at least one of an adverse event (e.g., an unexpected or unwanted experience related to use of the application that affects the user's medical condition), a serious adverse event (e.g., a serious harm or life-threatening event), a software bug (e.g., an exception occurs, slow response time, heavy performance), a user complaint (e.g., user unfamiliarity with the technology, user configuration with undesirable interface design), or a usability issue (e.g., unresponsiveness of a feature, missing functionality, or suggested improvement), etc. In some implementations, the event may be defined relative to another environment or situation. For example, an event may include, for example, an adverse event (e.g., an unexpected or unwanted experience associated with use of the application that affects a user's medical condition during a clinical trial), a serious adverse event (e.g., a serious harm or life-threatening event during a clinical trial), an incident (e.g., an unexpected or unwanted experience associated with use of the application that affects a user's medical condition outside of a clinical trial), or a serious incident (e.g., a serious harm or life-threatening event outside of a clinical trial). The event evaluator 150 may classify the event type 325 using the set of values ​​320 generated for the event type set. The event evaluator 150 may select or identify the event type 325 that corresponds to the highest value 320. For example, if the adverse event value 320 for a particular symptom is highest, the event evaluator 150 may classify the event type 325 as an adverse event having the defined symptom associated with use of the application 135.

[0073] The event evaluator 150 may use output from the ML architecture 165 to create, construct, or generate at least one data element 335. The data element 335 may identify or include one or more of the information resource 315′, the value 320, the event type 325, and the message data 330, etc. The message data 330 may include, for example, at least a portion of the free text 305 and metadata associated with the free text 305. The metadata may include, for example, a timestamp (e.g., a generation or receipt timestamp) associated with the message including the free text 305, a user identification, an application instance identification, a device identification, a device location, or a condition to be addressed by the application 135, etc. The data element 335 may be a data structure (e.g., a table, a matrix, a linked list, a tree, an array, or a class) for including one or more of the information resource 315′, the value 320, the event type 325, and the message data 330, etc. The data element 335 may be used as input to the generative model 170.

[0074] 4 is a block diagram of a process 400 for generating electronic documentation about an event in the system 100 for performing an action for the event. The process 400 may include or correspond to operations in the system 100 for evaluating free text in messages from users accompanying application usage. In the process 400, the report generator 155 running on the application monitoring service 105 may identify or determine whether the likelihood value 320 of the corresponding event meets a threshold. The threshold may define the value that the report generator 155 should provide as input to the generative model 170 to generate a report about the detected event. In some implementations, the report generator 155 may repeatedly compare the set of values ​​320 of the corresponding set of event types to the threshold.

[0075] If the value 320 does not meet the threshold (e.g., is less than the threshold), the report generator 155 may identify or determine that an event has not occurred. If none of the set of values ​​320 meets the threshold, the report generator 155 may determine that an event has not occurred. The report generator 155 may also refrain from further processing the free text 305 or the data elements 335. The report generator 155 may generate an indication that no events related to the use of the application 135 are present in the free text 305. The report generator 155 may transmit, send, or provide the indication along with the free text 305 to the administrator device 115. On the other hand, if the value 320 meets the threshold (e.g., is greater than or equal to the threshold), the report generator 155 may determine that a corresponding event related to the use of the application 135 has occurred. If at least one of the set of values ​​320 meets the threshold, the report generator 155 may determine that a corresponding event has occurred. The report generator 155 may also identify the event types 325 that correspond to the values ​​320 that meet the threshold. The report generator 155 may continue to process the free text 305 and data elements 335.

[0076] The report generator 155 may create, generate, or generate at least one model input 405 (which may also be referred to in this disclosure as a prompt) to be used as input to the generative model 170 based on at least a portion of the data elements 335. The generation of the model input 405 may occur in response to determining that the value 320 meets a threshold. The model input 405 may be based on one or more of the free text 305, the information resource 315′, the value 320, the event type 325, and the message data 330, etc. In some implementations, the model input 405 may be based on at least a portion of the free text 305 and the value 320. In some implementations, the report generator 155 may create the model input 405 to include contextual information according to a template. The template may include a predefined set of strings and a set of placeholders. The predefined set of strings may include, for example, commands or instructions that call for the generative model 170 to create a particular type of output, such as the text string “Create a report for this [event type] using the information….” The set of placeholders may be for including information from data elements 335. The context information may be derived from one or more of free text 305, information resource 315', value 320, event type 325, and message data 330. The context information may include, for example, further information about the user, application 135, and the condition to be addressed.

[0077] The report generator 155 may feed, apply, or provide the model inputs 405 to the generative model 170. During application, the report generator 155 may process the model inputs 405 using a set of layers in the generative model 170. As described above, the generative model 170 may include a tokenization layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer, etc. The report generator 155 may process the input strings (alphanumeric codes) of the model inputs 405 using the tokenization layer of the generative model 170 to generate a set of word vectors (which may also be referred to as word tokens or tokens in this disclosure) for the input set. Each word vector may be a vector representation of at least one corresponding input (e.g., a portion of data elements 335) in an n-dimensional feature space (using a word embedding table).

[0078] The report generator 155 may apply the set of word vectors to an input embedding layer to generate a corresponding set of embedded representations. The report generator 155 may identify the position of each string in the set of strings in the model input 405. In response to the identification, the report generator 155 may apply a positional encoder to the position of each string to generate a positional encoding for each embedded representation corresponding to the string by expanding the embedded representation. The report generator 155 may apply the set of embedded representations, along with the corresponding set of positional encodings generated from the model input 405, to an encoder stack of the generative model 170. Upon application, the report generator 155 may process the set of embedded representations, along with the corresponding set of positional encodings, through layers (e.g., attention layers and feedforward layers) in each encoder in the encoder stack. In response to the processing, the report generator 155 may generate another set of embedded representations to feed forward to the encoders in the encoder stack. The report generator 155 may then provide the output of the encoder stack to a decoder stack.

[0079] In conjunction, the report generator 155 may input a starting input (which may be referred to as a starting token in this disclosure) using another tokenization layer of the generative model 170 and generate one corresponding word vector. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a word embedding table). The report generator 155 may apply the set of word vectors to an input embedding layer to generate a corresponding set of embedded representations. The report generator 155 may identify the position of each string in the set of strings of the target set. In response to the identification, the report generator 155 may apply a positional encoder to the position of each string to generate a positional encoding for each embedded representation corresponding to the string by expanding the embedded representation.

[0080] The report generator 155 may apply the set of embedded representations with a corresponding set of positional encodings generated from the decoder stack of the generative model 170. The report generator 155 may also combine the output of the encoder stack during processing through the decoder stack. During application, the report generator 155 may process the set of embedded representations with the corresponding set of positional encodings according to the layers in each decoder in the decoder block (e.g., attention layer, encoder-decoder attention layer, feedforward layer). The report generator 155 may combine the output from the encoder with the input of the encoder-decoder attention layer in the decoder block. In response to processing, the report generator 155 may generate an output set of embedded representations to be fed forward to the output layer.

[0081] The report generator 155 may then feed the output from the decoder block to the output layer of the generative transformer layer. In doing so, the report generator 155 may process the embedded representations from the decoder block according to the linear and activation layers of the output layer. In response to the processing, the report generator 155 may calculate a probability for each embedded representation. The probability may represent the likelihood of an output occurring given an input token. Based on the probability, the report generator 155 may select an output token (e.g., the most probable diagnosis, analysis result, action, or recommendation) to form, create, or generate a portion of the output. The report generator 155 may repeat the above process using the layers of the generative model 170 to form the entire output.

[0082] In response to providing model inputs 405 to generative model 170, report generator 155 may create, generate, or obtain data for at least one electronic document 410 (which may also be referred to as a report or documentation in this disclosure). Electronic document 410 may characterize events associated with use of application 135. Electronic document 410 may include or identify one or more of at least one action 415 and event information 420, etc. Electronic document 410 may include natural language text descriptions of action 415 and event information 420. The action 415 may be selected from one or more of, for example, terminating use of the application 135 on the user device 110, modifying or restricting an operation (e.g., a feature or function) within the application 135 related to the event, sending a notification to the user device 110 or the support device 120, sending the electronic document 410 to the administrator device 115, sending the electronic document 410 to a remote device 125, or storing the electronic document 410 related to the event on the database 185. In some implementations, the electronic document 410 may include or identify one or more recommendations for the event. The recommendations may include steps that a user of the application 135 should take to address the event. For example, the recommendations in the electronic document 410 may include a set of steps for the user to locate a particular user interface element (e.g., a button) within the graphical user interface of the application 135.

[0083] Event information 420 may include information derived from portions of data element 335, such as value 320, event type 325, and message data 330. Event information 420 may identify or include diagnostic results of an event related to use of application 135. The diagnostic results may define or identify the occurrence, cause, or impact of the event on application 135, the user, the user device 110, etc. For example, a diagnostic result of event information 420 in electronic document 410 may include text beginning with, "The cause of the application slowdown is estimated to be due to the type of smartphone being used to play the auditory stimuli (probability 0.8)." Event information 420 may identify or include analytical results of an event related to use of application 135. The analytical results may identify or include a report documenting the event in detail, such as the nature and circumstances of the event, the event type, the source of any free text related to the event, the state of the application, the state of the user, the state of the user device 110, and statistical descriptors of the event. For example, the analysis result of the event information 420 of the electronic document 410 may include text beginning with, "The event occurred on March 15, 2025 at approximately 3:30:45 PM EST." At this point, the user experiences nausea while using the application.

[0084] 5 is a block diagram of a process 500 for enforcing policies that perform actions in system 100 for performing actions for events. Process 500 may include or correspond to operations in system 100 for performing actions using data from reports. In process 500, policy enforcer 160 running on application monitoring service 105 may take, perform, or implement at least one action using data for electronic document 410. Policy enforcer 160 may perform actions 415 as specified in electronic document 410. In some embodiments, policy enforcer 160 may identify or select action 415 to perform from a set of actions according to the data in electronic document 410.

[0085] To make the selection, policy enforcer 160 may parse electronic document 410 to extract or identify actions 415 from the content of electronic document 410. Policy enforcer 160 may use natural language processing (NLP) algorithms, such as parsing regular expression templates, named entities, a lexical analyzer, or pattern matching. For example, policy enforcer 160 may use a regular expression template to identify a fixed string "recommended action" and then identify a specific action 415, such as "reduce application brightness." In response to the selection, policy enforcer 160 may execute action 415. Execution of action 415 may be automatic (e.g., without approval) or in response to approval (e.g., approval by a system administrator of application 135).

[0086] If the action 415 is to terminate use of the application 135, the policy enforcer 160 may generate at least one instruction 505 commanding the user device 110 to terminate use of the application 135. The instruction 505 may specify that the user device 110 restrict or disable execution of the application 135 on the user device 110, or uninstall the application 135 from the user device 110. The instruction 505 may identify the instance of the application 135, the user device 110, or the user of the user device 110, etc. In response to generation, the policy enforcer 160 may send, transmit, or provide the instruction 505 to the user device 110. In response to receipt, the user device 110 may stop, abort, or terminate use of the application 135 on the user device 110.

[0087] If the action 415 is to modify a particular function of the application 135, the policy enforcer 160 may generate instructions 505 that instruct the application 135 to change, vary, or modify the function as specified by the action 415. The instructions 505 may specify that the application 135 is to change the behavior of the particular function of the application 135. The modifications may include, for example, changing the order of presentation of a user interface (or screen), moving the position of a user interface element within the user interface, changing the point at which a particular function is performed by the application 135, etc. In response to generating, the policy enforcer 160 may send, transmit, or provide the instructions 505 to the user device 110. In response to receiving, the user device 110 may modify the function of the application 135 in accordance with the instructions 505.

[0088] If the action 415 is to restrict a particular function of the application 135, the policy enforcer 160 may generate instructions 505 that command the application 135 to disable, deactivate, or restrict the function as specified by the action 415. The instructions 505 may specify that the application 135 should disable, prevent, or restrict the execution of a particular function. For example, the instructions 505 may disable the presentation of a particular lesson via the application 135, deactivate the ability to search for particular information, or restrict access to a function during a given time period (e.g., afternoon), etc. In response to generation, the policy enforcer 160 may send, transmit, or provide the instructions 505 to the user device 110. In response to receipt, the user device 110 may restrict the function of the application 135 in accordance with the instructions 505.

[0089] If the action 415 is to send a notification to be presented to a user, the policy enforcer 160 may generate instructions 505 instructing the presentation of at least a portion of the electronic document 410. The instructions 505 may include one or more recommendations, diagnostic information, or analysis information of the electronic document 410, etc. The instructions 505 may specify whether the notification should be received and presented to a user of the application 135, a user of the user device 110, or a user of the support device 120 of the application 135. In response to generating the instructions 505, the policy enforcer 160 may send, transmit, or provide the instructions 505 to the user device 110 (or support device 120) in accordance with the instructions 505. In response to receiving the instructions 505, the user device 110 (or support device 120) may display, render, or present the notification. For example, the user device 110 may present a message box within the user interface of the application 135 to provide a step-by-step guide through a lesson presented via the application 135.

[0090] If the action 415 is to provide a notification 510 to the administrator device 115, the policy enforcer 160 may generate at least one notification. The notification 510 may include at least a portion of the electronic document 410, such as the action 415, the event information 420 (e.g., diagnostic or analytical results), or a recommendation. The notification 510 may also include the original free text 305 and at least a portion of the data elements 335, such as the value 320, the event type 325, and the message data 330. In response to generation, the policy enforcer 160 may send, transmit, or provide the notification 510 to the administrator device 115. In response to receiving, the administrator device 115 may render, display, or present the notification 510. Using the information in the notification 510, the system administrator may determine what corrective action to take to address the occurrence of the event across one or more instances of the application 135. An example of the display of the notification 510 via a user interface on the administrator device 115 is shown in FIG. 6.

[0091] If the action 415 is to provide a report to the remote device 125, the policy enforcer 160 may generate at least one report 515. The report 515 may be communicated or made known to another entity other than the entity performing or managing the operation of the application 135. The report 515 may include at least a portion of the electronic document 410, such as the event information 420 (e.g., a diagnostic or analytical result). In some implementations, the policy enforcer 160 may determine whether to send the report 515 to the remote device 125 based on reporting criteria. The reporting criteria may identify or define conditions under which the report 515 should be provided to the remote device 125 (using at least a portion of the electronic document 410). For example, the reporting criteria may specify that if the event type 325 of the event is a serious adverse event, then the corresponding report 515 is sent to the remote device 125.

[0092] In making the determination, the policy enforcer 160 may identify or determine whether the event (or event type 325 or other data) meets the reporting criteria. If the event meets the reporting criteria, the policy enforcer 160 may transmit, provide, or send the report 515 to the remote device 125. Alternatively, if the event does not meet the reporting criteria, the policy enforcer 160 may refrain from sending the report 515 to the remote device 125. In some embodiments, the policy enforcer 160 may send the report 515 to the remote device 125 independent of the reporting criteria. For example, if the specified action 415 is to send a report to the remote device 125, the policy enforcer 160 may generate and send the report 515 to the remote device 125. In response to receiving the report 515, the remote device 125 may store and maintain the report 515 on a data repository. The remote device 125 may also display, render, or present the report 515 via a user interface.

[0093] If the action 415 is to store a report, the policy enforcer 160 may store and maintain the electronic document 410 in the database 185. The electronic document 410 may be stored in association with a user, a user device 110, an application 135, or free text 305, for example. The electronic document 410 may be maintained in the database 185 using any type of data structure, such as a table, a matrix, an array, a linked list, a tree, or a heap. The policy enforcer 160 may store the electronic document 410 in the database 185 for archival purposes. The electronic document 410 may be later retrieved (e.g., by a system administrator) for diagnostics or analysis of the application 135. In some implementations, the policy enforcer 160 may store and maintain the electronic document 410 for later transmission to the remote device 125. For example, the policy enforcer 160 may store electronic documents 410 for various events and, upon request, transmit the electronic documents 410 in batches to the remote device 125 as part of a reporting procedure.

[0094] In some embodiments, policy enforcer 160 may search for, identify, or receive feedback 520 related to electronic document 410. Feedback 520 may be received via a user interface. For example, a user interface may be used to present notification 510 (including portions of electronic document 410) and receive input for feedback 520 on administrator device 115. In some embodiments, feedback 520 may define or identify an updated value indicating the likelihood of an occurrence of an event related to use of application 135. In some embodiments, feedback 520 may identify a corrected event type for an event related to use of application 135.

[0095] Based on the feedback 520, the policy enforcer 160 may modify, change, or update one or more of the set of weights of the ML architecture 165. The weight updates may be similar to the weight updates as part of training in the process 200 detailed in this disclosure. For example, the policy enforcer 160 (or the model trainer 140) may calculate, generate, or determine at least one loss metric based on a comparison of the output values ​​320 and the feedback 520. The loss metric may indicate the degree of deviation of the values ​​320 from the feedback 520. The loss metric may be generated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), mean absolute error (MAE), quadratic loss, cross-entropy error, and Huber loss. Using the loss metric, the model trainer 140 may modify, change, or update at least one parameter of the classifier model 180.

[0096] In this manner, the application monitoring service 105 can process the free text 305 to detect events occurring during application use. The application monitoring service 105 can indirectly detect additional events affecting application performance and user experience through the free text 305. The application monitoring service 105 can also take action 415 to counter or address detected events that would otherwise go undetected. Using the NLP model 175 and classifier model 180 of the ML architecture 165 in combination with the generative model 170 can reduce the likelihood of hallucination in the electronic document 410 characterizing the events, improving the quality and reliability of the documentation of these detected events. The detection of such events and the implementation of action 415 can significantly improve the performance of the application 135, the user device 110 running the application 135, and the user experience. In the case of digital therapeutics, both of these can also increase adherence, improve health outcomes, and enhance the success of digital therapeutic interventions.

[0097] FIG. 6 illustrates an exemplary user interface 600 for presenting a user message and analysis report about an event in a system for performing actions for an event. The user interface 600 may correspond to a graphical user interface presented via the administrator device 115. The user interface 600 may include any number of user interface elements, such as a message element 605, a report element 610, a rejection element 615, and an approval element 620. The message element 605 may include free text entered by user “XYZ.” The free text may have been received as part of an email to a support representative or may indicate the user’s experience (e.g., “nauseated”) with a particular interface element (e.g., “animated bird icon”) within the application 135.

[0098] The report element 610 may display a portion of the electronic document 410 generated by the generative model 170. The report element 610 may include a list of candidate event type classifications and their respective probabilities of occurrence. Each candidate event type 325 (e.g., a user interface problem, a network connection problem, or a worsening medical condition) may correspond to one of the event type classifications based on the output of the ML architecture 165. Each probability may correspond to a value indicating a likelihood based on the output of the ML architecture 165. The report element 610 may also include one or more actions to be performed to address or mitigate an event associated with the use of the application 135. Additionally, on the user interface 600, a reject element 615 may be used by a system administrator to not perform an action on a detected event. In response to interacting with the reject element 615, the policy enforcer 160 may refrain from performing the action. The approval element 620 may be used by a system administrator to perform a recommended action. In response to interacting with the approval element 620, the policy enforcer 160 may perform the action.

[0099] FIG. 7 is a flow diagram of a method 700 for performing an action for an event related to application usage detected from free text data. Method 700 may be implemented or performed using any of the components detailed in this disclosure, such as application monitoring service 105 or system 800. In method 700, a computing system may receive free text related to a user of an application (705). The computing system may apply a machine learning (ML) architecture to the free text (710). The computing system may determine a likelihood value of the event based on the application of the ML architecture to the free text (715). The computing system may identify an information resource using at least a portion of the free text (720). The computing system may determine whether the value meets a threshold (725). If the value does not meet the threshold (e.g., is less than the threshold), the computing system may determine an absence of the event (730). On the other hand, if the value meets the threshold (e.g., is greater than or equal to the threshold), the computing system may provide input to a generative model based on the value and the information resource (735). The computing system may generate a report in response to applying the inputs to the generative model (740). The computing system may perform an action using data from the report (745). B. Network and Computing Environment

[0100] Various operations described herein may be implemented on a computer system. FIG. 8 illustrates a simplified block diagram of an exemplary server system 800, client computing system 814, and network 826 that can be used to implement some embodiments of the present disclosure. In various embodiments, the server system 800 or a similar system may implement a service or server, or portions thereof, described herein. The client computing system 814 or a similar system may implement a client described herein. The system 100 described herein may be similar to the server system 800. The server system 800 may have a modular design incorporating multiple modules 802 (e.g., blades in a blade server implementation). While two modules 802 are illustrated, any number may be provided. Each module 802 may include a processing unit 804 and local storage 806.

[0101] Processing unit 804 may include a single processor, which may have one or more cores, or multiple processors. In some implementations, processing unit 804 may include a general-purpose primary processor and one or more special-purpose coprocessors, such as a graphics processor, digital signal processor, etc. In some implementations, some or all of processing unit 804 may be implemented using customized circuitry, such as an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA). In some implementations, such integrated circuits execute instructions stored on the circuitry itself. In other implementations, processing unit 804 may execute instructions stored in local storage 806. Any combination of any type of processor may be included in processing unit 804.

[0102] The local storage 806 may include volatile storage media (e.g., DRAM, SRAM, SDRAM, etc.) and / or non-volatile media (e.g., magnetic or optical disks, flash memory, etc.). The storage media incorporated in the local storage 806 may be fixed, removable, or updatable, as appropriate. The local storage 806 may be physically or logically divided into various subunits, such as system memory, read-only memory (ROM), and permanent storage devices. The system memory may be a read-write memory device or a volatile read-write memory, such as dynamic random access memory. The system memory may store some or all of the instructions and data needed by the processing unit 804 at runtime. The ROM may store static data and instructions needed by the processing unit 804. The permanent storage device may be a non-volatile read-write memory device that can store instructions and data even when the module 802 is not powered on. The term "storage medium" as used in this disclosure includes any medium capable of storing data therein indefinitely (even in the face of overwriting, electrical disturbances, power outages, etc.), and does not include carrier waves and transitory electronic signals propagated wirelessly or via wired communications.

[0103] In some embodiments, local storage 806 may store one or more software programs to be executed by processing unit 804, such as an operating system and / or programs that embody various server functions, such as functions of system 100 or any other system described in this disclosure, or any other server functions associated with system 100 or any other system described in this disclosure.

[0104] "Software" generally refers to sequences of instructions that, when executed by processing unit 804, cause server system 800 (or portions thereof) to perform various operations and thus define one or more specific machine implementations that implement and execute the operations of the software program. The instructions may be stored as firmware resident in read-only memory and / or as program code stored on a non-volatile storage medium that can be loaded into a volatile working memory for execution by processing unit 804. The software may be embodied as a single program or as a collection of separate programs or program modules that interact as necessary. Processing unit 804 may retrieve program instructions to execute and data to process to perform the various operations described above from local storage 806 (or non-local storage, as described below).

[0105] In some server systems 800, multiple modules 802 may be interconnected via a bus or other interconnect 808 to form a local area network that facilitates communication between the modules 802 and other components of the server system 800. The interconnect 808 may be embodied using a variety of technologies, including server racks, hubs, routers, etc.

[0106] A wide area network (WAN) interface 810 may provide data communication capability between a local area network (e.g., via interconnect 808) and a network 826, such as the Internet. Other technologies may be used to communicatively couple server system 800 to network 826, including wired technologies (e.g., Ethernet, IEEE 802.3 standard) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standard).

[0107] In some implementations, local storage 806 is intended to provide working memory for processing unit 804, providing rapid access to programs and / or data to be processed while reducing traffic on interconnect 808. One or more mass storage subsystems 812 connectable to interconnect 808 may provide storage for larger amounts of data on the local area network. Mass storage subsystem 812 may be based on magnetic, optical, semiconductor, or other data storage media. Direct-attached storage, storage area networks, network-attached storage, etc. may be used. Any data store or other collection of data described in this disclosure as created, used, or maintained by a service or server may be stored in mass storage subsystem 812. In some implementations, additional data storage resources may be accessible (potentially with higher latency) via WAN interface 810.

[0108] Server system 800 may operate in response to requests received via WAN interface 810. For example, one of modules 802 may perform monitoring functions in response to received requests and assign individual tasks to other modules 802. Work allocation techniques may be used. As requests are processed, results may be returned to the requester via WAN interface 810. Such operations may be largely automated. Furthermore, in some embodiments, WAN interface 810 may interconnect multiple server systems 800 to provide a scalable system capable of managing large volumes of activity. Other techniques for managing server systems and server farms (collections of cooperating server systems) may be used, including dynamic resource allocation and reallocation.

[0109] Server system 800 may interact with various user-owned or user-operated devices over a wide area network, such as the Internet. An example of a user-operated device is illustrated in FIG. 8 as client computing system 814. Client computing system 814 may be embodied as a consumer device, such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch, eyeglasses), desktop computer, or laptop computer. For example, client computing system 814 may communicate over WAN interface 810. Client computing system 814 may include computer components such as a processing unit 816, a storage device 818, a network interface 820, a user input device 822, and a user output device 824. Client computing system 814 may be a computing device embodied in various form factors, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, other mobile computing device, or a wearable computing device.

[0110] The processing unit 816 and storage device 818 may be similar to the processing unit 804 and local storage 806 described above. Suitable devices may be selected based on the demands placed on the client computing system 814, for example, the client computing system 814 may be embodied as a "thin" client with limited processing power or as a high-performance computing device. The client computing system 814 may be provided with program code executable by the processing unit 816 to enable various interactions with the server system 800.

[0111] Network interface 820 may provide a connection to a network 826, such as a wide area network (e.g., the Internet), to which WAN interface 810 of server system 800 is also connected. In various embodiments, network interface 820 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0112] User input device 822 may include any device (or devices) through which a user can provide signals to client computing system 814. Client computing system 814 may interpret these signals as representing specific user requests or information. In various embodiments, user input device 822 may include at least one of a keyboard, a touchpad, a touchscreen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, etc.

[0113] The user output device 824 may include any device through which the client computing system 814 can provide information to a user. For example, the user output device 824 may include a display-to-display image generated by or delivered to the client computing system 814. The display may include various image generation technologies, such as, for example, a liquid crystal display (LCD), a light emitting diode (LED) including an organic light emitting diode (OLED), a projection system, a cathode ray tube (CRT), etc., along with supporting electronics (e.g., digital-to-analog converters, analog-to-digital converters, signal processors, etc.). Some embodiments may include devices such as a touchscreen that function as both an input and output device. In some embodiments, other user output devices 824 may be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, etc.

[0114] Some embodiments include electronic components, such as a microprocessor, storage, and memory, that store computer program instructions on a computer-readable storage medium. Many of the features described herein may be implemented as processes specified as a set of program instructions encoded on a computer-readable storage medium. When executed by one or more processing units, these program instructions cause the processing units to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as that produced by a compiler, and files containing higher-level code executed by a computer, electronic component, or microprocessor using an interpreter. Through suitable programming, processing units 804 and 816 may provide functionality for server system 800 and client computing system 814, including any or other functions described herein as being performed by a server or user.

[0115] It will be understood that the server system 800 and the client computing system 814 are illustrative and subject to variations and modifications. Computer systems used in conjunction with embodiments of the present disclosure may have other capabilities not specifically described herein. Furthermore, while the server system 800 and the client computing system 814 are described with reference to certain blocks, it should be understood that these blocks are defined for convenience of explanation and are not intended to imply a particular physical arrangement of components. For example, different blocks may, but need not, exist within the same facility, in the same server rack, or on the same motherboard. Furthermore, the blocks need not correspond to physically discrete components. Blocks may be configured to perform various operations, for example, by programming a processor or providing appropriate control circuitry, and various blocks may be reconfigurable or non-reconfigurable depending on how the initial configuration was obtained. Embodiments of the present disclosure may be realized in a variety of apparatuses, including electronic devices embodied using various combinations of circuitry and software.

[0116] While the present disclosure has been described with reference to specific embodiments, those skilled in the art will appreciate that numerous modifications are possible. Embodiments of the present disclosure may be implemented using a variety of computer systems and communication technologies, including, but not limited to, the specific examples described herein. Embodiments of the present disclosure may be implemented using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein may be performed on the same processor or on any combination of different processors. While components are configured to perform certain operations, such configuration may be achieved, for example, by designing electronic circuitry to perform the operations, by programming programmable electronic circuitry (such as a microprocessor) to perform the operations, or any combination thereof. Furthermore, while the above-described embodiments may refer to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware or software components may also be used, and that certain operations described as being implemented in hardware may also be implemented in software, or vice versa.

[0117] A computer program incorporating various features of the present disclosure may be encoded on and stored on a variety of computer-readable storage media. Suitable media include magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, and other non-transitory media. A computer-readable medium on which the program code is encoded may be packaged with a compatible electronic device, or the program code may be provided separately from the electronic device (e.g., via internet download or as a separately packaged computer-readable storage medium).

[0118] Therefore, although the disclosure has been described in terms of specific embodiments, it will be understood that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

1. 1. A method for performing an action for an event related to use of an application, comprising: identifying, by one or more processors, free text associated with the application, the free text to be evaluated with respect to at least one of a plurality of events associated with use of the application; applying, by the one or more processors, the free text to a machine learning (ML) architecture, the ML architecture being trained using a plurality of sample texts illustrating at least one of the plurality of phenomena associated with the use of the application; determining, by the one or more processors, a value indicative of a likelihood of an occurrence of an event associated with said use of said application based on application of said free text to said ML architecture; providing, by the one or more processors, model inputs based on the free text and the values ​​to a generative ML model to obtain data for an electronic document characterizing the events associated with the use of the application; performing, by the one or more processors, an action using the data for the electronic document; A method comprising:

2. 2. The method of claim 1, wherein applying the free text to the ML architecture further comprises applying the free text to the ML architecture including a natural language processing (NLP) model configured to access a plurality of information resources related to the application.

3. 3. The method of claim 2, wherein determining the value further comprises identifying an information resource of the plurality of information resources associated with the application based on application of the free text to the ML architecture using at least a portion of the free text.

4. 3. The method of claim 2, wherein providing the model input further comprises providing the model input to the generative ML model based on an information resource identified from the plurality of information resources associated with the application using at least a portion of the free text to obtain the data.

5. 10. The method of claim 1, wherein the ML architecture further comprises a classifier model constructed using the plurality of sample texts, each of the plurality of sample texts being labeled with a respective indication of the presence or absence of a respective phenomenon associated with the use of the application.

6. 2. The method of claim 1, wherein the plurality of events related to the use of the application include at least one of an adverse event, a serious adverse event, an incident, a serious incident, a software bug, a user complaint, or a usability issue.

7. 2. The method of claim 1, wherein determining the value further comprises classifying the event as at least one of an adverse event, a serious adverse event, an incident, a serious incident, a software bug, a user complaint, or a usability issue based on application of the free text to the ML architecture.

8. Performing the action comprises: determining that the event meets a reporting criterion for providing at least a portion of the electronic document to a remote device; transmitting the at least a portion of the electronic document to the remote device in response to determining that the event meets the reporting criteria; The method of claim 1 further comprising:

9. 2. The method of claim 1, wherein the generative ML model is trained using at least one corpus including sample inputs and sample outputs, the sample inputs identifying at least one of (i) a sample information resource associated with at least one of the plurality of events, or (ii) a sample value indicating a likelihood of the at least one event, and the sample outputs identifying at least one of (i) a diagnosis result of the at least one event, (ii) a sample action for the at least one event, or (iii) an analysis result of the at least one event.

10. 2. The method of claim 1, wherein performing the action for the event further includes selecting the action from a plurality of actions according to the data, the plurality of actions including at least one of: (i) terminating use of the application on a user device associated with the user; (ii) restricting operation of the application associated with the event; (iii) sending a notification to the user device for presentation to the user; (iv) providing the electronic document to an administrator device; or (v) storing the electronic document.

11. determining, by the one or more processors, that the value indicating the likelihood of the event meets a threshold; providing the model input further includes, in response to determining that the value satisfies the threshold, providing the model input to the generative ML model. The method of claim 1.

12. receiving, by the one or more processors, feedback via an interface identifying an updated value indicative of an updated likelihood of the event associated with the application; updating, by the one or more processors, at least one of a plurality of weights of the ML architecture based on the feedback; The method of claim 1 further comprising:

13. The method of claim 1 , further comprising generating, by the one or more processors, the model inputs according to a template to include context information based on the free text and the values.

14. 10. The method of claim 1, wherein identifying the free text further comprises obtaining the free text associated with the application from at least one of: (i) an email, (ii) a text message, (iii) a voice transcript, (iv) a chatbot message, (v) an electronic post, or (vi) a communication platform message.

15. identifying the free text includes: building an event listener on the application to monitor the free text generated by a user of the application; obtaining the free text via an application programming interface (API) of the event listener of the application; The method of claim 1 further comprising:

16. 2. The method of claim 1, wherein determining the value further comprises generating data elements that identify (i) an information resource associated with the application, (ii) the value, and (iii) a timestamp associated with a message that includes the free text.

17. 2. The method of claim 1, wherein providing the model input further comprises generating the electronic document including one or more recommendations for the event based on providing the model input to the generative ML model.

18. 10. The method of claim 1, wherein the application comprises a digital therapeutic application, and wherein an effective amount of medication to address a condition is administered to the user in conjunction with use of the digital therapeutic application.

19. 1. A system for events related to the use of an application, comprising: one or more processors coupled to a memory, identifying free text associated with the application, the free text to be evaluated for at least one of a plurality of events associated with use of the application; applying the free text to a machine learning (ML) architecture that has been trained using a plurality of sample texts that illustrate at least one of the plurality of phenomena associated with the use of the application; determining a value indicative of the likelihood of an event occurring associated with said use of said application based on application of said free text to said ML architecture; providing a generative ML model with model inputs based on the free text and the values ​​to obtain data for an electronic document characterizing the events associated with the use of the application; Performing an action using the data for the electronic document One or more processors configured to A system comprising:

20. 20. The system of claim 19, wherein the one or more processors are configured to apply the free text to the ML architecture including a natural language processing (NLP) model configured to access a plurality of information resources associated with the application.

21. 21. The system of claim 20, wherein the one or more processors are configured to identify, based on application of the free text to the ML architecture, an information resource of the plurality of information resources associated with the application using at least a portion of the free text.

22. 21. The system of claim 20, wherein the one or more processors are configured to provide the model input to the generative ML model based on an information resource identified from the plurality of information resources associated with the application using at least a portion of the free text to obtain the data.

23. 20. The system of claim 19, wherein the ML architecture further includes a classifier model constructed using the plurality of sample texts, each of the plurality of sample texts being labeled with a respective indication of the presence or absence of a respective phenomenon associated with the use of the application.

24. 20. The system of claim 19, wherein the one or more processors are configured to classify the event as at least one of an adverse event, a serious adverse event, an incident, a serious incident, a software bug, a user complaint, or a usability issue based on application of the free text to the ML architecture.

25. The one or more processors: determining that the event meets reporting criteria for providing at least a portion of the electronic document to a remote device; transmitting the at least a portion of the electronic document to the remote device in response to determining that the event meets the reporting criteria. It is configured as follows:

20. The system of claim 19.

26. 20. The system of claim 19, wherein the generative ML model is trained using at least one corpus including sample inputs and sample outputs, the sample inputs identifying at least one of (i) a sample information resource associated with at least one of the plurality of events, or (ii) a sample value indicating a likelihood of the at least one event, and the sample outputs identifying at least one of (i) a diagnosis of the at least one event, (ii) a sample action for the at least one event, or (iii) an analysis result of the at least one event.

27. 20. The system of claim 19, wherein the one or more processors are configured to select the action from a plurality of actions according to the data, the plurality of actions including at least one of: (i) terminating use of the application on a user device associated with the user; (ii) restricting operation of the application related to the event; (iii) sending a notification to the user device for presentation to the user; (iv) providing the electronic document to an administrator device; or (v) storing the electronic document.

28. the one or more processors are further configured to determine that the value indicating the likelihood of the event meets a threshold; the one or more processors are configured to, in response to determining that the value satisfies the threshold, provide the model input to the generative ML model.

20. The system of claim 19.

29. 20. The system of claim 19, wherein the one or more processors are configured to obtain the free text associated with the application from at least one of: (i) an email, (ii) a text message, (iii) an audio transcript, (iv) a chatbot message, (v) an electronic post, or (vi) a communication platform message.

30. 20. The system of claim 19, wherein the application includes a digital therapeutic application, and wherein an effective amount of medication to address a condition is administered to the user in conjunction with use of the digital therapeutic application.

Images