Systems and methods for regulating provision of messages with content from disparate sources based on risk and feedback data

Abstract

To provide systems and methods for regulating a digital therapeutic content for provision.SOLUTION: In a computing system in which a validation model is trained and applied to a content, a method for providing a user with a content includes: identifying a digital therapeutic content; applying the digital therapeutic content to a machine learning (ML) model having a set of weights; determining, from applying the digital therapeutic content to the ML model, an indication as any of compliance or non-compliance; and storing, using one or more data structures, an association between the digital therapeutic content and the indication used to control provision of the digital therapeutic content via a network, by restricting the digital therapeutic content from provision in response to determining the indication of non-compliance and permitting the digital therapeutic content to be provided in response to determining the indication of compliance.SELECTED DRAWING: Figure 14

Description

[Background technology]

[0001] In a networked environment, a server can send messages to an end user device to provide various information to the end user. The content in the messages can include erroneous or malicious information, or information that is not suitable for providing to the end user. In the context of digital therapeutics, for example, a message can include digital therapeutic content that includes inaccurate or unnecessary information regarding the end user's condition, which can have a negative impact on the end user's condition, such as worsening of the end user's symptoms or reduced compliance with the digital therapeutic.

[0002] Since end-user digital therapeutic treatment regimens include a wide variety of complex and personalized content, generating information about an individual's treatment regimen can be tedious and time-consuming to create manually. For digital therapeutics in particular, there are multiple topic categories that can provide input for effective treatment and can include knowledge from a variety of sources as factors. There is also a wide range of materials used to develop digital therapeutics in various mediums and formats. The complexity of individual end-users and regimens can make content generation resource-intensive and error-prone. Furthermore, human-generated content can lack scalability to a broader audience while at the same time lacking specificity to a particular end-user's symptoms.

[0003] Automated generation of digital therapeutic content using artificial intelligence (AI) techniques can involve similar challenges as well as the introduction of content items that are misleading, inaccurate, or improperly worded or visualized. For example, AI-generated content may contain inaccurate or insensitive content and may not be relevant to the end user. Furthermore, this problem can be exacerbated when multiple AI models are used to generate content items for a digital therapeutic regimen. Parsing through large volumes of AI-generated content for errors, relevance, and specificity is time-consuming and computationally challenging.

[0004] The lack of accuracy, relevance, and specificity in manually crafted or automatically generated content for end users, as well as the difficulty in regulating content on a larger scale for digital therapeutic regimens, can result in the wasteful consumption of computing resources (e.g., processors and memory) and network bandwidth by providing ineffective messages. From a human-computer interaction (HCI) perspective, these issues can lead to a lack of user interaction. In the context of digital therapeutics, users receiving such content may suffer reduced compliance with treatment regimens, which may result in worsening or no improvement in the end user's symptoms. Summary of the Invention

[0005] To address these and other challenges, the service described herein can manage an architecture of multiple interconnected risk models based on multiple domains and evaluate content created by human creators or generative AI (e.g., generative transformer models). Based on the evaluation, the service can determine what content, if any, to provide to the end user in a message (or item of content) to address the end user's specific symptoms with respect to a digital therapeutic application. Additionally, the service can make suggestions, if any, to modify the content, receive feedback on the provided content, and iteratively train both the risk model and the generative AI model maintained by the messaging system. Content items created using generative transformer models (e.g., large-scale language models or text-to-image models) can include content presented via the end user's device with the goal of preventing, mitigating, or treating the end user's symptoms. Content items can also include information provided for presentation on web pages and applications that are distinct from the digital therapeutic regimen provided to a particular user. By providing these capabilities, the service can improve the creation of digital therapeutic content and regulate the provision of digital therapeutic content through continuous learning in a single architecture.

[0006] The service may interface with a database that maintains a set of generative models as well as pre-generated content. To access the model or database, the service may receive input from an administrator. The input may generate or identify digital therapeutic content, such as identifying parameters, symptoms of interest, and domains for evaluating the content. The input may be provided to the generative transformer model as prompts. The prompts may be personalized based on the end user submitting the text input, thereby providing curated input to the generative transformer model to generate the content items. In addition, the system may use the input to search the database for generated or previously generated content items. In some embodiments, the service may determine that the text input corresponds to a content item already stored in the database and may not generate a prompt to provide to the generative transformer model. The service may identify a set of content items related to the text input from multiple sources, including content items generated by the generative transformer model based on prompts generated from the input, from previously AI-generated content items, from human-created content items, or from a combination thereof. The number of sources for digital therapeutic content may vary due to inherent differences in training between AI models and human creators.

[0007] Identification of content from AI or human sources allows the service to validate whether the content is suitable for provision to an end user (e.g., whether it will help alleviate or improve the end user's symptoms). To do so, the service can select one or more risk models to apply to the content items to determine a risk score for each content item. Each risk model can be selected for various parameters of the content to provide customized recommendations on which of the identified content items to provide in association with the text input. For example, the service can select a risk model based on, among other things, the audience indicated in the text input, the domain indicated in the text input, or the administrator providing the text input. Messages provided to an end user participating in a digital therapeutic regimen can fall into one or more domains, such as a regulatory domain, a medical domain, a product domain, or an audience experience domain. Selecting one or more risk models based on one or more domains indicated in the input provides further customization for the end user and helps improve compliance with the digital therapeutic regimen.

[0008] To arrive at an assessment of the content for risk, a set of risk models can be trained by the service to assess the risk associated with each content item identified according to its respective domain. Each risk model can be trained by the service on examples including the content item to determine whether the content item should be provided to the end user. In some embodiments, the risk models can be provided as a training data set including the content items. Each content item can be associated with a label including an expected indicator that identifies suitability or unsuitability for serving. In some embodiments, the indicator can include a risk score that is assigned to the content item for training purposes. The risk score can indicate suitability or unsuitability for serving, such that a risk score below a threshold risk score can indicate suitability. The risk models can be trained on a plurality of different content items with different corresponding domains, audiences, or combinations thereof to create multiple risk models, each adapted to a different content item.

[0009] During training, the service may apply each example content item to the risk model and determine a risk score or suitability indicator for each content item for serving. The determined suitability indicators may be compared to suitability indicators associated with the training content items. If the comparison indicates that the determined indicators correspond to the labeled indicators in the training data, the service may determine that the risk model successfully classified the content item. Otherwise, if the comparison indicates that the determined indicators correspond to the labeled indicators in the training data, the service may determine that the risk model failed to successfully classify the content item. The service may update the weights of the risk model accordingly. Additionally, the risk model may be trained to provide or indicate portions of the content items that may be edited to change the incompatible content items into compatible content items, and may present these portions to the administrator. For example, the risk model may be trained to present suggested edits, portions of the content items to change, or words to replace within the content items. These portions may be accepted or rejected by the administrator, further serving to train the risk model. Additionally, training of the risk model may be partially iterative, with an administrator providing feedback during training to maintain or modify the metrics output by the risk model. The risk model may accept responses indicating that content items indicated as suitable or unsuitable were improperly classified. In this manner, the risk model may continually learn and successfully determine suitable content items for facilitating a digital therapeutic treatment regimen.

[0010] With the establishment of the risk models, the service can apply the content item to each risk model and generate a risk score for the content item. The risk score can be used to identify whether the content item should be provided to an end user. The risk score can be determined by the risk model by providing the content item to the risk model for evaluation. The risk model can also take in additional inputs, such as information about the end user, text input, the individual generative transformer models that produced the content item, or the historical performance history of each generative transformer model. From the inputs, the service can select which risk model to apply based on the indicated domain.

[0011] Based on the risk score, the service may rank the content items and use the ranking to select the content items. For example, the service may identify the content items corresponding to the highest rank (e.g., lowest risk score) to provide to the end user. The service may also rank the content items based on other criteria, such as feedback from the audience to previous digital therapeutic content, audience preferences for previous digital therapeutic content, identification of the corresponding model used to generate the content item, or audience behavior on previous digital therapeutic content, among others. Through these means, the content provided to the end user may go through various filtering, analysis, and content checks to ensure that the content provided to the end user is most effective for their digital therapeutic regimen.

[0012] Selection of the content items based on the risk scores may cause the service to present the content items along with their corresponding risk scores to the administrator. The service may generate instructions to display the content items on the administrator device. The display may include the selected content items, their respective risk scores, their respective domains, and their respective audiences, among other information regarding the selected content items. Upon presentation, the administrator may provide interaction with the administrator device to provide feedback regarding the selected content items. The interaction may include modifications to the content items, such as modifying visual, auditory, or tactile properties of the content items. From the interaction, the service may generate feedback data. The feedback data may include information such as content, modifications to the content, or modifications to indicators included in a message provided to the user. Feedback data from the administrator may be used to update the generative transformer model itself. As an example, the service may use the feedback data to calculate a loss metric as a function of the feedback data, and subsequently use the loss metric to update weights in the generative transformer model. The feedback data may be used to generate subsequent prompts when using the generative transformer model to create messages for end users. For example, the service may add at least a portion of the feedback data as part of the user information for the administrator or message generation parameters when generating the prompt.

[0013] Upon receiving feedback from the administrator, or upon determining that there is no administrator feedback, the service may store the content item in a database of content items. The service may store the content item in association with an indicator. The service may store the content item with an indicator indicating whether the content item may be provided to the end user. For example, the service may store the content item in association with a suitability indicator indicating that the content item may be provided to the end user.

[0014] Storage of the content item enables the service to send a message including the content item for presentation at the end user device. The content items may be presented in a variety of media. For example, an application running on the end user device may present the content of the message. When presented via the end user device, the message may prompt the end user to perform an activity to facilitate the digital therapeutic treatment regimen. The application on the end user device may monitor the end user's interaction with the message or the application itself. The interaction may include, for example, an indication that a particular activity has been performed or input of the end user's reaction to the presentation of the message, among other responses. Using the detected interaction, the application may generate and send a response to provide to the service.

[0015] In addition, the service may send messages to end user devices for presentation to a wider audience. For example, the service may send messages that include content items of publicly accessible web pages, such as the main article and supplemental content (e.g., in an inline frame on the web page). The content items may include information about clinical trials with participants for digital therapeutic applications. The content items may be submitted for publication, such as in a medical journal, newspaper, or online database. For example, the content items may be included as top-level content of a web page or as supplemental content inserted into a portion of a web page (e.g., in an inline frame). The content items may be presented in a wide range of media.

[0016] When the service receives a response from the end user device, it can use it to update the generative Transformer model itself. The service can analyze the response of the end user device and generate feedback data. The feedback data can include information such as the content included in the message provided to the user and an indication of whether the end user performed an activity identified in the content of the message. The feedback data can be used to update the generative Transformer model itself. As an example, the service can use the feedback data to calculate a loss metric as a function of the feedback data, and then use the loss metric to update weights in the generative Transformer model. The feedback data can be used to generate subsequent prompts when using the generative Transformer model to generate messages for the end user. For example, the service can add at least a portion of the feedback data as part of the user information or message generation parameters when generating a prompt. The service can combine feedback data from any number of previously presented messages when generating a prompt to input to the generative Transformer model.

[0017] In this way, the service may iteratively and continuously factor in feedback from administrator response data before setting up the content and end-user feedback data when providing content to both the generative transformation model maintained by the service and the risk model. Output from the risk model may be provided to the generative transformation model to continuously train the generative transformation model for different domains, audiences, and text inputs. This technical solution allows content generated by other AI models (in addition to the content generated by the generative transformer model maintained by the messaging system) to be validated as suitable and then ranked against each other to determine the most relevant content to provide to the end user.

[0018] Additionally, with regard to human moderation of content, the technical solution may enable scalability for creating personalized digital therapeutic content by enabling the generation and validation of a wide variety of personalized content for provision to end users. Providing more relevant content may reduce resource consumption by reducing the computational power spent on generating and sending messages that are not relevant to the end user. Enabling flexibility, scalability, and specificity may optimize or reduce the consumption of computing resources (e.g., processors and memory) and network bandwidth that would otherwise be wasted by providing ineffective content.

[0019] In the context of digital therapeutics, new generation of content may consist of changes in the end-user's condition, such as improvement or worsening of the end-user's symptoms or progress in a treatment regimen. By iteratively incorporating feedback and continually training the generative transformer model and risk model, the HCI may be improved by providing more relevant and accurate content for a particular end-user. The content provided by leveraging the risk model may provide a higher quality of interaction with the application by the end-user. Additionally, increased engagement may result in a higher level of adherence by the end-user to the treatment regimen, thereby increasing the likelihood of preventing, alleviating, or treating the end-user's symptoms.

[0020] Aspects of the present disclosure are directed to a system, method, and computer-readable medium for configuring the generation of digital therapeutic content for delivery. A computing system may receive a text input including one or more parameters via a user interface. The one or more parameters may identify an audience for which the digital therapeutic content is generated and at least one domain of a set of domains to test the digital therapeutic content for risk. The set of domains may include a regulatory domain and a medical domain. The computing system may identify a set of content items generated by a corresponding set of generative transformer models, each using a prompt generated based on the text input. Each of the set of content items may include a respective digital therapeutic content generated from a respective model of the set of generative transformer models. The computing system may select at least one risk model corresponding to the at least one domain from a set of risk models for the set of domains. The computing system may apply the at least one risk model to each content item of the set of content items and determine a corresponding risk score. The computing system may select a content item from the set of content items based on the risk score of the content item. The computing system may present the content items including the respective digital therapeutic content via the user interface.

[0021] In some embodiments, the computing system may use the text input to retrieve at least one previously generated content item stored in a database. In response to determining that a second risk score of the at least one content item meets a second threshold, the computing system may add the at least one content item to the set of content items. In some embodiments, the computing system may determine that no previously generated content items are stored in the database corresponding to the text input. In response to determining that no previously generated content items are stored, the computing system may invoke a set of generative transformer models to generate a set of content items using prompts generated based on the text input.

[0022] In some embodiments, the computing system may apply a second risk model to one or more parameters of the text input to determine a second risk score. The computing system may present, via the interface, an indication that the second risk score exceeds a threshold. In some embodiments, the computing system may apply multiple risk models associated with a set of domains to the content item to determine a corresponding set of risk scores. The set of domains may further include a product domain and an audience experience domain. The computing system may present, via a user interface, information identifying a set of risk scores for the content item.

[0023] In some embodiments, the computing system may rank the set of content items based on at least one of (i) audience feedback to the previous digital therapeutic content, (ii) audience preferences for the previous digital therapeutic content, (iii) identifying a corresponding model used to generate the content items, and (iv) audience behavior with respect to the previous digital therapeutic content. In some embodiments, the computing system may apply a second risk model to one or more parameters of the text input to determine a second risk score. The computing system may apply a set of generative models to generate a set of content items corresponding to the second risk scores meeting a threshold.

[0024] In some embodiments, the computing system may receive, via a user interface, information identifying one or more portions of the content item to be modified. The computing system may store the association of the information with the content item in a database. In some embodiments, the computing system may identify, from a set of generative Transformer models, a generative Transformer model to be updated based on at least one of the text input, the content item, or a response from a presentation of the content item. The computing system may provide at least a portion of the text input, the content item, or the response data to update the generative Transformer model identified from the set of generative Transformer models.

[0025] In some embodiments, each content item in the series of content items may include at least one of textual or visual content provided to a device for presentation in a session and address a symptom of an audience, where the audience is receiving a medication to address the symptom at least partially contemporaneously with the session.

[0026] Aspects of the present disclosure are directed to a system, method, and computer-readable medium for regulating digital therapeutic content for delivery. A computing system may identify a first digital therapeutic content to be provided over a network. The computing system may apply the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output. The ML model may be trained by identifying a training dataset including a set of examples, each example of the set of examples identifying a respective second digital therapeutic content and a first indicator identifying either a match or a mismatch for delivery. The ML model may be trained by applying the second digital therapeutic content from an example of the set of examples of the training dataset to the ML model to generate a second output. The ML model may be trained by determining a second indicator from the second output as either a match or a mismatch of the second digital therapeutic content used to control delivery. The ML model may be trained by comparing the first indicator from the examples of the training dataset to the second indicator determined by the ML model. The ML model may be trained by updating at least one of the set of weights of the ML model according to the comparison. The computing system may determine an indicator from applying the first digital therapeutic content to the ML model as being either suitable or unsuitable. The computing system may store, using one or more data structures, a link between the first digital therapeutic content and the indicator used to control provision of the first digital therapeutic content over the network by (i) restricting provision of the first digital therapeutic content in response to determining the indicator as unsuitable, and (ii) allowing provision of the first digital therapeutic content in response to determining the indicator as suitable.

[0027] In some embodiments, a computing system may receive an identification that the first digital therapeutic content is associated with at least one domain of a set of domains. The computing system may select an ML model corresponding to the domain to apply to the first digital therapeutic content from a plurality of ML models. The computing system may determine the indicator by determining the indicator as either suitable or unsuitable with respect to the domain. In some embodiments, the computing system may generate one or more modifications in the first digital therapeutic content from applying the ML model in response to determining the indicator of the first digital therapeutic content as unsuitable.

[0028] In some embodiments, the computing system may select one of the first or second content items from the set of content items as the first digital therapeutic content for delivery based on a suitability or non-suitability indicator for each of the set of content items determined using the ML model. In some embodiments, the computing system may receive, via a user interface, a selection of a second indicator identifying the first digital therapeutic content as being suitable or non-suitable. The computing system may override the indicator from the ML model with the second indicator received via the user interface.

[0029] In some embodiments, each example of the set of examples of the training dataset further identifies a first risk score identifying a degree of suitability or non-suitability for provision of the second digital therapeutic content. The computing system may determine the second indicator by determining a second risk score identifying a degree of suitability or non-suitability for provision of the first digital therapeutic content. In some embodiments, the first digital therapeutic content may include at least one of textual or visual content provided to the device for presentation in a session addressing a symptom of the audience, where the audience is receiving a medication to address the symptom at least partially contemporaneously with the session.

[0030] Aspects of the present disclosure are directed to a system, method, and computer-readable medium for regulating the provision of digital therapeutic content. A computing system may identify a training dataset including a set of examples, each of which identifies a respective first digital therapeutic content and a first indicator identifying either compatibility or non-compatibility, and control the provision of the first digital therapeutic content over a network by (i) restricting the provision of the first digital therapeutic content when non-compatibility and (ii) allowing the provision of the first digital therapeutic content when compatibility. The computing system may apply the first digital therapeutic content from one of the set of examples of the training dataset to a machine learning (ML) model having a set of weights to generate an output. The computing system may determine a second indicator of either compatibility or non-compatibility of the first digital therapeutic content used to control the provision. The computing system may compare the first indicator from the examples of the training dataset to the second indicator determined by the ML model. The computing system may update at least one of the set of weights of the ML model according to a comparison between the first indicator and the second indicator. The computing system may store a set of weights for the ML model to apply to determine suitability or non-suitability of the second digital therapeutic content.

[0031] In some embodiments, the computing system may receive, via a user interface, a selection of indicators identifying the second digital therapeutic content as compatible or incompatible. The computing system may compare the indicators determined from the output of the ML model with the indicators from the user interface. The computing system may retrain the ML model according to the comparison. In some embodiments, the computing system may generate one or more portions to modify in the second digital therapeutic content from applying the ML model in response to determining an indicator of the second digital therapeutic content as incompatible.

[0032] In some embodiments, the computing system may retrain the ML model using one or more portions to modify the second digital therapeutic content. In some embodiments, the computing system may modify prompts applied to a generative transformer model used to output the first digital therapeutic content in response to determining a non-fit of the second digital therapeutic content, where the generative transformer model is part of the ML model or is distinct from the ML model. In some embodiments, the computing system may retrain a generative transformer model used to generate the first digital therapeutic content based on an indication of either non-fit or fit with the second digital therapeutic content.

[0033] In some embodiments, each example of the set of examples of the training dataset may further identify a first risk score that identifies a degree of suitability or unsuitability for delivery of the first digital therapeutic content. The computing system may apply the first digital therapeutic content from one example of the set of examples of the training dataset to the ML model and determine a second risk score that identifies a degree of suitability or unsuitability for delivery of the first digital therapeutic content. The computing system may update at least one of the set of weights of the ML model according to a comparison between the first risk score and the second risk score for the example.

[0034] In some embodiments, each example of the set of examples of the training dataset may include an identification of the first digital therapeutic content as a domain of the set of domains and a first indicator identifying either a match or a mismatch with respect to the domain. The computing system may identify an ML model from the set of ML models to apply to the first digital therapeutic content corresponding to the domain. The computing system may determine the second indicator by determining a second indicator of either a match or a mismatch of the first digital therapeutic.

[0035] The above and other objects, aspects, features, and advantages of the present disclosure will become more clearly and better understood by referring to the following description taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0036] [Figure 1] FIG. 1 is a block diagram of a system for generating and regulating content for messages targeted to address a user's symptoms, according to an example embodiment. [Diagram 2] FIG. 2 is a block diagram of a content identification process in a system for generating and regulating content for targeted messages according to an example embodiment. [Diagram 3]FIG. 1 is a block diagram of a process for selecting one or more models to apply to determine a risk score in a system for generating and regulating content for targeted messages, according to an example embodiment. [Figure 4] FIG. 2 is a block diagram of a process for selecting content items to send in a message to an administrator device in a system for generating and regulating targeted messages, according to an example embodiment. [Figure 5A] FIG. 1 is a block diagram of a process for updating a generative transformer model in a system for regulating and generating content for targeted messages, according to an example embodiment. [Figure 5B] FIG. 1 is a block diagram of a process for updating a generative transformer model in a system for regulating and generating content for targeted messages, according to an example embodiment. [Figure 5C] FIG. 1 is a block diagram of a process for updating a generative transformer model in a system for regulating and generating content for targeted messages, according to an example embodiment. [Figure 6] FIG. 2 is a block diagram of a system architecture for generating content items in a system for regulating and generating content for targeted messages according to an example embodiment. [Figure 7] 1 is a flow chart of a method for generating and regulating content for targeted messages, according to an example embodiment. [Figure 8] FIG. 1 is a block diagram of a process for training and applying a validation model to content according to an example embodiment. [Figure 9] 1 is a block diagram of a process for training a validation model using a training dataset in a system for training and applying a validation model to content according to an example embodiment. [Figure 10]FIG. 1 is a block diagram of a process for capturing feedback for a validation model in a system for training and applying a validation model to content, according to an example embodiment. [Figure 11] FIG. 2 is a block diagram of a process for providing content items to users in a system for training and applying a validation model to content, according to an example embodiment. [Figure 12A] 1 is an example user interface for a system for training and applying a validation model to content, according to an example embodiment. [Figure 12B] 1 is an example user interface for a system for training and applying a validation model to content, according to an example embodiment. [Figure 12C] 1 is an example user interface for a system for training and applying a validation model to content, according to an example embodiment. [Figure 13] 1 is a flowchart of a method for training and applying a validation model to content in a system for training and applying a validation model to content, according to an example embodiment. [Figure 14] 1 is a flowchart of a method for providing content to a user in a system for training and applying a validation model to content, according to an example embodiment. [Figure 15] 1 is a block diagram of a server system and a client computer system according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] In order to understand the following description of the various embodiments, the following listing of sections of this specification, and their respective contents, may be helpful.

[0038] Section A describes systems and methods for generating and regulating content for messages targeted to address a user's symptoms.

[0039] Section B describes systems and methods for training and applying validation models to content.

[0040] Section C describes network and computing environments that may be useful in practicing the embodiments described herein.

[0041] A. Systems and methods for generating and regulating content for messages targeted to address a user's symptoms Referring now to FIG. 1, a block diagram of a system 100 for generating and regulating content for messages targeted to address end-user symptoms is shown. In overview, the system 100 may include at least one data processing system 105, a set of end user devices 110A-N (hereinafter generally referred to as end user devices 110), and at least one administrator device 190 communicatively coupled to each other via at least one network 115. The administrator device 190 may include at least one application 120. The application 120 may include or provide at least one user interface 125 having one or more user interface (UI) elements 130A-N (hereinafter generally referred to as UI elements 130). The data processing system 105 may include, among other things, at least one content handler 140, at least one content controller 145, at least one risk evaluator 150, at least one feedback handler 155, at least one model trainer 160, at least one generative transformer model 165A-N, and at least one risk model 185A-N. The data processing system 105 may include, or have access to, at least one database 135. The database 135 may store, maintain, or include, among other things, one or more end user profiles 170A-N (hereinafter generally referred to as end user profiles 170) and one or more content items 180A-N. The functionality of the application 120 in the administrator device 190 may be implemented in part in the data processing system 105, or vice versa.

[0042] Within the data processing system 105, the content handler 140 may receive input and generate content items 180 through generative transformer models 165A-N (hereinafter generally referred to as generative transformer models 165) and provide selected content items 180 associated with a session initiated by an administrator of the application 120 at the administrator device 190. The content controller 145 may identify the content items 180. The risk evaluator 150 may apply the risk models 185A-N to determine a risk score for each of the content items 180. The feedback handler 155 may use responses from the administrator device 190 to generate feedback and update the generative transformer model 165, the content items 180, or the risk models 185A-N (hereinafter generally referred to as risk models 185). The model trainer 160 may train, improve, or update the generative transformer model 165 or the risk model 185 associated with a session initiated by an administrator of the application 120.

[0043] More specifically, data processing system 105 may be any computing device including one or more processors coupled with memory and software capable of executing the various processes and tasks described herein. Data processing system 105 may communicate with one or more user devices 110, administrator devices 190, and databases 135 via network 115. Data processing system 105 may be located within, installed in, or associated with at least one computer system. A computer system may correspond to a data center, branch office, or site where one or more computers corresponding to data processing system 105 are located.

[0044] The end user device 110 may be any computing device having one or more processors connected with memory and software capable of executing the various processes and tasks described herein. The end user device 110 may communicate with the data processing system 105, the administrator device 190, and the database 135 via the network 115. The end user device 110 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), or laptop computer. The end user device may be provided with one or more content items 180 via the data processing system 105. Alternatively, the end user device may request one or more content items 180 via an interaction with the data processing system 105, such as via an application associated with the end user device 110.

[0045] The administrator device 190 may be any computing device comprising one or more processors coupled with memory and software capable of executing the various processes and tasks described herein. The administrator device 190 may be associated with an entity in conjunction with the data processing system 105 to control and regulate the generation of the content items 180. The administrator device 190 may communicate with the data processing system 105, the user devices 110, and the database 135 via the network 115. The administrator device 190 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), or laptop computer. The administrator device 190 may be used to access the application 120. In some embodiments, the application 120 may be downloaded and installed on the administrator device 190 (e.g., via a digital distribution platform). In some embodiments, the application 120 may be a web application having resources accessible via the network 115.

[0046] An application 120 executing on the administrator device 190 may interface with the data processing system 105 to generate or modify one or more content items 180. In some embodiments, the application 120 may be an application that interfaces with a digital therapeutic application that addresses at least one symptom of the end user to generate one or more messages for delivery to an audience of end users associated with the end user device 110. End user conditions may include, for example, chronic pain (e.g., associated with or including arthritis, migraine, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive stress disorder, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin conditions (e.g., atopic dermatitis, psoriasis, self-inflicted dermatosis, and eczema), cognitive disorders (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis, and schizophrenia), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, emotionally unstable personality disorder, and attention deficit hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, tobacco use disorder, or hallucinogen disorder), and other conditions (e.g., narcolepsy and oncology, or cancer), among others.

[0047] The end user may be administered medication to address the symptoms at least in part at the same time. By way of example, if the medication is for pain, the end user may be administered acetaminophen, a nonsteroidal anti-inflammatory composition, an antidepressant, an anticonvulsant, or other composition, among others. For skin lesions, the end user may be administered steroids, antihistamines, or topic antiseptics, among others. For cognitive impairment, the end user may be administered cholinesterase inhibitors or memantine, among others. For narcolepsy, the end user may be administered stimulants or antidepressants, among others. The end user may participate in other psychological therapies for these symptoms. In some embodiments, the content items 180 may be provided to the end user within a digital therapeutic application to reach the end user's endpoint. The endpoint may be, for example, the end user's physical or mental goals, the completion of a medication regimen, or an endpoint indicated by a physician or the end user. At least one of the end user devices 110 may have a digital therapeutic application and may provide a session (sometimes referred to herein as a therapy session) that addresses at least one symptom of the end user.

[0048] In some embodiments, application 120 may be an application that generates one or more content items 180 for submission to medical journals, government agencies, or subject matter experts. Application 120 may be an application that generates or modifies summaries associated with one or more of clinical trials, pilots, journal entries, or publications, among other things. For example, application 120 may be an application that generates clinical summary reports, with content relating to patient symptoms, medications, psychotherapy, or treatments, among other things.

[0049] The application 120 may include, or may be presented or provided to, an administrator of the administrator device 190, a user interface 125 that includes one or more user interface elements 130A-N (hereinafter generally referred to as UI elements 130) according to a configuration of the application 120. The UI elements 130 may correspond to visual components of the user interface 125, such as command buttons, text boxes, check boxes, radio buttons, menu items, and sliders, among others. In some embodiments, the administrator, in conjunction with the data processing system 105, may interact with the UI elements 130 to provide feedback, responses, or other interactions, and to generate and modify content items 180.

[0050] The database 135 may store and maintain various resources and data associated with the data processing system 105 and the applications 120. The database 135 may include a database management system (DBMS) to arrange and organize data maintained therein, such as end user profiles 170, content items 180, risk models 185, or generative transformer models 165, among others. The database 135 may communicate with the data processing system 105, the administrator device 190, and one or more end user devices 110 via the network 115. While performing various operations, the data processing system 105 and the applications 120 may access the database 135 and read identified data therefrom. The data processing system 105 and the applications 120 may write data to the database 135 from the performance of such operations.

[0051] In the database 135, each end user profile 170 (sometimes referred to herein as end user account or end user information) can store and maintain information about an end user through the end user device 110. Each end user profile 170 can be associated with or correspond to a respective end user to whom content items 180 have been provided. The end user profile 170 can identify various information about the end user, such as an end user identifier, symptoms addressed, information about sessions conducted by the end user (e.g., completed activities or lessons or other content items 180 generated or modified by an administrator), preferences, user trait information, and progress in addressing symptoms (e.g., completion of endpoints), among others. Information about sessions can include various parameters of previous sessions conducted by the end user, which can be initially null. Preferences can include message preferences. Message preferences can include treatment preferences and end user input preferences, such as preferred message type or message timing. Message preferences can also include preferences determined by the data processing system 105, such as the type of message to which the end user may respond. The preferences may include summary preferences, such as words, phrases, or content items 180 preferred by the administrator for inclusion in the content items 180. The end-user profiles 170 may be continually updated by the applications 120 and the data processing system 105.

[0052] In some embodiments, the end-user profile 170 may identify or include information about the treatment regimen undertaken by the end-user, such as the type of treatment (e.g., medical, pharmaceutical, or psychotherapy), duration (e.g., daily, weekly, quarterly, annually), and frequency (e.g., daily, weekly, quarterly, annually), among other things. The end-user profile 170 may include at least one activity log of messages provided to the end-user, interactions by the end-user identifying the performance of that particular end-user, and responses from the end-user device 110 associated with the end-user, among other things. The end-user profile 170 may be stored and maintained in the database 135 using one or more files (e.g., extensible markup language (XML)), comma-separated values ​​(CSV) delimited text files, or structured query language (SQL) files). The end-user profile 170 may be iteratively updated as the end-user conducts additional sessions, provides input, or responds to content items 180.

[0053] The content items 180 may be in any format, such as text, images, audio, video, or multimedia content, or any combination thereof, among others. The content items 180 may be stored and maintained in the database 135 using one or more files. By way of example, in the case of text, the content items 180 may be stored as text files (TXT), rich text files (RTF), extensible markup language (XML), and hypertext markup language (HTTP), among others. In the case of images, the content items 180 may be stored as joint photographic experts' group (JPEG) format, portable network graphics (PNG) format, graphics interchange format (GIF), or scalable vector graphics (SVG) format, among others. If audio, content item 180 may be stored as a waveform audio file (WAV), motion pictures expert group format (e.g., MP3 and MP4), and Ogg Vorbis (OGG) format, among others. If video, content item 180 may be stored as a motion pictures expert group format (e.g., MP3 and MP4), QuickTime movie (MOV), and Windows Movie Video (WMV), among others.For multimedia content, the content items 180 can be audio video interleave (AVI), Motion Pictures Experts Group formats (e.g., MP3 and MP4), QuickTime Movie (MOV), and Windows Movie Video (WMV), among others.

[0054] Each content item 180 may identify or include information to be presented via the end user device 110 or administrator device 190. For example, the content item 180 may be presented to the end user or administrator through a message sent to the end user device 110 or administrator device 190, respectively. The message may be in any format, such as short message / messaging service (SMS), multimedia messaging service (MMS), among others, or as instructions for presentation via a display associated with the end user device 110 or administrator device 190.

[0055] Message content items 180 may include reminders to perform a session task. The messages may be taken from a library of pre-generated psychotherapy messages or a library of pre-generated engagement (reminder) messages. The messages may include reminders for the end user to complete a therapy session, take a medication dose, or complete a regimen task. The messages may include an activity for the end user to perform or a lesson in which the end user is engaged. Content items 180 may include a response mechanism, such as a link, chat box, or indicator to respond to the message.

[0056] A content item 180 may include or correspond to one or more pieces of text, such as an article, abstract, or publication. For example, a content item 180 may include a research paper, a review article, a case report, a clinical trial protocol, or an editorial, among others. A content item 180 may include text for submission to a government agency, a subject matter expert, a scientific journal, or a conference, among others. For example, a content item 180 may include a clinical trial protocol regarding a treatment provided for an end-user's condition for submission to the Food and Drug Administration (FDA), a medical journal, or for internal distribution.

[0057] Content items 180A-N may be generated and stored in database 135 prior to [x] generating and modifying content items 180 operating on administrator device 190. Content items 180 may be human-generated, computer-generated, or a combination thereof. In some embodiments, an administrator may provide content items 180 through an application 120 operating on administrator device 190. For example, an administrator may upload, provide, or transfer one or more content items 180 to application 120 for storage in database 135 during a session of generating and modifying content items 180. Content items 180 may be computer-generated, such as by a generative transformer model 165. In some embodiments, an administrator may provide input through an application 120 operating on administrator device 190 to create one or more content items 180 using generative transformer model 165. For example, an administrator may provide text, images, video, or other representations as input to generate content items 180. One or more generative transformer models 165 can generate one or more content items 180 from prompts generated by input from the administrator. In some embodiments, the data processing system 105 can modify the input to include additional information and generate prompts to provide to the generative transformer models 165. Each generative transformer model 165 can generate one or more content items 180 based on the provided prompts.

[0058] Generative Transformer model 165 may receive input in the form of a series of strings (e.g., from a text input) and output content (e.g., content item 180) in one or more modalities (e.g., in the form of a text string, audio content, image, video, or multimedia content). Generative Transformer model 165 may be a machine learning model following a Transformer model (e.g., a generative pre-trained model, or a bidirectional encoder representation from a Transformer). Generative Transformer model 165 may be a large language model (LLM), a text-to-image model, a text-to-speech model, or a text-to-video model, among others. In some embodiments, generative Transformer model 165 may be part of data processing system 105 (e.g., as shown). In some embodiments, generative Transformer model 165 may be part of a server separate from data processing system 105 and communicate with data processing system 105 via network 115.

[0059] One or more of the generative transformer models 165 may be trained and maintained by the data processing system 105. The generative transformer model 165 may include a set of weights arranged across a set of layers according to a transformer architecture. Under the architecture, the generative transformer model 165 may include, among others, at least one tokenization layer (sometimes referred to herein as a tokenizer), at least one input embedding layer, at least one positional encoder, at least one encoder stack, at least one decoder stack, and at least one output layer interconnected with each other (e.g., via forward, reverse, or skip connections). In some embodiments, the generative transformer layer 165 may lack an encoder stack (e.g., in the case of an encoder-only architecture) or a decoder stack (e.g., in the case of a decoder-only model architecture). The tokenization layer may convert raw input in the form of a set of strings into a corresponding set of word vectors (also referred to herein as tokens or vectors) in an n-dimensional feature space. The input embedding layer may use the set of word vectors to generate a set of embeddings. Each embedding can be a low-dimensional representation of a corresponding word vector and can capture semantic and syntactic information of the string associated with the word vector. A positional encoder can generate a positional encoding for each input embedding depending on the position of the corresponding word vector, or by extension, the position of the string within the input sequence of strings.

[0060] Subsequently, in the generative transformer model 165, the encoder stack may include a series of encoders. Each encoder may include, among others, at least one attention layer and at least one feedforward layer. The attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each input embedding, indicating the degree of attention the embedding should pay attention to, and generate a weighted sum of the series of input embeddings. The feedforward layer may apply a linear transformation with a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the attention layer. The output may be fed to another encoder in the encoder stack in the generative transformer layer 165. When the encoder is a terminal encoder in the encoder stack, the output may be fed to a decoder stack.

[0061] The decoder stack may include, among others, at least one attention layer, at least one encoder / decoder attention layer, and at least one feedforward layer. In the decoder stack, the attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each output embedding (e.g., an embedding generated from a target output or an expected output). The encoder / decoder attention layer may combine inputs from an attention layer in the decoder stack with an output from one of the encoders in the encoder stack, and may calculate an attention score from the combined inputs. The feedforward layer may apply a linear transformation with a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the encoder / decoder attention layer. The output of the decoder may be sent to another decoder in the decoder stack. When the decoder is a terminal decoder in the decoder stack, the output may be sent to an output layer.

[0062] The output layer of the generative transformer model 165 may include, among others, at least one linear layer and at least one activation layer. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to calculate the token scores. The activation layer may apply an activation function (e.g., softmax, sigmoid, or rectified linear unit) to the output of the linear function to convert the token scores into probabilities (or distributions). The probabilities may represent the likelihood of the occurrence of an output token given an input token. The output layer may use the probabilities to select an output token (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability). This may be repeated over a series of input tokens, and the resulting series of output tokens may be used to form the output of the generative transformer model 165 as a whole. Although described primarily in terms of Transformer models herein, the data processing system 105 may also use other machine learning models to generate and output content. In some implementations, the data processing system 105 may generate and output content using one or more models maintained by an external system. For example, the data processing system may generate the content using one or more models such as ChatGPT created by OpenAI, BARD created by Google, or LLaMA created by Meta, among others.

[0063] Each generative transformer model 165 may create one or more of the content items 180 based on the prompts provided to the generative transformer model 165. Each content item 180 created from the prompts generated from the text input provided to the data processing system 105 may be different due to differences between each of the generative transformer models 165. As such, content item 180A may be more suitable for presentation to an end user (initially through an administrator via user interface 125) than other content items 180B-N. For example, content item 180 generated by generative transformer model 165 may include errors, irrelevant content, or hallucinations. In some embodiments, content item 180A generated by generative transformer model 165A may not be relevant to a particular user due to the information in content item 180A, the symptoms addressed by content item 180A, the presentation style of content item 180A, or the grand assertions provided by content item 180A. For example, content item 180A may assert that it is the “best” method of treatment for a given condition, but this cannot be asserted and erroneous information is provided. For example, content item 180A may encourage a user to eat a meat-based diet without recognizing that the user previously indicated vegetarianism. For example, content item 180A may be in a style of text presentation, even though the user's historical behavior since the previous session indicates that the user is more consistent and faithful to the session when presented with video content. For example, content item 180A may generate data that has not been verified or proven to be true. To moderate the content items 180 created by generative transformer model 165, data processing system 105 may train and employ one or more risk models 185 on the content items 180 and determine a risk score associated with each content item 180.

[0064] Each risk model 185 can be a machine learning model trained to determine a risk score associated with a content item 180, a prompt, or a combination thereof. The risk models 185 can be trained as described herein (in conjunction with training the validation model) to calculate a risk score for the content item 180, the prompt, or a combination thereof. The risk scores generated by the risk models 185 can be further used to continually train the generative transformer model 165 to provide more relevant, more accurate, or less risky content items 180 over time. The risk models 185 can include one or more natural language models, including the generative transformer models 165 described herein. The risk models 185 can include one or more classifier models, such as a Naive Bayes Classifier, a support vector machine (SVM) ensemble classifier, a kernel approximation, a K-nearest neighbor classifier, or a decision tree, among others.

[0065] One or more of the risk models 185 can accept the prompt as an input. By accepting the prompt as an input, the one or more risk models 185 can generate a risk score associated with the likelihood of the particular prompt and generate a desired content item 180. The desired content item 180 can include a content item 180 in a format specified by the prompt, for a group or audience of people specified in the prompt, for a domain specified in the prompt, with a desired accuracy (e.g., accurate information, relevant data set), or with a desired relevance (e.g., an end user receiving the content item 180 as part of a digital therapeutic session, or an administrator receiving text of a desired article type), among others. One or more of the risk models 185 can accept the generated content item 180 as an input. By accepting the content item 180 as an input, the one or more risk models 185 can generate a risk score associated with the likelihood of the content item 180 being a desired content item 180, as described above.

[0066] 2, a block diagram of a process 200 for identifying content in the system 100 for generating and regulating content for targeted messages is shown. The process 200 may include or correspond to operations performed in the system 100 for generating and regulating content for targeted messages. Under the process 200, the content handler 140 may receive, read, or identify text input that creates a prompt 230 that is provided to the generative transformer model 165. The risk evaluator 150 may use the risk model 185 to generate a risk score 235 for the prompt 230.

[0067] The administrator 205 may provide an interaction 210 with the user interface 125 via a UI element 130 to create or modify a content item 180. The interaction 210 may be part of a request to create or modify a submitted activity or message. For example, the interaction 210 may be provided in response to a submission by the application 120 to create or modify a content item 180 at the administrator device 190. The interaction 210 may include an interaction with a UI element 130. In some embodiments, the interaction 210 may include the administrator 205 providing text through the user interface 125, among other things, a text box, a drop-down menu, or speech-to-text. In some embodiments, the interaction 210 may include the administrator 205 making a selection via a drop-down box, a button, or another UI element 130. For example, the interaction 210 may include the administrator 205 making a selection of a type of content item 180 (e.g., journal article, clinical trial report, etc.).

[0068] The application 120 running on the administrator device 190 can accept the interaction 210 and generate a text input 225 based on the interaction 210. In some embodiments, the interaction 210 includes a text input that the application 120 can utilize in generating the input 225. In some embodiments, the application 120 generates the input 225 from a non-text interaction, such as a button or dropdown selection, among others. The input 225 (also referred to as the text input 225) can include one or more parameters used to define the generation of a message presented to the administrator 205, such as an audience 215A-N or a domain identifier 220A-N.

[0069] The application 120 can identify audiences 215A-N (hereinafter generally referred to as audiences 215) indicated in the input 225. The audience 215 can refer to or include one or more people or one or more institutions for which the content item 180 is intended. For example, the audience 215 can include a grouping of patients according to similar demographics, such as patients suffering from similar symptoms, similar message preferences, severity of symptoms, or medication prescriptions, among others. For example, the audience 215 can include a grouping of subject matter experts in similar fields of study, similar educational backgrounds, or similar geographies, among others. For example, the audience 215 can include entities such as government agencies, educational institutions, or non-profit organizations.

[0070] The application 120 can identify domain identifiers 220A-N (hereinafter generally referred to as domain identifiers 220) indicated in the input 225. The domain identifiers 220 can correspond to or indicate one or more domains of the input 225. A domain can correspond to an intent to generate a content item 180. A domain can correspond to one or more of the generative transformer models 165. For example, the generative transformer model 165A can correspond to a first domain such that the generative transformer model 165A utilizes, is trained against, or generates the content item 180 based on the domain. A domain can include a library of words, phrases, or rules for generating a content item 180, such as generating a content item 180 for information about a particular digital therapeutic treatment. A domain can include domains such as an audience experience domain, a regulatory domain, a suitability domain, a scientific or medical domain, or a product domain, among others.

[0071] The domain identifiers 220 may correspond to domains that test the content items 180 for risk, such as a science or medical domain (e.g., criteria related to scientific research or medical literature, such as whether the information is medically or scientifically accurate, clinically and statistically relevant, presented in a scientifically balanced manner, etc.), a regulatory domain (e.g., criteria related to regulatory guidance, such as no safety or effectiveness claims are made before the product is cleared or approved by a regulatory agency), an audience experience domain (e.g., criteria related to user experience, such as clear and understandable instructions, easily accessible technical assistance, more attractive reward systems, etc.), a suitability domain (e.g., criteria related to suitability requirements), a product domain (e.g., criteria related to product requirements, such as including certain features within the scope of the project), a commercial domain (e.g., criteria related to commercial launch requirements), or a marketing domain (e.g., criteria related to marketing products), among others. The domains may be derived from end-user behavior, end-user preferences, or profile information about a given audience 215. A domain may be identified by an end-user profile 170 associated with an audience 215. For example, an audience 215 may include an end-user profile 170 that is related to conditions such as chronic pain, skin conditions, cognitive disorders, mental health conditions, or substance use disorders, among others.

[0072] In some embodiments, the administrator 205's profile may indicate the domain. For example, the administrator 205's settings (or profile) may indicate that the administrator 205 is a product manager, and the application may thereby identify the product domain for the input 225. In some embodiments, the application 120 may include preferences, as selected by the administrator 205 or determined from the administrator 205's settings, to identify the domain. In some embodiments, the administrator 205 may identify the domain through interaction 210. For example, the administrator 205 may select the domain from a drop-down list of domains or may provide text indicating the domain through interaction 210.

[0073] In some embodiments, the science or medical domain may define a selection of content items 180 for end users suffering from a particular condition. In some embodiments, the medical domain may define appropriate psycho-educational lessons for a particular medical condition, appropriate activities for the end user to engage in as part of a treatment for a particular medical condition, or text that meets a reading level specific to a particular patient subpopulation for the generation of the content items 180. In some embodiments, the medical domain may define templates for the content items 180, such as a format for an article or journal publication, among others. In some embodiments, the audience experience domain may include end user preferences obtained from profile information in the end user profile 170. The end user preferences may correspond to the type of messages preferred by the end user as identified in the end user profile 170 of the end user corresponding to the audience 215. The profile information may include other data points about the end user, such as progress of treatment, difficulty, stage towards achieving an endpoint, or end user behavior, among others. End user behavior may include, for example, the types of activities performed by the end user, as specified in the end user profile 170 .

[0074] In some embodiments, the product domain may include user experience criteria for a particular audience. The marketing domain may define a library of marketing phrases for generation of content items 180. In some embodiments, the marketing domain may define templates for content items 180, such as product release statements, product advisories, press releases, or marketing materials for a product. In some embodiments, the regulatory domain may define regulatory guidance for establishing and employing clinical trials and provide various environments for the generative transformer model 165 to generate the desired content items 180 (e.g., promotional materials should not mention that a device is safe or effective before it has been cleared or approved by a regulatory agency). In some embodiments, the regulatory domain may provide templates or lists of content items 180 for regulatory submissions, clinical trial protocols, press releases, or recall statements, among others.

[0075] A content handler 140 executing on the data processing system 105 may read, obtain, or receive input 225 from an end user device 110. In generating the input 225, the content handler 140 may perform any of the functions of the application 120. In some embodiments, the content handler 140 may read information from one or more end user profiles 170 to generate the input 225. In some embodiments, the content handler 140 may identify a profile corresponding to the administrator 205 from the input 225. The content handler 140 may identify, among other things, an identifier (e.g., name), information about sessions conducted by the administrator 205, an activity log, message preferences. The content handler 140 may identify parameters of the input 225. For example, the content handler may identify an audience 215 or domain identifier 220 of the input 225.

[0076] In some embodiments, upon receiving the input 225, the risk evaluator 150 may determine a risk score 235 for the input 225. The risk evaluator 150 may determine the input 225 based on at least parameters of the input 225, such as the audience 215 or domain identifier 220 of the input 225. The risk evaluator 150 may apply one or more risk models 185 to the input 225 (or prompts obtained from the input 220) and generate a risk score 235 for the input 225. Upon applying the one or more risk models 185 to the input, the risk evaluator 150 may generate a risk score 235 for the input.

[0077] Applying the risk models 185 to the input 225 can include selecting one or more risk models 185. The risk evaluator 150 may select one or more risk models 185 to apply to the input 225 based on parameters of the input 225. For example, the risk evaluator 150 may select a first risk model 185A for a domain identifier 220 corresponding to a medical domain and a second risk model 185B for a domain identifier 220 corresponding to a product domain. In some embodiments, the parameters of the input 225 may specify or include one or more domain identifiers 220 or audiences 215. The risk evaluator 150 may select one or more risk models 185 corresponding to each of the parameters specified in the input 225.

[0078] Applying the risk models 185 may include providing the input 225 as an input to one or more selected risk models 185. Providing the input 225 may include the risk evaluator 150 applying one or more parameters of the input 225. For example, the risk evaluator 150 may provide as input the domain identifier 220 of the input 225, the audience 215 of the input 225, other parameters of the input 225, or a combination thereof. In some embodiments, the risk evaluator 150 may provide a first parameter of the input 225 to a first risk model 185A and a second parameter of the input 225 to a second risk model 185B. For example, the risk evaluator 150 may apply different, same, or overlapping risk models 185 to each of the parameters (e.g., the audience 215, the domain identifier 220, the format type of the content item, etc.). In this manner, one or more risk scores 235 may be generated for each of the parameters of the input 225 .

[0079] The risk score 235 generated by the risk evaluator 150 based on the inputs 225 can be a number, score, level, or other identifier indicating the likelihood that the inputs 225 correspond to generating a desired content item 180. In some embodiments, a low risk score may indicate that the content item 180 generated based on the inputs 225 is more likely to correspond to the desired audience 215, domain, format type, or other parameter of the inputs 225. In some embodiments, a low risk score may indicate that the content item 180 generated based on the inputs 225 is more likely to meet a regulatory or accuracy compliance threshold, such as the accuracy of the data, information, citations, or other facts contained within the content item 180. In some embodiments, a low risk score may indicate that the content item 180 generated based on the inputs 225 is more likely to meet a relevance threshold, such as relevance to the audience 215 or compliance with end-user or administrator preferences. For example, relevance may relate to information in the end-user profile 170, such as symptoms suffered by the end-user or audience 215, administrator 205 preferences for types of content items 180, among others. In some embodiments, a low risk score may indicate that the input 225 is unlikely to include the subset of words or phrases, or that the input 225 is unlikely to generate content items 180 that include the subset of words or phrases.

[0080] The content handler 140 may generate the prompt 230 based on the input 225, including the audience 215 and the domain identifier 220. In some embodiments, the content handler 140 may generate the prompt 230 in response to a risk score 235 meeting a threshold. For example, the content handler 140 may generate the prompt 230 if the risk score 235 is below a threshold risk score. In some embodiments, the content handler 140 may not generate the prompt 230 for generating the content item 180 when the risk score 235 is equal to or greater than the threshold risk score. The content handler 140 may provide an indication that the risk score 235 is equal to or greater than the threshold risk level for presentation in the user interface 125. For example, the content handler 140 may generate a presentation that displays the risk score 235. The indication may include, among other things, an identification of the risk score 235 for each parameter of the input 225, the overall risk score 235 for the input 225.

[0081] The content handler 140 may generate the prompt based on information identified from the profile of the administrator 205, such as previous sessions for creating or modifying content items by the administrator 205, the occupation of the administrator 205, the preferred content items of the administrator 205, domains associated with or selected by the administrator 205, among others. The content handler 140 may generate the prompt 230 based on a string in the input 225. For example, the text input 225 may indicate or include a word, phrase, or string that identifies parameters for the content item 180 to be generated. The qualities may include, among others, the audience 215, the domain identifier 220, the file type of the content item 180, the format type (e.g., journal article, image, animation, clinical trial protocol, etc.), or the size of the content item (e.g., file size, word count, number of pages). The content handler 140 may generate the prompts based on information identified from an end user profile 170 corresponding to the audience 215, such as, among other things, the endpoints the audience 215 must achieve to address the symptoms, the symptoms being addressed, activity logs, end user traits, an end user identifier, message preferences, information about sessions conducted by the end user, and progress in addressing the symptoms.

[0082] The content handler 140 may write, create, or generate at least one prompt 230 using information from the end user profile and one or more parameters. When generating, the content handler 140 may add, insert, or include information from the end user profile 170 and one or more parameters into a template for generating the prompt. The template may include a set of defined strings and one or more placeholders that are inserted between strings for input to the generative transformer model 165. The strings may correspond to predefined, fixed text that is included as part of the input prompt for the generative transformer model 165. By traversing the template and inserting data, the content handler 140 may form and build the prompt 230.

[0083] 3, a block diagram is shown of a process 300 for selecting one or more risk models 185 to apply to determine risk scores 305A-N in a system 100 for generating and regulating content for targeted messages. The process 300 may include or correspond to operations performed in a system 100 for generating and regulating content items 180 for targeted messages. Under the process 300, the content controller 145 may identify one or more content items 180. A risk evaluator 150 running in the data processing system 105 may identify one or more risk models 185 based on the domain 220 of the content items 180. The risk evaluator 150 may apply the models 185 to the content items 180 to determine one or more risk scores 305A-N for the content items 180.

[0084] Upon generating the prompt 230, the content handler 140 may send or apply the prompt 230 to the generative transformer model 165. When applying, the content handler 140 may process the prompt 230 using a series of layers in the generative transformer model 165. As described above, the generative transformer model 165 may include, among others, a tokenization layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer. The content handler 140 may process the prompt 230 using the tokenizer layer of the generative transformer model 165 to generate a series of word vectors (sometimes referred to herein as word tokens or tokens) for the input set. Each word vector may be a vector representation of at least a portion of the prompt 230 in an n-dimensional feature space (e.g., using a word embedding table). The content handler 140 may apply the series of word vectors to the input embedding layer to generate a corresponding series of embeddings.

[0085] Through processing, content handler 140 may calculate a probability for each embedding. The probability may represent the likelihood of an output occurring given the input token. Based on the probability, content handler 140 may select the output token (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability) and form, create, or generate at least a portion of content item 180. Content item 180 output by generative transformer model 165 may include, among others, text content, image content, audio content, video, or multimedia content, or any combination thereof.

[0086] In some embodiments, the content items 180 created by the generative transformer model 165 may include textual content, image content, audio content, video, or multimedia content, among others, or any combination thereof. For example, the presentation of the content items 180 may include an image of a smiley face to further encourage the user to achieve their goal. Additionally, the content items 180 may include video clips of loved ones, motivational speakers, medical professionals, influencers, or therapists, among others, or any combination thereof.

[0087] The content controller 145 may identify content items 180 for provision to the administrator 205 via the applications 120 running on the administrator device 190. Identifying the content items 180 may include retrieving, receiving, or obtaining the content items 180 from one or more sources, such as from the generative transformer model 165 or the database 135. The content controller 145 may identify content items 180 in the database 135 that correspond to the input 225. For example, the content controller 145 may identify pre-generated, produced, or uploaded content items 180 stored in the database 135.

[0088] The content controller 145 may determine that the stored content items 180 correspond to the input 225 provided by the administrator 205. The content controller 145 may identify strings in the input 225 or parameters of the input 225 that relate to, correspond to, or are associated with the stored content items 180. For example, the input 225 may indicate an audience 215A that is also identified in the stored content item 180A. The content controller 145 may determine that one or more strings in the input 225 match or are similar to one or more strings in the content items 180 stored in the database 135. In this manner, the content controller 145 can identify the content items 180 that correspond to the input 225 in the database 135 and can retrieve one or more content items 180 that were previously generated and stored in the database 135.

[0089] The content controller 145 may prevent the content handlers 140 from generating the prompt 230 in response to determining that one or more content items 180 in the database 135 correspond to the input 225. In some embodiments, the content controller 145 may access the database 135 to determine whether one or more content items 180 in the database 135 correspond to the input 225 prior to the generation of the prompt 230 by the content handlers 140 or prior to providing the generated prompt 230 to the generative transformer model 165. The content controller 145 may instruct the content handlers 140 to refrain from generating the prompt 230 or to refrain from providing the generated prompt 230 to the generative transformer model 165. In some embodiments, the content controller 145 may prevent the content handlers 140 from providing the generated prompt 230 to the generative transformer model 165 in response to the content handlers 140 identifying one or more corresponding content items 180 in the database 135.

[0090] The content controller 145 may determine that no pre-generated content item 180 is stored in the database corresponding to the input 225. The content controller 145 may access the database 135 and determine that no pre-generated content item 180 is stored in the database 135 corresponding to the input 225. In some embodiments, the content controller 145 may access the database 135 and make the determination prior to the generation of the prompt 230 by the content handler 140 or before the content handler 140 provides the generated prompt to the generative transformer model 165 to generate the content item 180. In response to determining that no pre-generated content item 180 is stored in the database 135, the content controller 145 may invoke the generative transformer model 165 to generate the content item 180 using the prompt 230 created based on the input 225. Invoking the generative transformer model 165 may include the content controller 145 instructing the content handler 140 to generate a prompt 230 or to provide the generated prompt 230 to the generative transformer model 165 to generate the content item 180.

[0091] The content controller 145 may rank the identified content items 180 according to one or more criteria. The content controller 145 may rank the identified content items 180 upon identifying the content items 180. Alternatively, the content controller 145 may rank the identified content items 180 prior to providing the content items 180 to the administrator device 190. The one or more criteria used by the content controller 145 to rank the content items 180 may include feedback from the audience 215 on previous digital therapeutic content. For example, the data processing system 105 may receive feedback by the audience 215 on content items previously presented to the audience 215. The content controller 145 may rank the generated content items 180 based on the feedback from the audience 215. The feedback may include a numerical rating of the content item 180, textual feedback describing the rating of the content item 180, or a binary indication of the content item (eg, "approved" or "rejected").

[0092] The content controller 145 may rank the generated content items 180 based on criteria including the audience's preferences for previous digital therapeutic content. For example, the audience may indicate a preferred type of content item 180, such as a preferred format (video, text, audio), a preferred duration for presenting the content item 180, a preferred style of the content item 180 (color, font type, paragraph formatting, etc.), among others. The criteria used by the content controller 145 to rank the content items 180 may include the audience's behavior with respect to previous digital therapeutic content. For example, the end user profile 170 presented to or included in the audience 215 may indicate one or more content items 180 or types of content items 180 that have elicited more response, engagement, or adherence from the audience 215 participating in a digital therapeutic regimen.

[0093] The content controller 145 may rank the content items 180 based on the reading level of the content items 180. For example, the content controller 145 may determine the reading level of each of the identified content items 180 and may rank the content items 180 according to the reading level. Reading level may refer to the age, amount of education, or literacy level at which one reads or understands a passage of text, such as a text-based content item 180. In some embodiments, the content controller 145 may rank the content items 180 based on the connection of the reading level of the content items 180 to the reading level indicated by the audience 215. For example, the content controller 145 may rank the content item 180A higher or more favorably than the content item 180B if the content item 180A has the same or similar reading level as the audience 215.

[0094] The one or more criteria may include identifying a corresponding model used to generate the content item 180. For example, each content item 180 may be generated by a respective generative Transformer model 165. The content controller 145 may rank the content items 180 generated by the generative Transformer model 165 according to which generative Transformer model 165 generated the respective content item 180. In some embodiments, a content item 180A generated by a generative Transformer model 165A maintained, trained, or associated with the data processing system 105 may yield a higher or more favorable ranking than a content item 180B generated by a generative Transformer model 165B not maintained or trained by the data processing system 105.

[0095] The risk evaluator 150 may select, choose, or identify a risk model 185 from the database 135 to apply to the identified content item 180. The risk evaluator 150 may identify the risk model 185 in response to receiving input 225 from an application running on the administrator device 190. The risk evaluator 150 may retrieve the risk model 185 in response to generation of a prompt 230 by the content handler 140. The risk evaluator 150 may access the database 135 and retrieve the risk model 185 to apply to the identified content item 180 at any time.

[0096] The risk evaluator 150 may select a model 185 based on one or more domains 310 associated with the input 225. The risk evaluator 150 may identify one or more domains 310 associated with the input 225 based on the domain identifier 220 of the input 225. In some embodiments, one or more of the risk models 185 may correspond to a particular domain of the domains 310. For example, a first risk model 185A may correspond to a medical domain and a second risk model 185B may correspond to an audience experience domain. The risk evaluator 150 may select one or more risk models 185 that correspond to the domain 310 of the input 225. Corresponding to the input 225 may refer to the domain identifier 220 of the input 225 matching the domain for which the one or more risk models 185 are trained. One or more risk models 185 may be trained for one or more specific domains 310 by having a training set provided to the one or more risk models 185 trained for the specific domain that includes phrases, wording, or content items of the specific domain.

[0097] The risk evaluator 150 may select multiple models 185 corresponding to one or more domains 310. In some embodiments, the input 225 may identify multiple domains 310. For example, the input 225 may identify multiple domains 310, such as a product domain, a medical domain, and a regulatory domain. The risk evaluator 150 may select a risk model 185 corresponding to each of the domains 310 identified in the input 225. For example, for an input 225 identifying a product domain, a medical domain, and a regulatory domain, the risk evaluator 150 may identify a first risk model 185A corresponding to the product domain, a second risk model 185B corresponding to the medical domain, and a third risk model 185C corresponding to the regulatory domain. In some embodiments, the risk models 185 can correspond to multiple domains. For example, the risk model 185A can correspond to a medical domain and a scientific domain. The risk evaluator 150 may select a risk model 185 that corresponds to one or more of the domains identified by the domain identifiers 220 of the input 225 .

[0098] The risk evaluator 150 may apply the selected risk model 185 to the identified content items 180 to generate one or more risk scores 305A-N (hereinafter generally referred to as risk scores 305). In some embodiments, applying the risk model 185 to the content items 180 may include the risk evaluator 150 providing the identified content items 180 as inputs to the model 185 to generate a risk score 305 associated with each content item. In some embodiments, a subset of the content items 180 may be applied to the risk model 185. For example, the content items generated by the first generative Transformer model 165A may not be provided to the first risk model 185A. In this manner, the content items 180 generated by the generative Transformer model 165 maintained or trained by the data processing system 105 may not rely on one or more of the risk models 185.

[0099] The risk evaluator 150 may provide the content item 180 to a number of risk models 185 corresponding to each domain identifier 220 of the input 225 and generate a respective risk score 305 from each risk model 185 applied to the content item 180. In some embodiments, the risk evaluator 150 may generate a risk score 305 for each domain 310 associated with the input 225. The risk score 305 may be a score that identifies a level of risk associated with the content item 180 or the input 225. In some embodiments, the risk score 305 may define a level of risk for the domain 310. The risk score 305 may be similar to or may include the risk score 235.

[0100] The risk score 305 may be used to determine whether the content item 180 is permitted or restricted from being provided to the audience 215 (e.g., directly or through the administrator 205 via an application 120 running on the administrator device 190). In some embodiments, a first risk score 305A that is lower, smaller, or less than the second risk score 305B may indicate that the first risk score is more likely to be suitable for provision. In general, a low risk score 305 may indicate that the content item 180 generated based on the input 225 is more likely to correspond to the desired audience 215, domain, format type, or other parameters of the input 225. Generating the risk score 305 may include the risk evaluator 150 applying a risk model 185 to determine how well each content item 180 fits into the domain 310.

[0101] Using the risk model 185, the risk evaluator 150 may determine a risk score 305 indicating the extent to which the generated content item 180 information meets the criteria of the domain 310 associated with the risk model 185. For example, the risk score 305 may indicate the degree of conformance or satisfaction of a phrase, word, or portion with the criteria of the particular domain 310 associated with the risk model 185. For example, a regulated domain may specify that including "will" or "is" in relation to efficacy should be avoided. If the content includes such a claim (e.g., "this product will improve your condition"), the corresponding risk model 185 may output a high risk score 305 indicating the criteria of the regulated domain have not been met.

[0102] The risk score 305 may correspond to suitability or non-suitability of the content item 180 according to a set of criteria for a particular domain. Suitability of the content item 180 may refer to the content item 180 being suitable for serving. Suitability for serving may include the content item 180 exceeding a threshold of suitability for a parameter identified in the input 225, such as the domain or audience 215. In some embodiments, a risk score 305 below a threshold risk score may indicate that the content item 180 corresponding to the risk score 305 is suitable. In some embodiments, a risk score 305 equal to or greater than the threshold risk score may indicate that the content item 180 corresponding to the risk score 305 is not suitable.

[0103] 4, a block diagram is shown of a process 400 for selecting a content item 180A to send in a message to an administrator device 190 in a system 100 for generating and regulating content for targeted messages. The process 400 may include or correspond to operations performed in the system 100 for generating and regulating content items 180 for targeted messages. Under the process 400, a content controller 145 executing in a data processing system 105 may select one or more content items 180 based on their corresponding risk scores 305. The content controller 145 may provide one or more content items 180 to a content handler 140 for provision to the administrator device 190.

[0104] The content controller 145 may select one or more content items 180A based on the risk scores 305 associated with the content items 180A. Once the risk scores 305 are generated by the risk evaluator 150, the content controller 145 may select one or more content items 180A based on their respective risk scores. In some embodiments, the content controller 145 may select the content item 180A with the lowest risk score. The content controller 145 may select one or more content items 180A with a risk score below a threshold risk score. Alternatively, the content controller 145 may select the content item 180A or set of content items 180A with the lowest risk score. In some cases, the content controller 145 may select one or more content items 180A with a risk score equal to or greater than a threshold risk score, or a set of content items 180A with the highest risk score. The highest or lowest risk score 305 may be determined to be the maximum or minimum risk score 305, respectively, of the set of risk scores 305 generated by the risk evaluator 150 for the input 225.

[0105] The content controller 145 may select the content item 180A based on the content controller 145's ranking of the content items. As described herein, the content controller 145 may rank the content items 180 based on various criteria including, but not limited to, audience feedback on previous digital therapeutic content, audience preferences for previous digital therapeutic content, identification of the corresponding generative transformer model 165 used to generate the content item 180, or audience behavior on previous digital therapeutic content, among others. The content controller 145 may select the content item 180A that has the most favorable or highest ranking by the content controller 145. In some embodiments, the content controller 145 may select the content item 180A based on the highest ranked content item 180A among a set of content items that have a risk score below a threshold risk score. For example, the content controller 145 may select the content item 180A that is most likely to increase the audience's adherence to a digital therapeutic regimen that also has a risk score 305 below the threshold risk score.

[0106] The content handler 140 may provide the selected content item 180A to the administrator device 190. The content handler 140 may provide the selected content item 180A in a message sent to the administrator device 190. The message may include instructions for presentation of the content item 180A on the user interface 125 by an application 120 running on the administrator device 190. The user interface 125 may present the content item 180A according to the instructions in the message.

[0107] For transmission, the content handler 140 may generate at least one instruction for presenting the content item 180A to send to the administrator device 190. The instruction may include an identifier for the message or the content item 180A. In some embodiments, the instruction may be a code, a data packet, or a control for presenting a message to the administrator 205. The instruction may include processing instructions for displaying the message on the application 120 through the UI element 130 of the user interface 125. The instruction may include instructions for the administrator 205 to perform with respect to the session for creating and modifying the content item. For example, the instruction may display a message including the content item 180A that instructs the end user to perform a particular activity associated with the session. In some embodiments, the instruction may follow a messaging protocol such as short message service (SMS) or multimedia messaging service (MMS). The instruction may identify the administrator device 190 to send the message to (e.g., using a phone number or network address) as well as the content item 180A of the message in the payload. Upon generation, the content handler 140 may send instructions to the administrator device 190 over the network 115 .

[0108] Upon receipt, the administrator device 190 may provide, display, or present the message via a display, such as a user interface 125. In some embodiments, an application 120 at the administrator device 190 may provide, display, or present the message via a user interface 125. For example, instructions for the message may prescribe, specify, or define the layout (e.g., positioning, size, and color) for individual UI elements 130 when the message is presented via the user interface 125 of the application 120. In some embodiments, the application 120 at the administrator device 190 may provide a sandbox environment to simulate the presentation of the message to an end user. For example, when the message is delivered according to a messaging protocol (e.g., SMS and MMS), the application 120 at the administrator device 190 may present a messaging application to mimic various SMS messaging applications and present the message.

[0109] In some embodiments, the message may include information regarding the risk scores 305 of the content items 180. For example, the message may identify the risk scores 305 associated with each content item 180. The message may indicate whether the content item 180B corresponds to a risk score 305B that meets a risk threshold. The application 120 may display a list, table, or other representation of the risk scores 305 associated with each content item. In some embodiments, the application 120 may display a list, mapping, or other representation indicating the generative transformer models 165 associated with each content item 180. For example, the application may display, for the content item 180B, the associated risk score 305B and the associated generative transformer model 165B. The application 120 may display information regarding the risk scores 305 via the user interface 125. For example, the application 120 may display a mapping between any of the content items 180 and, in particular, their respective risk scores 305, generative transformer models 165, or domains 310, as prompted.

[0110] In some embodiments, the message may include information regarding the risk score 235 corresponding to the input 225. The message may include an indication that the risk score 235 corresponding to the input 225 exceeds a threshold risk score. For example, the indication may include that the risk score 235 associated with the input 225 exceeds a threshold risk score. The application 120 may display the indication as a list, table, or other presentation using the UI element 130 of the user interface 125. In some embodiments, the application 120 may display a list, mapping, or other presentation showing parameters associated with the input 225. For example, the application may display the associated risk score 235, the associated domain identifier 220, or the associated audience 215, among others, for the input 225 as an indication.

[0111] 5A-5C, a block diagram of a process 500 for updating a generative transformer model in a system 100 for regulating and generating content for targeted messages is shown. The process 500 may include or correspond to operations performed in the system 100 for generating and regulating content items 180 for targeted messages. Under the process 500, an application 120 in an administrator device 190 may receive an interaction 210 from an administrator 205. A feedback handler 155 may receive a generated response 515 from the administrator device 190. The feedback handler 155 may receive a generated response 515' from a presentation of the content item at an end user device 110. The feedback handler 155 may provide feedback data 505 based on the response to a model trainer 160 to train a generative transformer model 165.

[0112] 5A , upon presentation of the content item 180A to the administrator 205, the administrator device 190 may monitor for at least one interaction 210 by the administrator 205 in response to the presentation of a message including information regarding at least one of the content items 180A or the risk score 305 or 235. The interaction 210 may include data (e.g., text) entered by the administrator 205 in response to the message. In some embodiments, the interaction 210 may include data identifying one or more portions of the content item 180A to be retained or modified. For example, the administrator 205 may indicate, through the interaction 210, a selection of one or more portions of the content item 180A through the UI element 130. The one or more portions may correspond to a portion of the content item 180A to be modified. In some embodiments, the administrator 205 may modify a portion of the content item 180A through the interaction 210 with the UI element 130. For example, the administrator 205 can provide interactions 210 to edit (e.g., delete, add, format) the text of the content item 180A on the user interface 125, edit the color or style of the content item 180A, or edit other presentations of the content item 180A.

[0113] Upon detecting the interaction 210, the administrator device 190 may output, create, or generate at least one administrator response 515 for transmission to the data processing system 105. The administrator response 515 may indicate, include, or identify the portion of the content item 180A to be maintained or modified. For example, the administrator response 515 may include a connection of the content item 180A to be modified with one or more portions of the content item 180A to be modified. The administrator response 515 may include a time of the response 515.

[0114] 5B, the end user 505 may provide an interaction 510 with the presented content item. In some embodiments, the content item 180A may be presented to the end user 505 on the end user device 110. Upon approval from the administrator 205, the content item 180A may be stored in the database 135 for provision to the end user 505. In some implementations, the end user device 110 may access the database 135 and retrieve, retrieve, or access the content item 180A stored therein. The end user device 110 may present the content item 180A to the end user 505 via a user interface 125 operating on the end user device 110. The end user device 110 may present the content item 180A, among other things, as part of a digital therapy session addressing symptoms of the end user 505, as open text such as a publication or magazine article, or as an advertisement on a web browser or other application operating on the end user device 110.

[0115] The end user device 110 may monitor an interaction 510 with the content item 180A in the user interface 125. The interaction 510 may include some of the functionality of the interaction 210 by the administrator 205. For example, the interaction 510 may be contemporaneous with the presentation of a message including the content item 180A. For example, the interaction 510 may correspond to an interaction of playing a video clip included in the message through the presentation of a play button through an application running on the end user device 110. In some embodiments, the interaction 510 may be subsequent to the presentation of the message at the end user device 110. As an example, the interaction 510 may correspond to a series of interactions and recording end user feedback regarding the content item 180A in the message subsequent to the presentation of the message to the end user.

[0116] The interaction 510 may include data (e.g., text) entered by the end user 505 in response to the message. In some embodiments, the interaction 510 may include data identifying one or more portions of the content item 180A to be modified. For example, the end user 505 may indicate a selection of one or more content items 180A to be presented to the end user 505 through the interaction 510. The end user 505 may be prompted to provide feedback based on the content items 180A, such as whether the end user 505 enjoyed the content or participated in an activity indicated by the content. In some cases, the end user device 110 may monitor the interaction 510 with one or more of the UI elements 130. For example, the end user device 110 may monitor the time between the presentation of the message including the content item 180A and the interaction 510, compliance with a treatment regimen when the content item 180A was presented, or subsequent activities performed by the end user 505 with respect to the presentation of the content item 180A.

[0117] When the interaction 510 is detected, the end user device 110 may output, create, or generate at least one end user response 515' for transmission to the data processing system 105. The end user response 515' may indicate, include, or identify the UI element 130 shown in the end user interaction 510 for the presentation of the message. The end user response 515' may include a time of the response 515', such as the time between the presentation of the message including the content item 180A and receipt of the interaction 510. The end user response 515' may indicate other times associated with the presentation of the message including the content item 180A. For example, the end user response 515' may include a reading time for the end user 505 to read the content item 180A.

[0118] In some embodiments, an application at the end user device 110 may use the detected interaction 510 to generate an end user response 515'. The response 515' may identify, among other information, the event associated with the interaction 510 by the end user 505, a timestamp of the presentation of the message to the end user 505, a timestamp of the event, and an identifier of the end user 505. The end user response 515' may include data entered by the end user 505 via a user interface 125 operating on the end user device 110. In some embodiments, an application operating on the end user device 110 may maintain a timer that tracks the time elapsed since the presentation of the message to the end user 505. The application may compare the elapsed time to a time limit for the message. If the elapsed time exceeds the time limit, the application may generate an end user response 515' to indicate the lack of end user interaction with the message. Upon generation, the application or the end user device 110 may provide, deliver, or transmit the response 515' to the feedback handler 155.

[0119] 5C , the feedback handler 155 may create, generate, or generate feedback data 505 based on the administrator response 515 and the end user response 515′. The feedback data 505 may be used to train the generative transformer model 165. The feedback data 505 may include information obtained from the administrator response 515. For example, the feedback data 505 may include an identification of one or more portions to be retained or modified. The feedback data 505 may include information obtained from the end user response 515. The feedback data 505 may include an indication of whether the end user interacted with the content item when presented.

[0120] In some embodiments, the feedback handler 155 may generate feedback data 505 for subsequent generation of the content item by the generation transformer model 165. In some embodiments, the feedback data 505 may identify or include information that is used as one or more parameters that define the subsequent content item that is generated and presented for the end user 505. During generation, the feedback handler 155 may store and maintain in the database 135 a connection between the feedback data 505 and the end user profile 170. The feedback data 505 may indicate one or more portions to be modified in the content item 180. During generation, the feedback handler 155 may store and maintain in the database 135 a connection of the one or more portions with the content item 180A.

[0121] In some embodiments, the feedback handler 155 may generate feedback data 505 including information used to update weights associated with the generative transformer model 165. The feedback data 505 may be generated to include information from the message content items 180A, the input 225, or responses 515 from the administrator 405. The feedback data 505 generated from the end user responses 515 may indicate the degree to which the presented message elicits interaction from the end user 505. The number or duration of the end user's 505 interactions with the content items 180A may be recorded and included within the feedback data 505. Upon generation, the feedback handler 155 may include end user interaction information and the message content items 180A in the feedback data 505.

[0122] The model trainer 160 may use the feedback data 505 to modify, adjust, or update weights of the generative transformer models 165. The model trainer 160 may identify one or more of the generative transformer models 165 to update based on the feedback data 505. The feedback data 505 may be aggregated across multiple responses 515 from an administrator 205 or from multiple administrators 205. For example, the content item 180A may be reviewed multiple times by different administrators or the same administrator 205. The feedback data 505 may similarly be aggregated across multiple responses 515' from an end user 505 or from multiple end users 505. For example, the content item 180A may be presented to one or more end users 205 at one or more user devices 110 during each digital therapeutic session. Each end user 205 presented with the content item 180A may provide an interaction 510 in response to the presentation of the content item 180A.

[0123] In general, the model trainer 160 may update weights to allow creation of messages with high performance metrics or low risk scores 305 or 235 and penalize output messages with low performance metrics. Training or fine-tuning the generative transformer model 165 using feedback data 505 may be similar to training or fine-tuning the risk model 185 as described herein. To train, the model trainer 160 may define, select, or identify at least a portion of each feedback data 505 as training data and at least a portion of each feedback data 505 as test data. The training data may be used to input into the generative transformer model 165 and create output that is compared to the test data. A portion of each feedback data 505 may at least partially overlap with and correspond to a subset of the text strings in the feedback data 505.

[0124] The model trainer 160 may send or apply strings of training data from the feedback data 505 to the generative transformer model 165. When applying, the model trainer 160 may process the input strings according to a series of layers in the generative transformer model 165. As described above, the generative transformer model 165 may include, among others, a tokenization layer, an input embedding layer, a positional encoder, an encoder stack, a decoder stack, and an output layer. The model trainer 160 may use the tokenizer layer of the generative transformer model 165 to process the input strings of training data (words or phrases in alphanumeric format) and generate a set of word vectors for the input set. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a word embedding table). The model trainer 160 may apply the set of word vectors to the input embedding layer to generate a corresponding set of embeddings. The model trainer 160 may identify the position of each string within the set of strings of the source set. By specifying, model trainer 160 can apply a positional encoder to the position of each string to generate a positional encoding, and in turn, an embedding, for each embedding corresponding to the string.

[0125] Model trainer 160 may apply the set of embeddings along with the corresponding set of positional encodings generated from the input set of feedback data 505 to the encoder stack of generative transformer model 165. When applying, model trainer 160 may process the set of embeddings along with the corresponding set of positional encodings according to the layers (e.g., attention layers and feedforward layers) in each encoder of the encoder block. From the processing, model trainer 160 may generate another set of embeddings to forward to the encoders in the encoder stack. Model trainer 160 may then send the output of the encoder stack to the decoder stack.

[0126] Together, model trainer 160 may process the data (e.g., text, image, audio, video, or multimedia content) of the test data using another tokenizer layer of generative transformer model 165 to generate a set of word vectors for the test data. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a word embedding table). Model trainer 160 may apply the set of word vectors to the input embedding layer to generate a corresponding set of embeddings. Model trainer 160 may identify the position of each string within the set of strings of the target set. Upon identification, model trainer 160 may apply a positional encoder to the position of each string to generate a positional encoding, and in turn, an embedding, for each embedding corresponding to the string.

[0127] The model trainer 160 may apply the set of embeddings along with the corresponding set of positional encodings generated from the destination set of feedback data 505 to the decoder stack of the generative transformer model 165. The model trainer 160 may combine the output of the currently processed encoder stack through the decoder stack. In applying, the model trainer 160 may process the set of embeddings along with the corresponding set of positional encodings according to the layers (e.g., attention layer, encoder / decoder attention layer, feedforward layer) in each decoder of the decoder block. The model trainer 160 may combine the output from the encoder with the input of the encoder / decoder attention layer in the decoder block. From the processing, the model trainer 160 may generate an output set of embeddings to forward to the output layer.

[0128] Subsequently, the model trainer 160 may send the output from the decoder block to the output layer of the generative transformer layer 165. In doing so, the model trainer 160 may process the embeddings from the decoder block according to the linear and activation layers of the output layer. Through processing, the model trainer 160 may calculate a probability for each embedding. The probability may represent the likelihood of the occurrence of the output given the input token. Based on the probability, the model trainer 160 may select the output token (e.g., at least a portion of the output text, image, audio, video, or multimedia content having the highest probability) and form, create, or generate a feedback output. The feedback output may include text content, image content, audio content, video, or multimedia content, among others, or any combination thereof.

[0129] Upon generation, the model trainer 160 can compare the output from the generative transformer model 165 to the feedback data 505 used to generate the feedback output. The comparison can be made between the probabilities (or distributions) of various tokens for the content from the output (e.g., words for the text output) and the probabilities of the tokens in the feedback data 505. As an example, the model trainer 160 can determine the difference between the probability distribution of the output and the feedback data 505 to compare. The probability distribution may identify the probability for each token of the candidate in the output or feedback data 505 tokens. Based on the comparison, the model trainer 160 can calculate, determine, or generate a loss metric. The loss metric may indicate the deviation of the output from the expected output as defined by the feedback data 505 used to generate the output. The loss metric may be calculated by following any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), quadratic loss, cross entropy error, and Huber loss, among others.

[0130] Using the loss metric, the model trainer 160 can update one or more weights in the series of layers of the generative transformer model 165. The weight updates may be according to backpropagation with one or more parameters (e.g., learning rate, momentum, weight decay, and number of iterations) and an optimization function (sometimes referred to herein as an objective function). The optimization function may define one or more parameters for which the weights of the generative transformer model 165 are updated. The model trainer 160 can iteratively train the generative transformer model 165 until convergence. Upon convergence, the model trainer 160 can save and maintain a set of weights for the series of layers of the generative transformer model 165 for use.

[0131] In this manner, the data processing system 105 may iteratively and continuously factor in responses 515 from the presentation of the content items 180 to the administrator 205 and to the end users 505 to improve the generation of messages that fit the particular domain 310. Additionally, the data processing system 105 may iteratively and continuously factor in responses 515' from the end users 505 to improve the generation of new effective content for subsequent messages in each domain 310. Compared to content generated by a general generative transformer model 165, the incorporation of feedback data 505 may enable the generation of new content items that are more targeted and relevant to the audience 215, domain 310, and end user 505 preferences. In the context of digital therapeutics, the new generation of content items may be factored in changes in usage, such as improvement or worsening of the end user's symptoms or progression of a treatment regimen. Through the use of the generative transformer model 165, content can be specifically generated to target end users in a flexible manner, expanding the personalization of content to multiple audiences. Enabling flexibility, scalability, and specificity can optimize or reduce the consumption of computing resources (e.g., processors and memory) and network bandwidth that would otherwise be wasted by providing ineffective content. From a human-computer interaction (HCI) perspective, content generated by utilizing the generative transformer model 165 can enhance the quality of an end user's interaction with the end user device 110. Additionally, increased engagement can result in an increased level of end user adherence to a treatment regimen. Increased adherence, in turn, increases the likelihood of preventing, alleviating, or treating the end user's symptoms.

[0132] Referring now to FIG. 6, a flow chart is shown of the architecture of a system 600 for generating content items in the system 100 for generating and regulating content for targeted messages. The system 600 may include or correspond to operations performed in the system 100 for generating and regulating content items 180 for targeted messages. The system 600 may include a moderator, accept input from an internal administrator, provide content items or pre-generated content from a model, and receive optional feedback from the administrator at the time of generation and from the end user at the time of presentation. Upon receiving the input, the moderator may forward the input to the guardrail. The input may include text data entered by the administrator and may create or search digital therapeutic content. The guardrail may assess the risk of the input using one or more natural language processing (NLP) models and risk models. If the input is determined to be high risk (e.g., a risk score above a threshold), the guardrail may return an indication that the input is high risk to the moderator for presentation to the administrator. On the other hand, if the input is determined to be low risk (e.g., a risk score below a threshold), the guardrail may pass the input to the generator.

[0133] Upon receiving the input, the generator may use NLP techniques (e.g., auto-summarization) to search for previously generated digital therapeutic content in a database. The database may store and maintain content across multiple domains, such as audience experience, regulatory, suitability, science or medicine, and products. Together, the generator may pass the input in the form of prompts to a series of generative models. Each generative model may be a generative transformer model and may use the prompts to generate novel digital therapeutic content. The generator may integrate the content items found in the database and output by the generative models and provide the content items to the guardrail. The guardrail may then evaluate each content item using the risk model. For each content item, the guardrail may determine a risk score indicating the degree of risk or suitability with a particular domain for the risk model. Using the risk score, the guardrail may select a subset of content items to provide to the moderator.

[0134] The moderator may use a ranking algorithm to rank the content items based on end user profile information and other information such as the domain, among other things. Based on the ranking, the moderator may select one or more content items to present to the end user. The content items may be presented to an administrator for entry of corrections. The content items may be provided to the end users to gauge audience response to the information contained therein. The moderator may receive feedback from the end users that identifies, among other things, audience responses or administrator corrections. The moderator may provide feedback to the trainer, who may then use the data in the feedback to improve the generative model as well as the content items stored and maintained in the database.

[0135] Referring now to FIG. 7, a method 700 for generating and regulating content for targeted messages is illustrated. Method 700 may be performed or implemented using any of the components detailed herein, such as data processing system 105, administrator device 190, end user device 110, and database 135, among others. Under method 700, a computing system (e.g., data processing system 105 or administrator device 190, or both) may receive a text input (e.g., input 225) (705). The computing system may determine a risk score (e.g., risk score 235) for the input (710). The computing system may determine whether the risk score meets a threshold (715). In response to determining that the risk score does not meet the threshold, the computing system may terminate processing (720). In response to determining that the risk score meets the threshold, the computing system may search for a content item (e.g., content item 180) for the text input (725). The computing system may determine (730) whether there is a content item in a database (e.g., database 135) that corresponds to the text input. In response to determining that there is no content item for the text input, the computing system may apply (735) the text input to a generative model (e.g., generative transformer model 165). In response to determining that there is one or more content items for the text input or in response to applying the text input to the generative model, the computing system may identify the content items (740). The computing system may determine (745) an output risk score (e.g., risk score 305). The computing system may determine (750) whether the risk score meets a threshold. If the risk score does not meet the threshold, the computing system may filter out the content item (755). In response to the risk score meeting the threshold, the computing system may select (760) a content item. The computing system may provide (765) the content item.

[0136] B. Systems and methods for training and applying validation models to content Referring now to FIG. 8, a block diagram of a process 800 for training and applying a validation model to content is shown. In overview, the system 800 may include at least one data validation system 805, an administrator device 810, model services 865A-865N (hereinafter generally referred to as model services 865), a database 820, and an end user device 825 communicatively connected to each other via at least one network 815. The model service 865 may include at least one generative model 855. The data validation system 805 may include, among other things, at least one model trainer 830, at least one model applier 835, at least one feedback handler 840, at least one policy enforcer 845, and at least one validation model 850. The data validation system 805 may include or have access to at least one database 820. The database 820 may, among other things, store, maintain, or include one or more content items 860A-N (hereinafter generally referred to as content items 860). Within the data validation system 805, a model trainer 830 may train a validation model 850. A model applier 835 may apply the validation model 850 and generate an output data set. A feedback handler 840 may receive feedback from the administrator device 810. A policy enforcer 845 may provide the content items to the end user. In some embodiments, the data validation system 805 may be part of a data processing system 105 as further described herein.

[0137] More specifically, the data validation system 805 may be any computing device including one or more processors coupled with memory and software capable of executing the various processes and tasks described herein. The data validation system 805 may communicate with end user devices 825, administrator devices 810, model services 865, and databases 820 via a network 815. The data validation system 805 may be located within, installed in, or associated with at least one computer system. The computer system may correspond to a data center, branch office, or site where one or more computers corresponding to the data validation system 805 are located.

[0138] The end user device 825 may be any computing device having one or more processors connected with memory and software capable of executing the various processes and tasks described herein. The end user device 825 may be similar to or include the end user device 110. The end user device 825 may communicate with the data verification system 805, the administrator device 810, the model service 865, and the database 820 via the network 815. The end user device 825 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), or laptop computer. The end user device 825 may be provided with one or more content items 860 via the data verification system 805. Alternatively, the end user device 825 may request one or more content items 860 via an interaction with the data verification system 805, such as via an application associated with the user device 825.

[0139] The administrator device 810 (sometimes referred to herein as the administrator device) may be any computing device having one or more processors connected with memory and software capable of executing various processes and tasks described herein. The administrator device 810 may be similar to or include the administrator device 190. The administrator device 810 may communicate with the data validation system 805, the end user devices 825, the model service 865, and the database 820 via a network 815. The administrator device 810 may be a smartphone, other mobile phone, a tablet computer, a wearable computing device (e.g., a smart watch, eyeglasses), or a laptop computer. The administrator device 810 may be used to access applications.

[0140] The database 820 may store and maintain various resources and data associated with the data validation system 805 and the model service 865. The database 820 may include a database management system (DBMS) to, among other things, arrange and organize the data maintained therein, such as the content items 860A-N. The database 820 may communicate with the data validation system 805, the administrator device 810, the end user devices 825, and the model service 865 via the network 815. While performing various operations, the data validation system 805, the model service 865, and the administrator device 810 may access the database 820 and read identified data therefrom. The data validation system 805, the model service 865, and the administrator device 810 may write data to the database 820 from the performance of such operations.

[0141] In database 820, content items 860 may be in any format, such as text, images, audio, video, or multimedia content, or any combination thereof, among others. Content items 860 may be similar to or include content items 180. Content items 860 may be stored and maintained in database 820 using one or more files. By way of example, in the case of text, content items 860 may be stored as text files (TXT), rich text files (RTF), extensible markup language (XML), and hypertext markup language (HTTP), among others. In the case of images, the content item 860 may be stored in the joint photographic experts' group (JPEG) format, portable network graphics (PNG) format, graphics interchange format (GIF), or scalable vector graphics (SVG) format, among others. In the case of audio, the content item 860 may be stored in the waveform audio file (WAV), motion pictures expert group format (e.g., MP3 and MP4), and Ogg Vorbis (OGG) format, among others. In the case of video, the content item 860 may be stored in the motion pictures expert group format (e.g., MP3 and MP4), QuickTime movie (MOV), and Windows Movie Video (WMV), among others.For multimedia content, the content item 860 can be audio video interleave (AVI), Motion Pictures Experts Group formats (e.g., MP3 and MP4), QuickTime Movie (MOV), and Windows Movie Video (WMV), among others.

[0142] Each content item 860 may identify or include information to be presented via the end user device 825 or administrator device 810. For example, the content item 860 may be presented to the end user or administrator through a message sent to the end user device 825 or administrator device 810, respectively. The message may be in any format, such as short message / messaging service (SMS), multimedia messaging service (MMS), among others, or as instructions for presentation via a display associated with the user device 825 or administrator device 810.

[0143] The message content item 860 may include a reminder to perform a session task. The message may be taken from a library of pre-generated psychotherapy messages or a library of pre-generated engagement (reminder) messages. The message may include a reminder for the end user to complete a therapy session, take a medication dose, or complete a regimen task. The message may include an activity for the end user to perform or a lesson in which the end user is engaged. The content item 860 may include a response mechanism, such as a link, chat box, or indicator to respond to the message.

[0144] The content items 860 may include or correspond to one or more pieces of text, such as articles, abstracts, or publications. For example, the content items 860 may include research papers, review articles, case reports, clinical trial protocols, or editorials, among others. The content items 860 may include texts for submission to government agencies, subject matter experts, scientific journals, or conferences, among others. For example, the content items 860 may include clinical trial protocols regarding treatments provided for end-user symptoms for submission to the Food and Drug Administration (FDA), medical journals, or for internal distribution.

[0145] End user conditions may include, for example, chronic pain (e.g., associated with or including arthritis, migraine, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive stress disorder, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin conditions (e.g., atopic dermatitis, psoriasis, self-inflicted dermatosis, and eczema), cognitive disorders (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis, and schizophrenia), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, emotionally unstable personality disorder, and attention deficit hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, tobacco use disorder, or hallucinogen disorder), and other conditions (e.g., narcolepsy and oncology, or cancer), among others.

[0146] While the content item 860 generated during the session of generating or modifying the content item 860 is provided, the end user may be, at least in part, simultaneously administered medication to address the symptoms. By way of example, if the medication is for pain, the end user may be administered acetaminophen, a nonsteroidal anti-inflammatory composition, an antidepressant, an anticonvulsant, or other composition, among others. In the case of skin lesions, the end user may be administered steroids, antihistamines, or topical antiseptics, among others. In the case of cognitive impairment, the end user may be administered cholinesterase inhibitors or memantine, among others. In the case of narcolepsy, the end user may be administered stimulants or antidepressants, among others. The end user may also participate in other psychological therapies for these symptoms.

[0147] The content items 860 can be human-created, computer-generated, or a combination thereof. In some embodiments, a human can provide the content items 860 through the administrator device 810. For example, the administrator device 810 can upload, provide, or transfer one or more content items 860 for storage in the database 820. The content items 860 can be computer-generated, such as by a generative model 855. In some embodiments, the administrator device 810 can provide input to a model service 865, which can use the generative model 855 to generate one or more content items 860. For example, the administrator device 810 can provide text, images, video, or other representations as input to generate the content items 860. The model service 865 can use the generative model 855 to generate one or more content items 860 from prompts generated by input from the administrator device 810.

[0148] The model service 865 may be or include any computing device capable of hosting one or more generative models 855 and generating content items 860. The model service 865 may be local to the system 800 or may be remote from the system 800 and accessed by the data validation system 805 over the network 815. In some embodiments, the data validation system 805 may maintain one or more model services 865. In some embodiments, the data validation system 805 may access a remote model service 865, provide input, and generate one or more content items 860. The model service 865 may maintain one or more generative models 855.

[0149] A generative model 855 (sometimes referred to herein as a generative transformer model 855) may receive input in the form of a series of strings (e.g., from a text input) and output content (e.g., content item 860) in one or more modalities (e.g., in the form of a text string, audio content, image, video, or multimedia content). The generative model 855 may be similar to or include a generative transformer model 165. The generative model 855 may be a machine learning model following a transformer model (e.g., a generative pre-trained model, or a bidirectional encoder representation from a transformer). The generative transformer model 165 may be a large language model (LLM), a text-to-image model, a text-to-speech model, or a text-to-video model, among others. In some embodiments, the generative model 855 may be part of a data validation system 805. Alternatively, the data validation system 805 may include a model service 865. In some embodiments, the generative model 855 may be part of a server separate from the data validation system 805 and communicate with the data validation system 805 via the network 815 .

[0150] One or more of the generative models 855 may be trained and maintained by the data processing system 105. The generative model 855 may include a set of weights arranged across a set of layers according to a transformer architecture. Under the architecture, the generative model 855 may include, among others, at least one tokenization layer (sometimes referred to herein as a tokenizer), at least one input embedding layer, at least one positional encoder, at least one encoder stack, at least one decoder stack, and at least one output layer interconnected with each other (e.g., via forward, reverse, or skip connections). In some embodiments, the generative transformer layer may lack an encoder stack (e.g., in the case of an encoder-only architecture) or a decoder stack (e.g., in the case of a decoder-only model architecture). The tokenization layer may convert raw input in the form of a set of strings into a corresponding set of word vectors (also referred to herein as tokens or vectors) in an n-dimensional feature space. The input embedding layer may use the set of word vectors to generate a set of embeddings. Each embedding can be a low-dimensional representation of a corresponding word vector and can capture semantic and syntactic information of the string associated with the word vector. A positional encoder can generate a positional encoding for each input embedding depending on the position of the corresponding word vector, or by extension, the position of the string within the input sequence of strings.

[0151] In the generative model 855, the encoder stack can include a series of encoders. Each encoder can include, among others, at least one attention layer and at least one feedforward layer. The attention layer (e.g., a multi-head self-attention layer) can calculate an attention score for each input embedding, indicating the degree of attention the embedding should pay attention to, and generate a weighted sum of the series of input embeddings. The feedforward layer can apply a linear transformation with a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the attention layer. The output can be fed to another encoder in the encoder stack in the generative transformer layer. When the encoder is a terminal encoder in the encoder stack, the output can be fed to the decoder stack.

[0152] The decoder stack may include, among others, at least one attention layer, at least one encoder / decoder attention layer, and at least one feedforward layer. In the decoder stack, the attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each output embedding (e.g., an embedding generated from a target output or an expected output). The encoder / decoder attention layer may combine inputs from an attention layer in the decoder stack with an output from one of the encoders in the encoder stack, and may calculate an attention score from the combined inputs. The feedforward layer may apply a linear transformation with a nonlinear activation (e.g., a rectified linear unit (ReLU)) to the output of the encoder / decoder attention layer. The output of the decoder may be sent to another decoder in the decoder stack. When the decoder is a terminal decoder in the decoder stack, the output may be sent to an output layer.

[0153] The output layer of the generative model 855 may include, among others, at least one linear layer and at least one activation layer. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to calculate the token scores. The activation layer may apply an activation function (e.g., softmax, sigmoid, or rectified linear unit) to the output of the linear function to convert the token scores into a probability (or distribution). The probability may represent the likelihood of the occurrence of the output token given the input token. The output layer may use the probability to select an output token (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability). This may be repeated over a series of input tokens, and the resulting series of output tokens may be used to form the output of the generative model 855 as a whole. Although described primarily in terms of Transformer models herein, the model service 865 may also use other machine learning models to generate and output content. In some implementations, the model service 865 may generate and output content using one or more models maintained by an external system. For example, the data processing system may generate the content using one or more models such as ChatGPT created by OpenAI, BARD created by Google, or LLaMA created by Meta, among others.

[0154] Each generative model 855 may create one or more of the content items 860 based on the prompts provided to the generative model 855. Each content item 860 created from prompts generated from text input provided to the data validation system 805 or model service 865 may be different due to differences between each of the generative models 855. As such, content item 860A may be more suitable for presentation to an end user or administrator than other content items 860B-N. For example, content item 860 generated by generative model 855 may include errors, irrelevant content, or hallucinations. In some embodiments, content item 860A generated by generative model 855 may not be relevant to a particular end user due to the information in content item 860A, the symptoms addressed by content item 860A, the presentation style of content item 860A, or the grand assertions provided by content item 860A. For example, content item 860A may assert that it is the "best" method of treatment for a given condition, but this cannot be asserted and erroneous information is provided. For example, content item 860A may encourage an end user to consume a meat-based diet without recognizing that the user has previously indicated a vegetarian diet. For example, content item 860A may be in a text presentation style, even though the end user's historical behavior since a previous session indicates that the end user is more consistent and faithful to the session when presented with video content. For example, content item 860A may generate data that has not been verified or proven to be true. To moderate the content item 860 created by generative model 855, data validation system 805 may train and apply one or more validation models 850 to content item 860 to determine the suitability of content item 860.

[0155] Each validation model 850 can be a machine learning model trained to determine whether a content item 860 meets or does not meet a set of criteria. The validation model 850 can be trained to calculate a risk score for the content item 860 corresponding to the match or non-match. Match can refer to or include a content item 860 that is below a threshold risk score and therefore may be served. Match can refer to or include a content item 860 that exceeds a threshold accuracy or meets a domain, audience, or combination thereof criteria as described herein. Thus, non-match can refer to or include a content item 860 that is above a threshold risk score and therefore may not be served by the data validation system 805. The determination of match or non-match for each content item 860 by the data validation system 805 can further be used to continually train the generative transformer model 165 to provide more accurate or less risky content items 860 over time. The validation model 850 can include one or more natural language models, including the generative transformer model 165 or generative model 855 described herein. The validation model 850 may include one or more classifier models such as a Naive Bayes Classifier, a support vector machine (SVM) ensemble classifier, a kernel approximation, a K-nearest neighbor classifier, or a decision tree, among others.

[0156] One or more of the validation models 850 can accept a content item 860 as an input. By accepting a content item 860 as an input, the one or more validation models 850 can generate a risk score corresponding to a degree of fit. Or, the one or more validation models 850 can generate an indication of fit or non-fit. The degree of fit or the indication of fit can be associated with the likelihood that the content item 860 is a desired content item 860. The desired content item 860 can include a content item 860 in a specified format, for a specified group or audience of people, for a domain, with a desired accuracy (e.g., accurate information, relevant data set), or with a desired relevance (e.g., a user receiving the content item 860 as part of a digital therapeutic session, or an administrator receiving a desired article type of text), among others.

[0157] 9, a block diagram is shown of a process 900 for training a validation model using a training dataset in a system 800 for training and applying a validation model to content. The process 900 may include or correspond to operations performed in the system 800 for training and applying a validation model to content. Under the process 800, a model trainer 830 may train a validation model 850 using a training dataset 905. A model applier 835 may apply content items of the training dataset 905, determine a loss 930 for the validation model 850, and iteratively train the validation model 850.

[0158] The model trainer 830 running in the data validation system 805 can retrieve, obtain, or receive a training dataset 905 from the database 820. The training dataset 905 can include a set of examples for use in training the validation model 850. The training dataset can include content items 910. The content items 910 can be similar to the content items 860 or the content items 180. The content items 910 can be example content items entered by the administrator device 810 that trains the validation model 850. For example, the content items 910 can include textual content, visual content, tactile content, auditory content, or a combination thereof to provide to a user device.

[0159] The content items 910 may be associated with an index 915. The index 915 may indicate the suitability or non-suitability of the content items 910 according to a set of criteria. The content items 910 may correspond to an index 915, such as described herein, that allows the index 915 to define whether the content items 910 are suitable or non-suitable. In some embodiments, the administrator device 810 may assign an index 915 to the content items 910 in the training data set 905. In this manner, the content items 910 may be annotated with the index 915 as suitable or non-suitable.

[0160] In some embodiments, the indicator 915 may correspond to or include a risk score, such as the risk score 305 described herein. The risk score may identify a degree of conformance or non-conformance with a set of criteria for delivery. For example, the risk score may identify a level, number, or score that corresponds to how desirable (e.g., how accurate, risky, relevant, etc.) the content item 1010 is for delivery to a user associated with the end-user device 110. In some embodiments, the administrator device 810 may assign a risk score as part of assigning the indicator 915.

[0161] In some cases, the content item 910 may include an identification of a domain and criteria corresponding to that domain. An index 915 associated with a content item 910 in the training dataset 905 may include a domain identifier, such as domain identifier 220, to identify a domain, such as domain 310, associated with the content item 910. The content item 910 may be associated with, correspond to, or be identified by a domain, such as the domains described herein. For example, the content item 910 may be associated with an index 915 that identifies a product domain, an audience experience domain, a medical domain, a scientific domain, or a regulatory domain, among others.

[0162] When training the model, the model trainer 830 may access the database 820 and retrieve, obtain, or identify a training dataset 905 used to train the validation model 850. In some embodiments, the model trainer 830 may retrieve or identify an example of the training dataset 905 including content items 910 stored in the database 820. In some embodiments, the model trainer 830 may access the database 820 and retrieve or identify a set of responses by end users (or feedback data obtained therefrom) to previously provided messages or content items. Each response may define or identify an indicator 915 as the performance of an activity by the end user in response to the presentation of the corresponding message. By way of example, the indicator 915 may include a response that may include, among other things, whether the end user performed a particular activity, an indicator of a favorable reaction to the message, one or more interactions in response to the presentation of the message, and a timestamp identifying the performance of the response.

[0163] In some embodiments, the model trainer 830 can initialize the validation model 850. For example, the model trainer 830 can instantiate the validation model 850 by assigning random values ​​to the validation model's 850 weights in the validation model's 850 layers. In some embodiments, the model trainer 830 can fine-tune a pre-trained machine learning model using a training dataset 905. To train or fine-tune, the model trainer 830 can define, select, or identify a training dataset 905 from the database 820. The training dataset 905 can be used to input the validation model 850 and create an output dataset 925. The output dataset 925 can include indices 915' that are compared to the indices 915. The content items 910 can at least partially overlap with and correspond to a subset of the text strings in the training dataset 905. For example, when the content items 910 include text from a message about a particular symptom, the training dataset 905 may correspond to a text description of the symptom and the output dataset 925 may correspond to text strings of an activity to perform, a psychoeducational lesson to engage in, a reminder to take a medication dose, or a notification to perform a particular activity, among others. The training dataset 905 corresponding to the input and the output dataset 925 may have little overlap. As an example, when the training dataset 905 includes connections between text and images, the content items 910 used as input may correspond to the text and the output dataset 925 used as destination may correspond to the images associated with the text.

[0164] The model applier 835 may apply the content item 910 to the validation model 850. For example, the model applier 835 may send or apply the content item 910 to the validation model 850. During application, the model applier 835 may process the content item 910 and generate an output data set 925 that includes indices 915'. In some embodiments, the model applier 835 may select a validation model 850 and apply the content item 910 based on a domain associated with the content item 910. For example, the model applier 835 may apply a content item associated with the medical domain to a first validation model and a content item associated with the regulatory domain to a second validation model. In this manner, a set of validation models 850 may be trained for a particular domain.

[0165] The validation model 850 may output an output data set 925 that includes indicators 915'. In some embodiments, the model applier 835 can determine whether the indicators 915' are either a match or a mismatch based on the output data set 925. The indicators 915' may be generated by the validation model 850 based on the content item 910. In some embodiments, the model trainer 830 can determine the indicators 915' with respect to a domain indicated in the content item 915. For example, the indicators 915' may correspond to a match within a particular domain. The indicators 915' may indicate whether the content item 910 corresponds to a domain that the validation model 850 supports.

[0166] In some embodiments, the output data set 925 can include portions 920A-N (hereinafter generally referred to as portions 920). The portions 920 can include subsections of the content item 910 to be modified or identified as subsections to be modified. The portions 920 can include indicators of the subsections to be modified. The validation model 850 can determine the portions 920 from the output data set 925 in response to a determination that the indicators 915' correspond to a non-conformance or fall below a degree of conformance. The portions 920 can indicate sections of text, images, words, wording, or other components of the content item 910 that are desirable to edit. In some embodiments, the administrator device 810 can edit the portions 920 of the output data set 925. In some embodiments, the administrator device 810 can provide the portions 920 in the output data set 925 during generation of the output data set 925. In this manner, the validation model 850 can accept edits to the portions 920, or selection of the portions 920 themselves, in further iterative training of the validation model 850.

[0167] Upon determining the index 915', the model trainer 830 can compare the index 915' to the index 915. The comparison can be made between the probabilities (or distributions) of various tokens for the content item 910 (e.g., words for text output) from the output dataset 925 and the probabilities of the tokens in the content item 910. As an example, the model trainer 830 can determine the difference between the probability distribution of the index 915' and the compared index 915. Based on the comparison, the model trainer 830 can calculate, determine, or generate a loss 930. The loss 930 may indicate the deviation of the index 915' from the index 915. The loss 930 may be calculated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error (MSE), quadratic loss, cross entropy error, and Huber loss, among others.

[0168] In some embodiments, the model trainer 830 may determine a loss 930 for the output dataset 925 based on the data retrieved from the database 820. In doing so, the model trainer 830 may compare the output dataset 925 to the content item 910 and calculate a similarity. The similarity may measure, correspond to, or be indicative of, for example, a level of semantic similarity (e.g., using a knowledge map when comparing between the text of the content item 910 and the output dataset 925), visual similarity (e.g., pixel-to-pixel value comparison when comparing between an image or frame of a video of the content item 910 and the output dataset 925), or phonetic similarity (e.g., using a correlation or cosine similarity measure between the audio of the content item 910 and the output dataset 925). The loss 930 may be a function of, among other things, the similarity, the domain, or a response that indicates whether the user responded to the content item 910 to which the output dataset 925 is being compared. Generally, the greater the loss 930, the more likely the generated output data set 925 will deviate from the preferences established by a given administrator device 810 in devising the indicators 915. Conversely, the smaller the loss 930, the less likely the generated output data set 925 will deviate from and be consistent with the indicators 915 established by the administrator device 810. The loss 930 may be calculated to train the validation model 850 and generate a risk score or indicator of suitability for the indicators 915, particularly for messages with a high probability of engagement by users, and for content items corresponding to a particular domain or audience.

[0169] Using the loss 930, the model trainer 830 can update one or more weights in the series of layers of the validation model 850. The weight updates can be according to backpropagation and an optimization function (sometimes referred to herein as an objective function) with one or more parameters (e.g., learning rate, momentum, weight decay, and number of iterations). The optimization function can define one or more parameters for which the weights of the validation model 850 are updated. The optimization function can be according to a stochastic gradient descent method and can include, for example, adaptive moment estimation (Adam), implicit update (ISGD), and adaptive gradient algorithm (AdaGrad), among others. The model trainer 830 and the model applier 835 can iteratively train and apply the validation model 850 until convergence. Upon convergence, the model trainer 830 can save and maintain a set of weights for the series of layers of the validation model 850.

[0170] Referring now to FIG. 10, a block diagram of a process 1000 for capturing feedback for a validation model in a system for training and applying a validation model to content is shown. The process 1000 may include or correspond to operations performed in the system for training and applying a validation model to content 800. Under the process 1000, a model applier 835 may apply content items 1010A-N (hereinafter generally referred to as content items 1010) to a trained validation model 850. The validation model 850 may generate output data sets 1025A-N (hereinafter generally referred to as output data sets 1025). A feedback handler 840 may present the output data sets 1025 to the administrator device 810 and may receive feedback data 1035 from the administrator device 810.

[0171] Upon convergence of the loss 930 to the threshold level, the model applier 835 may identify content items 1010 from the database 820 or the model service 865. The content items 1010 may be previously generated content items stored in the database 820. The content items 1010 may include content items generated by one or more of the model services 865. The model applier 835 may provide the identified content items 1010 as input to the validation model 850. In some embodiments, the model applier 835 may provide the content items 1010 based on the domain corresponding to the content items 1010 or the domain of each content item 1010. For example, a first content item 1010A may correspond to a product domain and a second content item 1010B may correspond to an audience experience domain. The model applier 835 may select a first validation model corresponding to a product domain to apply to the first content item 1010A, and a second validation model corresponding to an audience experience domain to apply to the second content item 1010B.

[0172] By applying the validation model 850, the model applier 835 can generate at least one output data set 1025. The output data set 1025 can be similar to the output data set 925. The output data set 1025 can indicate whether a content item of the content items 1010 is suitable for serving. The output data set 1025 can include an indicator 1015. The indicator 1015 can indicate whether a particular content item of the content items 1010 is suitable or unsuitable, as described herein. When the content item 1010 is identified as suitable, the content item 1010 can be allowed to be served (e.g., by the data validation system 805). On the other hand, when the content item 1010 is identified as unsuitable, the content item 1010 can be restricted (e.g., by the data validation system 805).

[0173] The validation model 850 can generate an indicator 1015 from the content item 1010 applied by the model applier 835. In some embodiments, the validation model 850 may generate an indicator of either fit or non-fit for a domain identified in the content item 1010. For example, the validation model 850 may determine that a first content item 1010A is not fit (e.g., the indicator 1015 indicates non-fit) for a particular domain, such as a medical or scientific domain, a regulatory domain, a product domain, or an audience experience domain, among others.

[0174] The output data set 1025 may include portions 1020A-N (hereinafter generally referred to as portions 1020). The portions 1020 may be similar to or may include portions 920. The portions 1020 may represent subsections of the content item 1010 that are to be modified or that are not conforming. In some embodiments, the portions 1020 may represent subsections of the content item 1010 that may be edited to make the content item 1010 conforming.

[0175] The feedback handler 840 may generate instructions to display the output data set 1025 on the administrator device 810 via a user interface 1030 of the administrator device 810. The user interface 1030 may be similar to or include the user interface 125 and may include UI elements such as UI elements 130. The user interface 1030 may be any input / output device that displays the output data set 1025 and accepts interaction from the administrator device 810 regarding the output data set 1025. The feedback handler 840 may generate instructions to display, provide, or present the output data set 1025 to the administrator device 810 via the user interface 1030.

[0176] The feedback handler 840 may display the output data set 1025 including the indicators 1015. In some embodiments, the feedback handler 840 may display the indicators 1015 corresponding to each content item 1010. For example, the feedback handler 840 may generate instructions for the user interface 1030 to display each content item 1010 corresponding to an indicator of suitability, non-suitability, or a risk score indicating the degree of suitability. The feedback handler 840 may generate instructions to display the portions 1020. In some embodiments, the feedback handler 840 may generate instructions to display each content item of the content items 1010 corresponding to the respective portions 1020. The feedback handler 840 may generate instructions to display the portions 1020 in response to a determination that the respective indicators 1015 indicate the respective content item 1010 as non-suitable.

[0177] Once the output data set 1025 is displayed, the feedback handler 840 may receive an interaction via the user interface 1030. The interaction may be similar to the interaction 210. The interaction may include actuation of one or more UI elements of the user interface 1030. The interaction may indicate one or more of the portions 1020 or the indicators 1015. The administrator device 810 may select one or more of the portions 1020 from display in the user interface 1030. The interaction may indicate one or more of the portions 1020 for editing. For example, the interaction may indicate the portion 1020A of the content item 1010. Or, the interaction may release or hide the portion 1020B determined by the validation model 850.

[0178] The interaction may include editing one or more portions, such as editing the text, display, image, audio, or other such characteristics of the content item 1010. For example, the interaction may be to change the volume, tone, or duration of the portion 1020A of the audio content item 1010. For example, the interaction may be to change the color, size, image, or duration of the display of a video or image content item 1010B. For example, the interaction may be to change strings, words, formatting, style, or font of a textual content item 1010C.

[0179] The interaction may include a modification to the indicator 1015. The administrator device 810 may change, modify, or override the indicator 1015 in the output data set 1025. In some embodiments, the administrator device 810 may determine that the indicator 1015 assigned by the validation model 850 is incorrect, inappropriate, or an incorrect value (in the case of a risk score indicating a degree of conformance). The administrator device 810 may provide a different indicator via an interaction that identifies the content item 1010 as conforming or non-conforming. For example, the validation model 850 may determine that the content item 1010 is conforming. The administrator device 810 may review the indicator 1025 and determine that the content item 1010 is in fact non-conforming. The administrator may override the indicator 1015 with a different indicator via a user interface 1030 associated with the administrator device 810 and determine the indicator 1015.

[0180] The administrator device 810 may send the interaction to the feedback handler 840. The feedback handler 840 may generate feedback data 1035 from the interaction. The feedback data 1035 may include information about the portion 1020. For example, the feedback data 1035 may include a portion selected by the administrator device 810, a portion edited by the administrator device 810, or edits to the portion 1020 by the administrator device 810. The feedback data 1035 may include information about an indicator 1015 that was overridden by the administrator device 810. For example, the feedback data 1035 may include a confirmation of the indicator 1015 generated by the validation model 850 for the content item 1010, a change to the indicator 1015 generated by the validation model 850 for the content item 1010, an assignment of an indicator 1015 or risk score to the content item 1010, or a selection of a domain or domains for the content item 1010.

[0181] The feedback handler 840 can provide feedback data 1035 to the validation model 850 to retrain the validation model 850. In some embodiments, the validation model 850 can be continually retrained by the model applier 835 and the model trainer 830 using the feedback data 1035 to further reduce the loss 930. The feedback data 1035 can be used to generate a new training data set that includes, among other things, one or more of the metrics from the administrator device 810, the portions from the administrator device 810, edits to the portions by the administrator device 810, and the domains provided by the administrator device 810. In this way, the validation model 850 can continually learn and improve across a variety of content items and domains.

[0182] The feedback handler 840 can determine a connection between the content item 1010 and the index 1015. The feedback handler 840 can store the connection in the database 820. The connection between the content item 1010 and the index 1015 can define the content item 1010A as matching and the content item 1010B as non-matching. The matching content item 1010A can be allowed to be provided to the user via the user device. The non-matching content item 1010B can be restricted from being provided to the user. In some embodiments, at least one of the connections can include a connection between another modality of text and the content item 1010, such as an image, audio, video, or multimedia content, among others. The connection between the other modality of text and the content item 1010 can be from a generalized source. For example, the generalized source connection can be obtained from a large, pre-defined corpus that identifies connections between words and images. The connections between the other modalities of text and the content items 1010 can be from a particular source in the knowledge domain. By way of example, the connections can be taken from clinical studies, medical journals, or web pages with other modalities of text and content. Each content item in the content items 1010 can be stored with its connections in the database 820 as part of the dataset.

[0183] 11, a block diagram of a process 1100 for providing content items to an administrator in a system for training and applying a validation model to content is shown. The process 1100 may include or correspond to operations performed in the system for training and applying a validation model to content 800. Under the process 1100, the policy enforcer 845 may select the content items 1010 to provide to the administrator device 810. The administrator device 810 may present the content items 1010 on a user interface 1115 in a digital therapeutic session to address the end user's symptoms.

[0184] Upon storing the connections between the content items 1010 and the indicators 1015, a policy enforcer 845 operating in the data validation system 805 may select one or more content items 1010 to provide to the administrator 1120. The policy enforcer 845 may select one or more content items 1010 based on a profile associated with the administrator 1120. In some embodiments, the policy enforcer 845 may select one or more content items 1010 based on preferences indicated by the profile, an audience indicated by the administrator 1120, among other things. The policy enforcer 845 may select the content items 1010 that correspond to the indicators 1015 that indicate suitability of the content items 1010. For example, the policy enforcer 845 may select one or more content items 1010 for provision that indicate suitability or that indicate a degree of suitability above a threshold degree of suitability. The policy enforcer 845 may restrict provision of the content item 1010 if the connections include an indicator 1015 of non-suitability. Conversely, the policy enforcer 845 may allow serving of the content item 1010 if the connection includes an indication 1015 of relevance.

[0185] Upon selecting the content item 1010 based at least on the indicator 1015, the policy enforcer 845 may provide the content item 1010 to the administrator device 810. The administrator device 810 may include an application 1110. The application 1110 operating on the administrator device 810 may be or may include a test digital therapeutic application that reviews the content item to be provided to the end user in conjunction with a regimen to address the end user's symptoms. The application 1110 operating on the administrator device 810 may generate instructions to display the content item 1010 in a user interface 1115 generated by the application 1110. The user interface 1115 may be similar to or include the user interfaces 1030, 125, including UI elements and accepting interaction from an administrator 1120 of the device that presents the user interface.

[0186] Using the validation model 850, the data validation system 805 may control the provision of content items, either generated by the generative transformer model or manually created by a human. This may automate the approval or restriction of content items, thereby reducing the amount of manual effort of reviewing content for violations of suitability. The provision of content items may extend the functionality provided to end users, thereby improving the utility of end user devices that provide digital therapeutics and reducing the wasteful consumption of computing resources (e.g., processing power and memory) that would otherwise be spent on ineffective content. In the context of digital therapeutics, the control of content items may prevent inappropriate or inappropriate information from being provided to end users, thereby improving the effectiveness of the digital therapeutic content. Furthermore, increased engagement may result in a higher level of adherence of users to the therapeutic regimen. Increased adherence, in turn, increases the likelihood of preventing, alleviating, or treating symptoms of the end user.

[0187] 12A-12C, example user interfaces of a system for training and applying a validation model to content are shown. The example user interfaces may be implemented or performed using any of the components detailed herein, such as data validation system 805, administrator device 810, and database 820, among others. Referring now to FIG. 12A, user interfaces 1200 and 1255 are shown presenting an overview of content item A to an administrator. User interface 1200 may be presented in conjunction with the presentation of a message including the content item as described herein. User interface 1200 may include information 1205, as well as UI elements 1210 and 1215. Information 1205 may include information about the generated content item that was input to one or more validation models as described herein. Information 1205 may include an associated risk score per domain of the content item, an audience for the content item, the content type of the content item, or the source of the model item (such as which generative model or which database the content item originates from). User interface 1200 includes UI element 1210, which allows viewing of all generated content. UI element 1210, when interacted with by an administrator, may cause user interface 1200 to change to display a list of all content items generated from a prompt or retrieved from a database. UI element 1215, when interacted with by an administrator, may show content similar to content item A. For example, interaction with UI element 1215 may cause user interface 1200 to display the content items that were used to train the generative model that generated content item A.

[0188] As an alternative embodiment of the user interface 1200, the user interface 1255 may be presented in conjunction with the presentation of a message including a content item as described herein. The user interface 1255 may include information 1260 and UI elements 1265 and 1270. The information 1260 may include information about the generated content item that was input to one or more validation models as described herein. The information 1260 may include an indication of relevant fit per domain of the content item, an audience for the content item, a content type of the content item, or a source of the model item (such as which generative model or which database the content item originates from). The user interface 1255 includes a UI element 1265 to allow viewing of all generated content. The UI element 1265, when interacted with by the administrator, may cause the user interface 1255 to change to display a list of all content items generated from the prompt or retrieved from the database. The UI element 1270, when interacted with by the administrator, may show content similar to content item A. For example, interaction with UI element 1270 may cause user interface 1255 to display the content items that were used to train the generative model that generated content item A.

[0189] 12B, a series of user interfaces 1220 and 1275 are shown. The user interface 1220 may indicate a possible risk score for each domain associated with the content item. For example, a possible risk score for a particular content item through modifications associated with the medical domain is 3. The user interface 1220 may include a UI button 1255. Upon interaction with the UI button 1255 by the administrator, the user interface 1220 may change to display a user interface 1225 as shown in FIG. 12C. As an alternative embodiment of the user interface 1220, the user interface 1275 may indicate an indication of possible suitability for each domain associated with the content item. For example, an indication of possible suitability through modifications associated with the medical domain is suitability. The user interface 1275 may include a UI button 1280. Upon interaction with the UI button 1280 by the administrator, the user interface 1275 may change to display a user interface 1225 as shown in FIG. 12C.

[0190] 12C, a user interface 1225 is shown. The user interface 1225 may display suggested modifications and obtain a possible risk score as shown in the user interface 1220, or a possible suitability indicator as shown in the user interface 1275. The user interface 1225 may display a content item 1230. The user interface 1225 may display a portion 1235 of the content item 1230 to modify. The user interface 1225 may include, among other things, a UI button 1245 and a UI button 1250. The user interface 1225 may display an alert 1240 regarding the content item 1230. The alert 1240 may indicate that the portion 1235 highlighted by the alert 1240 may not comply with one or more domains. The portion 1235 may display suggested edits or modifications to the content item 1230. The UI element 1245 may be for accepting the modifications suggested by the portion 1235. UI button 1250 allows the administrator to edit the modifications shown in portion 1235 .

[0191] Referring now to FIG. 13, a method 1300 for training and applying a validation model to content is illustrated. The method 1300 may be executed or performed using any of the components detailed herein, such as the data validation system 805, the administrator device 810, and the database 820, among others. Under the method 1300, a computing system (e.g., the data validation system 805 or the administrator device 810, or both) may identify (1305) a training data set (e.g., the training data set 905). The computing system may apply (1310) the content (e.g., the content item 910) to the model (e.g., the validation model 850). The computing system may determine (1315) an index (e.g., the index 915′). The computing system may determine a loss metric (e.g., the loss 930). The computing system may determine (1325) whether an input has been received. In response to receiving the input, the computing system may generate 1330 feedback (e.g., feedback data 1035). In response to not receiving the input, the computing system may determine 1335 a loss metric. The computing system may update 1340 the model weights.

[0192] Referring now to FIG. 14, a method 1400 for providing a verified content item is illustrated. The method 1400 may be executed or implemented using any of the components detailed herein, such as the data validation system 805, the administrator device 810, and the database 820, among others. Under the method 1400, a computing system (e.g., the data validation system 805 or the administrator device 810, or both) may identify a content item (e.g., the content item 1010) (1405). The computing system may apply the content to a model (1410). The computing system may determine an indicator (1415). The computing system may store the indicator (1420). The computing system may determine whether the indicator indicates suitability of the content (1425). In response to determining that the indicator does not indicate suitability, the computing system may restrict the provision (1430). In response to determining that the indicator indicates suitability, the computing system may allow the provision (1435). The computing system may provide 1440 the content item.

[0193] C. Network and Computing Environment Various operations described herein may be performed in a computer system. FIG. 15 shows a simplified block diagram of a representative server system 1500, a client computer system 1514, and a network 1526 that may be used to perform certain embodiments of the present disclosure. In various embodiments, the server system 1500 or a similar system may perform the services or servers described herein or portions thereof. The client computer system 1514 or a similar system may perform the clients described herein. The system 1500 described herein may be similar to the server system 1500. The server system 1500 may have a modular design incorporating multiple modules 1502 (e.g., blades in a blade server embodiment). Two modules 1502 are shown, but any number may be provided. Each module 1502 may include a processing unit 1504 and a local storage device 1506.

[0194] The processing unit 1504 may comprise a single processor having one or more cores, or multiple processors. In some embodiments, the processing unit 1504 may comprise a general-purpose primary processor as well as one or more special-purpose co-processors, such as a graphics processor, digital signal processor, etc. In some embodiments, some or all of the processing unit 1504 may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such integrated circuits execute instructions stored in the circuitry itself. In other embodiments, the processing unit 1504 may execute instructions stored in a local memory device 1506. Any type of processor in any combination may be included in the processing unit 1504.

[0195] The local storage 1506 may comprise a volatile storage medium (e.g., DRAM, SRAM, SDRAM, etc.) and / or a non-volatile storage medium (e.g., magnetic or optical disks, flash memory, etc.). The storage medium incorporated in the local storage 1506 may be fixed, removable, or upgradeable, as desired. The local storage 1506 may be physically or logically divided into various sub-units, such as a system memory, a read-only memory (ROM), and a permanent storage device. The system memory may be a volatile read / write memory, such as a read / write memory device or a dynamic random access memory. The system memory may store some or all of the instructions and data that the processing unit 1504 needs at run time. The ROM may store static data and instructions needed by the processing unit 1504. The permanent storage device may be a non-volatile read / write memory device that may store instructions and data even when the module 1502 is powered off. As used herein, the term "storage medium" includes any medium on which data can be stored indefinitely (even after being overwritten, subjected to electrical disturbances, power outages, etc.) and which does not include carrier waves and transitory electronic signals propagated via wireless or wired connections.

[0196] In some embodiments, the local storage device 1506 may store one or more software programs executed by the processing unit 1504, such as an operating system and / or programs that perform various server functions, such as the functions of system 100, 800, or any other system described herein, or any other server functions associated with system 100, 800, or any other system described herein.

[0197] "Software" generally refers to a sequence of instructions that, when executed by the processing unit 1504, cause the server system 1500 (or portions thereof) to perform various operations, thereby defining one or more specific machine embodiments that execute and implement the operations of the software program. The instructions may be stored as firmware residing in a read-only memory and / or as program code stored on a non-volatile storage medium that may be loaded into a volatile working memory for execution by the processing unit 1504. The software may be executed as a single program or a collection of separate interacting programs or program modules, as desired. From the local storage unit 1506 (or non-local storage units as described below), the processing unit 1504 may retrieve program instructions to execute and data to process to perform the various operations described above.

[0198] In some server systems 1500, multiple modules 1502 may be interconnected via a bus or other interconnect 1508 to form a local area network that supports communication between the modules 1502 and other components of the server system 1500. The interconnect 1508 may be implemented using a variety of technologies, including server racks, hubs, routers, etc.

[0199] A wide area network (WAN) interface 1510 may provide data communication capability between a local area network (e.g., through interconnect 1508) and a network 1526, such as the Internet. Other technologies, including wired (e.g., Ethernet, IEEE 802.3 standard) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standard), may also be used to communicatively connect the server system to the network 1526.

[0200] In some embodiments, the local storage device 1506 is intended to provide working memory for the processing device 1504, providing fast access to programs and / or data being processed while reducing traffic on the interconnect 1508. Storage of large amounts of data may be provided to the local area network by one or more mass storage subsystems 1512, which may be connected to the interconnect 1508. The mass storage subsystems 1512 may be based on magnetic, optical, semiconductor, or other data storage media. Direct attached storage devices, storage area networks, network attached storage devices, etc. may be used. Any data stores or other collections of data described herein, such as those created, consumed, or maintained by the services or servers, may be stored in the mass storage subsystem 1512. In some embodiments, additional data storage resources may be accessible via the WAN interface 1510 (potentially increasing latency).

[0201] The server system 1500 may operate in response to requests received via the WAN interface 1510. For example, one of the modules 1502 may perform supervisory functions in response to received requests and assign separate tasks to the other modules 1502. Work distribution techniques may be used. As requests are processed, results may be returned to the requester via the WAN interface 1510. Such operations may typically be automated. Additionally, in some embodiments, the WAN interface 1510 may connect multiple server systems 1500 together to provide a scalable system capable of managing large volumes of activity. Other techniques for managing server systems and server farms (groups of cooperating server systems) may also be used, including dynamic resource allocation and reallocation.

[0202] The server system 1500 can interact with various user-owned or user-operated devices over a wide area network such as the Internet. An example of a user-operated device is shown in FIG. 15 as a client computing system 1514. The client computing system 1514 can be implemented as a consumer device, such as, for example, a smart phone, other mobile phone, a tablet computer, a wearable computing device (e.g., smart watch, eyeglasses), a desktop computer, a laptop computer, etc.

[0203] For example, client computing system 1514 can communicate via WAN interface 1510. Client computing system 1514 can include computer components such as a processing unit 1516, a storage device 1518, a network interface 1520, a user input device 1522, and a user output device 1524. Client computing system 1514 can be computing devices implemented in a variety of form factors, such as desktop computers, laptop computers, tablet computers, smartphones, other mobile computing devices, wearable computing devices, and the like.

[0204] The processing unit 1516 and storage device 1518 may be similar to the processing unit 1504 and local storage device 1506 described above. A suitable device may be selected based on the requests made to the client computing system 1514. For example, the client computing system 1514 may be implemented as a "thin" client with limited processing capabilities or as a high-powered computing device. The client computing system 1514 may comprise program code executable by the processing unit 1516 to enable various interactions with the server system 1500.

[0205] The network interface 1520 may provide a connection to a network 1526, such as a wide area network (e.g., the Internet), to which the WAN interface 1510 of the server system 1500 is also connected. In various embodiments, the network interface 1520 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0206] User input device 1522 may include any device (or devices) through which a user may provide a signal to client computing system 1514. Client computing system 1514 may interpret the signal as indicating a user request or information. In various embodiments, user input device 1522 may include at least one of a keyboard, a touchpad, a touchscreen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, and the like.

[0207] User output devices 1524 include any device through which client computing system 1514 can provide information to a user. For example, user output devices 1524 can include images between displays generated by or delivered to client computing system 1514. Displays can incorporate a variety of image generating technologies, such as liquid crystal display (LCD), light-emitting diode (LED) displays including organic light-emitting diodes (OLED), projection systems, cathode ray tubes (CRT), and the like, along with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, and the like). Some embodiments can include devices such as touch screens that function as both input and output devices. In some embodiments, other user output devices 1524 can be provided in addition to or instead of displays. Examples include indicator lights, speakers, tactile "display" devices, printers, and the like.

[0208] Some embodiments include electronic components, such as a microprocessor, storage devices, and memory, that store computer program instructions on a computer-readable storage medium. Many of the features described herein may be implemented as a process specified as a series of program instructions encoded on a computer-readable storage medium. When executed by one or more processing devices, the program instructions cause the processing device to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as produced by a compiler, and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter. Through suitable programming, the processing devices 1504 and 1516 can provide various functions for the server system 1500 and the client computing system 1514, including any of the functions described herein as performed by a server or client, or other functions.

[0209] It will be understood that the server system 1500 and the client computing system 1514 are exemplary and that variations and modifications are possible. Computer systems used in connection with the embodiments of the present disclosure may have other capabilities not specifically described herein. Additionally, while the server system 1500 and the client computing system 1514 have been described in connection with specific blocks, it will be understood that these blocks are defined for convenience of explanation and are not intended to imply a specific physical arrangement of components. By way of example, the various blocks can, but need not, be present in the same facility, in the same server rack, or on the same motherboard. Additionally, the blocks need not correspond to physically distinct components. The blocks can be configured to perform various operations, for example, by programming a processor or providing appropriate control circuitry. Additionally, the various blocks may or may not be reconfigurable, depending on how the initial configuration is obtained. The embodiments of the present disclosure may be realized in a variety of apparatuses, including electronic devices implemented using any combination of circuitry and software.

[0210] Although the present disclosure has been described with respect to specific embodiments, those skilled in the art will recognize that numerous modifications are possible. The embodiments of the present disclosure may be implemented using a variety of computer systems and communication technologies, including, but not limited to, the specific examples described herein. The embodiments of the present disclosure may be implemented using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein may be executed on the same processor or any combination of different processors. Where a component is described as being configured to perform a particular operation, such configuration may be accomplished, for example, by designing an electronic circuit that performs the operation, by programming a programmable electronic circuit (such as a microprocessor) that performs the operation, or any combination thereof. Furthermore, while the above-described embodiments refer to specific hardware and software components, those skilled in the art will recognize that various combinations of hardware and / or software components may also be used, and that a particular operation described as being performed in hardware may also be performed in software, or vice versa.

[0211] A computer program incorporating various features of the present disclosure may be encoded and stored on various computer-readable storage media. Suitable media include magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memories, and other non-transitory media. A computer-readable medium encoded with the program code may be packaged with a compatible electronic device. Alternatively, the program code may be provided separately from the electronic device (e.g., via internet download or as a separately packaged computer-readable storage medium).

[0212] Therefore, although the disclosure has been described in terms of specific embodiments, it will be understood that the disclosure is intended to cover all modifications and equivalents that fall within the scope of the following claims.

Claims

1. 1. A method of regulating digital therapeutic content for distribution, comprising: Identifying, by one or more processors, a first digital therapeutic content provided over a network; applying, by the one or more processors, the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output, the ML model comprising: identifying a training data set including a plurality of examples, each example of the plurality of examples identifying a respective second digital therapeutic content and a first indicator identifying either a match or a match for delivery; applying the second digital therapeutic content from one of the examples of the training dataset to the ML model to generate a second output; determining from the second output a second indication of either suitability or non-suitability of the second digital therapeutic content used to control delivery; comparing the first metric from the examples of the training data set to the second metric determined by the ML model; updating at least one of the set of weights of the ML model according to the comparison; and and determining, by the one or more processors, an indication as either a fit or a non-fit from applying the first digital therapeutic content to the ML model; storing, by the one or more processors, a link between the first digital therapeutic content and the indicator used to control provision of the first digital therapeutic content over the network by (i) restricting provision of the first digital therapeutic content in response to determining the indicator as incompatible, and (ii) allowing provision of the first digital therapeutic content in response to determining the indicator as incompatible, using one or more data structures; The method includes:

2. receiving, by the one or more processors, an identification that the first digital therapeutic content is associated with at least one domain of a plurality of domains; selecting, by the one or more processors, from a plurality of ML models, an ML model corresponding to the domain to apply to the first digital therapeutic content; Further comprising: The method of claim 1 , wherein determining the indicator further comprises determining the indicator as either a match or a mismatch with respect to the domain.

3. 10. The method of claim 1, further comprising generating, by the one or more processors, one or more portions of the first digital therapeutic content to correct from applying the ML model in response to determining, by the one or more processors, the indication of the first digital therapeutic content as incompatible.

4. 2. The method of claim 1, further comprising selecting, by the one or more processors, from a plurality of content items as the first digital therapeutic content for delivery, either a first content item or a second content item based on an indication of suitability or non-suitability for each of the plurality of content items determined using the ML model.

5. receiving, by the one or more processors, a selection via a user interface of a second indication identifying the first digital therapeutic content as being compatible or incompatible; overriding, by the one or more processors, the indication from the ML model with the second indication received via the user interface; The method of claim 1 further comprising:

6. each instance of the plurality of instances of the training dataset further identifies a first risk score that identifies a degree of suitability or unsuitability for provision of the second digital therapeutic content; 2. The method of claim 1, wherein determining the second indicator further comprises determining a second risk score identifying a degree of suitability or non-suitability for provision of the first digital therapeutic content.

7. 10. The method of claim 1, wherein the first digital therapeutic content includes at least one of textual or visual content provided to a device for presentation in a session that addresses a symptom of an audience, the audience receiving a medication to address the symptom at least partially contemporaneously with the session.

8. 1. A method of training a model to regulate provision of digital therapeutic content, comprising: identifying, by one or more processors, a training data set including a plurality of examples, each example of the plurality of examples identifying a respective first digital therapeutic content and a first indicator identifying either a match or a mismatch, and controlling provision of the first digital therapeutic content over a network by (i) restricting provision of the first digital therapeutic content when mismatched, and (ii) allowing provision of the first digital therapeutic content when matched; applying, by the one or more processors, the first digital therapeutic content from one example of the plurality of examples of the training dataset to a machine learning (ML) model having a set of weights to generate an output; determining, by the one or more processors, from the output, a second indication of either suitability or non-suitability of the first digital therapeutic content for use in controlling delivery; comparing, by the one or more processors, the first metric from the examples of the training data set to the second metric determined by the ML model; updating, by the one or more processors, at least one of the set of weights of the ML model according to the comparison between the first indicator and the second indicator; storing, by the one or more processors, the set of weights of the ML model for application to determine suitability or non-suitability of a second digital therapeutic content; The method includes:

9. receiving, by the one or more processors, via a user interface, a selection of an indication identifying the second digital therapeutic content as being compatible or incompatible; comparing, by the one or more processors, the indicators determined from the output of the ML model with the indicators from the user interface; retraining, by the one or more processors, the ML model according to the comparison; and The method of claim 8 further comprising:

10. 10. The method of claim 8, further comprising generating, by the one or more processors, one or more portions of the second digital therapeutic content to modify from applying the ML model in response to determining, by the one or more processors, the indicator of the second digital therapeutic content as being incompatible.

11. 11. The method of claim 10, further comprising retraining, by the one or more processors, the ML model using the one or more portions of the second digital therapeutic content that are modified.

12. 10. The method of claim 8, further comprising modifying, by the one or more processors, a prompt applied to a generative Transformer model used to output the first digital therapeutic content in response to determining the incompatibility of the second digital therapeutic content, the generative Transformer model being part of the ML model or distinct from the ML model.

13. 10. The method of claim 8, further comprising retraining, by the one or more processors, a generative transformer model used to generate the first digital therapeutic content based on the indication of either the non-match or match of the second digital therapeutic content.

14. each instance of the plurality of instances of the training dataset further identifies a first risk score that identifies a degree of suitability or unsuitability for provision of the first digital therapeutic content; applying includes applying the first digital therapeutic content from one of the plurality of examples of the training dataset to the ML model and determining a second risk score identifying the degree of suitability or non-suitability for delivery of the first digital therapeutic content; For said example, updating at least one of said set of weights of said ML model according to a comparison between said first risk score and said second risk score. The method of claim 8 further comprising:

15. Each example of the plurality of examples of the training dataset further includes an identification of the first digital therapeutic content as a domain of a plurality of domains and the first indicator identifying either a match or a mismatch with respect to the domain; identifying, by the one or more processors, from a plurality of ML models, an ML model corresponding to the domain to apply to the first digital therapeutic content; 10. The method of claim 8, wherein determining the second indication further comprises determining the second indication of either suitability or non-suitability of the first digital therapeutic.

16. 1. A system for regulating digital therapeutic content for delivery, comprising: One or more processors coupled to the memory, Identifying a first digital therapeutic content provided over the network; applying the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output, the ML model identifying a training data set including a plurality of examples, each example of the plurality of examples identifying a respective second digital therapeutic content and a first indicator identifying either a match or a match for delivery; applying the second digital therapeutic content from one of the examples of the training dataset to the ML model to generate a second output; determining from the second output a second indication of either suitability or non-suitability of the second digital therapeutic content used to control delivery; comparing the first metric from the examples of the training data set to the second metric determined by the ML model; updating at least one of the set of weights of the ML model according to the comparison; and Trained by determining an indication as either a match or a mismatch from applying the first digital therapeutic content to the ML model; Using one or more data structures to store a link between the first digital therapeutic content and the indicator that is used to control provision of the first digital therapeutic content over the network by (i) restricting provision of the first digital therapeutic content in response to determining the indicator as incompatible, and (ii) allowing provision of the first digital therapeutic content in response to determining the indicator as compatible. one or more processors configured to A system comprising:

17. The one or more processors: receiving an identification that the first digital therapeutic content is associated with at least one domain of a plurality of domains; selecting, from a plurality of ML models, the ML model corresponding to the domain to apply to the first digital therapeutic content; Determining the indicator as either a match or a mismatch with respect to the domain. The system of claim 16 further configured to:

18. 17. The system of claim 16, wherein the one or more processors are further configured to generate one or more portions of the first digital therapeutic content to modify from applying the ML model in response to determining the indicator of the first digital therapeutic content as incompatible.

19. 17. The system of claim 16, wherein the one or more processors are further configured to select, from a plurality of content items, either a first content item or a second content item as the first digital therapeutic content for delivery based on an indication of suitability or non-suitability for each of the plurality of content items determined using the ML model.

20. 17. The system of claim 16, wherein the first digital therapeutic content includes at least one of textual or visual content provided to a device for presentation in a session that addresses a symptom of an audience, the audience receiving a medication to address the symptom at least partially contemporaneously with the session.

Images