System and method for providing messages with content from different sources based on risk and feedback data adjustments

Abstract

Aspects of the present disclosure relate to systems, methods, and computer-readable media for adjusting digital therapy content for provision. A computing system may identify digital treatment content provided via a network. The computing system may apply the digital treatment content to a machine learning (ML) model having a set of weights. The computing system may determine an indication of one of compliance or non-compliance based on application of the digital treatment content to the ML model. A computing system may use one or more data structures to store associations between digital treatment content and indications for controlling the provision of digital treatment content via a network by (i) restricting provision of first digital treatment content in response to determining an indication of non-compliance, and (ii) allowing the first digital treatment content to be provided in response to determining the indication of compliance.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 18 / 377,905 filed on October 9, 2023 and U.S. Patent Application No. 18 / 377,931 filed on October 9, 2023, the entire contents of which are incorporated herein by reference. Background Art

[0003] In a networked environment, a server may send a message to an end-user device to provide various information to the end-user. The content in the message may potentially contain erroneous or malicious information, or other content that is inappropriate to provide to the end-user. In the context of digital therapy, the message may contain digital therapy content that includes inaccurate or redundant information, such as information related to the end-user's condition, which may have an adverse effect on the end-user's condition, such as a worsening of the end-user's condition or compliance with the digital therapy.

[0004] Due to the complexity of digital treatment plans included in end users and the wide variety of personalized topics, generating information related to individual treatment plans would be tedious and time-consuming to create manually. Specifically, for digital treatments, there may be multiple subject categories that can provide input for effective treatment, taking into account knowledge from various sources. There is also a wide range of materials for developing digital treatments in different media and formats. The complexity of each individual end user and plan will make content generation resource intensive and error-prone. In addition, manually created content may lack scalability to a wider audience while lacking sufficient specificity to a specific end user condition.

[0005] Automatically generating digital therapeutic content using artificial intelligence (AI) techniques may encounter similar difficulties, as well as introducing illusions, inaccurate subject matter, or inappropriate wording or visualization of content items. For example, AI-generated content may contain inaccurate or insensitive subject matter and may also lack relevance to the end user. In addition, the problem may be exacerbated when multiple AI models are used to generate content items for digital therapeutic regimens. Parsing large amounts of AI-generated content for inaccuracies, relevance, and specificity would take a significant amount of time and be computationally difficult.

[0006] Due to the lack of accuracy, relevance, and specificity in manually created or automatically generated content for end users, as well as the difficulty in adapting the content of digital treatment plans at a larger scale, the provision of invalid messages will lead to a waste of computing resources (e.g., processors and memory) and network bandwidth. From a human-computer interaction (HCI) perspective, these issues can potentially lead to a lack of user interaction. In the context of digital therapeutics, users who receive such content will have low compliance with the treatment plan, resulting in worsening or no improvement in the end user's condition. Summary of the invention

[0007] To address these and other challenges, the services described herein can manage an interconnected architecture of multiple risk models based on multiple domains to evaluate content generated by human creators or generative AI (e.g., generative transformer models). Based on the evaluation, the service can determine which content (if any) to provide to the end user in a message (or content item) to address its specific condition for digital therapy applications. In addition, the service can provide suggestions for correcting the content (if any), and receive feedback on the provided content to iteratively train both the risk model and the generative AI model maintained by the message system. Content items created using generative transformer models (e.g., large language models or text-to-image models) can include content to be presented by means of the end user's device, with the purpose of preventing, mitigating or treating the end user's condition. Content items can also include information provided for presentation on web pages and applications, which is separate from the digital therapy solutions provided to specific users. By providing these capabilities, the service can improve the creation of digital therapy content and adjust the provision of digital therapy content through continuous learning in a single architecture.

[0008] The service can interact with a set of generative models and a database that maintains pre-generated content. In order to access the model or database, the service can receive input from an administrator. The input can identify parameters for generating or identifying digital therapeutic content, such as target conditions and domains for evaluating the content. The input can be provided as a prompt to the generative transformer model. The prompt can be personalized based on the end user submitting the text input, thereby providing a curated input to the generative transformer model to create a content item. In addition, the system can use the input to search for generated content items or previously generated content items from a database. In some embodiments, the service can determine that the text input corresponds to a content item already stored in the database, and the prompt provided to the generative transformer model may not be generated. The service can identify a set of content items related to the text input from multiple sources, including content items generated by a generative transformer model based on prompts created from input, from previously AI-generated content items, from artificially created content items, or a combination thereof. Due to the inherent differences in training between artificial intelligence models and human creators, a large number of sources of digital therapeutic content can vary.

[0009] By identifying content from AI or human sources, the service can verify whether the content is suitable for providing to the end user (e.g., to help the end user alleviate or improve their condition). For verification, the service can select one or more risk models to apply to the content items to determine the risk score of each content item. Each risk model can be selected for different parameters of the content to provide customized recommendations for which of the identified content items are provided relative to the text input. For example, the service can select a risk model based on the audience indicated in the text input, the domain indicated in the text input, or the administrator who provided the text input. The messages provided to the end user during participation in the digital treatment program can belong to one or more domains, such as a regulatory domain, a medical domain, a product domain, or an audience experience domain. Selecting one or more risk models based on one or more domains indicated in the input provides further customization for the end user, which can help improve compliance with the digital treatment program.

[0010] Before assessing risky content, the service department may train a set of risk models to assess the risks associated with each content item identified according to the corresponding domain. Each risk model may be trained by a service person on examples including content items to determine whether the content item should be provided to the end user. In some embodiments, a training data set including content items may be provided to the risk model. Each content item may be associated with a label that includes an expected indication of compliance or non-compliance with the provided content. In some embodiments, the indication may include a risk score assigned to the content item for training purposes. The risk score may indicate compliance or non-compliance of the supply, such that a risk score below a threshold risk score may indicate compliance. The risk model may be trained on multiple different content items having different corresponding domains, audiences, or a combination thereof to produce multiple risk models, each customized for a different content item.

[0011] During training, the service may apply the content item from each example to the risk model to determine a risk score or compliance indication for each content item provided. The determined compliance indication may be compared with the compliance indication associated with the training content item. If the comparison shows that the determined indication corresponds to the marked indication in the training data, the service may determine that the risk model has successfully classified the content item. Otherwise, if the comparison shows that the determined indication corresponds to the marked indication in the training data, the service may determine that the risk model has not successfully classified the content item. The service may update the weight of the risk model accordingly. In addition, the risk model may be trained to provide or indicate portions of the content item that can be edited to change the non-compliant content item to a compliant content item, and these portions may be presented to the administrator. For example, the risk model may be trained to present suggested edits, portions of the content item to be changed, or replacement words within the content item. The administrator may accept or reject these portions to further help train the risk model. In addition, the training of the risk model may be partially interactive, and the administrator may provide feedback during training to maintain or correct the indication output by the risk model. The risk model may accept a response indicating that the content item indicated as compliant or non-compliant has been improperly classified. In this way, the risk model can continuously learn to better identify compliant content items to facilitate digital treatment regimens.

[0012] As risk models are built, the service can apply content items to each risk model to generate a risk score for the content item. The risk score can be used to determine whether the content item should be provided to the end user. The risk score is determined by the risk model by providing the content item to the risk model for evaluation. The risk model can also take additional inputs, such as information related to the end user, text input, the individual generative transformer model that created the content item, or the previous performance history of each generative transformer model. The service can select a risk model to apply based on the indicated domain of the input.

[0013] Based on the risk scores, the service can rank the content items and use the ranking to select the content items. For example, the service can identify the content items corresponding to the highest ranking (e.g., the lowest risk score) to be provided to the end user. The service can also rank the content items based on other criteria, such as feedback from the audience to previous digital therapeutic content, audience preferences for previous digital therapeutic content, identification of corresponding models used to generate content items, or audience behavior in response to previous digital therapeutic content, etc. Through these means, the content provided to the end user can be subjected to various filtering, parsing, and content checks to ensure that the content provided to the end user is the most effective for their digital therapeutic regimen.

[0014] By selecting content items based on the risk scores of the content items, the service can present the content items and their corresponding risk scores to the administrator. The service can generate instructions to display the content items on the administrator device. The display can include the selected content items, the corresponding risk scores, the corresponding domains, and the corresponding audiences among other information related to the selected content items. Through the presentation, the administrator can provide interaction with the administrator device to provide feedback on the selected content items. The interaction can include modifications to the content items, such as modifications to the visual, auditory, or tactile features of the content items. Through the interaction, the service can generate feedback data. The feedback data can include information such as content included in the message provided to the user, modifications to the content, or modifications to the instructions. The feedback data from the administrator can be used to update the generative transformer model itself. For example, the service can use the feedback data to calculate a loss metric as a function of the feedback data, and then use the loss metric to update the weights in the generative transformer model. The feedback data can also be used to generate subsequent prompts when creating messages for end users using the generative transformer model. For example, when generating prompts, the service can add at least a portion of the feedback data as part of the administrator's user information or message generation parameters.

[0015] Upon receiving feedback from the administrator or determining that there is no administrator feedback, the service may store the content item in a database of content items. The service may store the content item in association with the indication. The service may also store the content item with an indication indicating whether the content item is to be provided to the end user. For example, the service may store the content item in association with an indication that the content item may be provided to the end user.

[0016] By storing the content items, the service can send a message containing the content items for presentation on the end-user device. Multiple content items can be presented in different media. For example, an application running on the end-user device can present the content of the message. When presented via the end-user device, the message can guide the end-user to perform activities that advance the digital treatment plan. The application on the end-user device can monitor the end-user's interaction with the message or the application itself. The interaction can include, for example, an indication that a specified activity has been performed or input of the end-user's reaction to the presentation of the message, as well as other responses. Using the detected interaction, the application can generate and send a response to provide to the service.

[0017] In addition, the service can send messages to end-user devices for presentation to a wider audience. For example, the service can send messages that include content items on a publicly accessible web page, such as a primary article and auxiliary content (e.g., within an inline frame on the web page). The content items can include information about clinical trials for participants in digital therapeutic applications. The content items can be transmitted for publication, such as in a medical journal, newspaper, or online database. For example, the content items can be included as primary content on a web page or as supplemental content inserted into a portion of a web page (e.g., an inline frame). The content items can be presented across a wide array of media.

[0018] Upon receipt, the service may use the response from the end-user device to update the generative transformer model itself. The service may parse the response of the end-user device to generate feedback data. The feedback data may include information such as the content included in the message provided to the user and an indication of whether the end-user performed an activity specified in the content of the message. The feedback data may be used to update the generative transformer model itself. For example, the service may use the feedback data to calculate a loss metric as a function of the feedback data, and then use the loss metric to update the weights in the generative transformer model. The feedback data may also be used to generate subsequent prompts when creating messages for the end-user using the generative transformer model. For example, when generating the prompt, the service may add at least a portion of the feedback data as part of the user information or message generation parameters. When creating a prompt for input to the generative transformer model, the service may combine feedback data from any number of previously presented messages.

[0019] In this way, the service can iteratively and continuously consider feedback from administrators' response data before providing content, as well as feedback data from end users when providing content into both the generative transformation model and the risk model maintained by the service. The output from the risk model can be provided to the generative transformation model to continuously train the generative transformation model for various domains, audiences, and text inputs. This technical solution enables content generated by other AI models (in addition to the content generated by the generative transformer model maintained by the messaging system) to be verified as compliant and subsequently ranked against each other to determine the most relevant content to provide to the end user.

[0020] Additionally, relative to human adaptation of content, the technical solution can achieve scalability for creating personalized digital therapeutic content by enabling the generation and validation of large amounts of diverse personalized content for provision to end users. Providing more relevant content can reduce resource consumption by reducing the computing power spent on generating and sending irrelevant messages to end users. The implementation of flexibility, scalability, and specificity can optimize or reduce the consumption of computing resources (e.g., processors and memory) and network bandwidth that would otherwise be wasted by providing ineffective content.

[0021] In the context of digital therapeutics, newly generated content can take into account changes in the end user's state, such as improvement or deterioration of the end user's condition or progression through a treatment regimen. By iteratively incorporating feedback to continuously train the generative transformer model and the risk model, HCI can be improved by providing more relevant and accurate content for specific end users. Content provided using the risk model can produce higher quality interactions of the end user with the application. In addition, an increase in engagement can lead to a higher level of compliance with the treatment regimen by the end user, resulting in a greater likelihood of preventing, alleviating, or treating the end user's condition.

[0022] Various aspects of the present disclosure relate to systems, methods, and computer-readable media configured for the generation of digital therapeutic content provided. A computing system may receive text input including one or more parameters via a user interface. The one or more parameters may identify an audience for which digital therapeutic content is to be generated and at least one domain in a set of domains for testing the risk of digital therapeutic content. The set of domains may include a regulatory domain and a medical domain. The computing system may use a prompt created based on the text input to identify a set of content items generated by a corresponding set of generative transformer models. Each of the set of content items may include corresponding digital therapeutic content generated from a corresponding model of the set of generative transformer models. The computing system may select at least one risk model corresponding to the at least one domain from a risk model group of the set of domains. The computing system may apply the at least one risk model to each content item in the set of content items to determine a corresponding risk score. The computing system may select a content item from the set of content items based on the risk score of the content item. The computing system may present a content item including corresponding digital therapeutic content via a user interface.

[0023] In some embodiments, the computing system may retrieve at least one content item previously generated and stored on a database using the text input. The computing system may add the at least one content item to the set of content items in response to determining that a second risk score of the at least one content item satisfies a second threshold. In some embodiments, the computing system may determine that no previously generated content items are stored on a database corresponding to the text input. In response to determining that no previously generated content items are stored, the computing system may invoke a generative transformer model group to generate a set of content items using a prompt created based on the text input.

[0024] In some embodiments, the computing system may apply a second risk model to one or more parameters of the text input to determine a second risk score. The computing system may present, via an interface, an indication that the second risk score exceeds a threshold. In some embodiments, the computing system may apply a plurality of risk models associated with the set of domains to the content item to determine a corresponding set of risk scores. The set of domains may also include a product domain and an audience experience domain. The computing system may present, via a user interface, information identifying a set of risk scores for the content item.

[0025] In some embodiments, the computing system may rank the set of content items based on at least one of (i) feedback from the audience to previous digital therapeutic content, (ii) audience preferences for previous digital therapeutic content, (iii) identification of corresponding models used to generate content items, and (iv) audience behavior in response to previous digital therapeutic content. In some embodiments, the computing system may apply a second risk model to one or more parameters of the text input to determine a second risk score. The computing system may apply the set of generative models to generate the set of content items in response to the second risk score satisfying a threshold.

[0026] In some embodiments, the computing system may receive information identifying one or more portions to be modified in the content item via a user interface. The computing system may store an association of the information with the content item on a database. In some embodiments, the computing system may identify a generative transformer model to be updated from a set of generative transformer models based on at least one of a textual input, a content item, or a response from a presentation of the content item. The computing system may provide at least a portion of the textual input, the content item, or the response data to update a generative transformer model identified from a set of generative transformer models.

[0027] In some embodiments, each content item in the set of content items may include at least one of textual content or visual content that is to be provided to a device for presentation in a session to address an audience's condition, wherein the audience is taking medication to at least partially address the condition in accordance with the session.

[0028] Various aspects of the present disclosure relate to systems, methods, and computer-readable media for adjusting digital therapeutic content provided. A computing system may identify first digital therapeutic content provided via a network. The computing system may apply the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output. The ML model may be trained by identifying a training data set, the training data set comprising a set of examples, each example of the set of examples identifying a corresponding second digital therapeutic content for providing and identifying a first indication of one of compliance or non-compliance. The ML model may be trained by applying the second digital therapeutic content of an example of the set of examples from the training data set to the ML model to generate a second output. The ML model may be trained by determining a second indication for controlling one of compliance or non-compliance of the second digital therapeutic content provided according to the second output. The ML model may be trained by comparing the first indication of the example from the training data set with the second indication determined by the ML model. The ML model may be trained by updating at least one of the weight sets of the ML model according to the comparison. The computing system may determine an indication of one of compliance or non-compliance according to applying the first digital therapeutic content to the ML model. The computing system may use one or more data structures to store an association between first digital therapeutic content and an indication that is used to control provision of the first digital therapeutic content via a network by: (i) limiting provision of the first digital therapeutic content in response to an indication determining non-compliance, and (ii) allowing provision of the first digital therapeutic content in response to an indication determining compliance.

[0029] In some embodiments, a computing system may receive an identification of first digital therapeutic content associated with at least one domain in a set of domains. The computing system may select an ML model corresponding to the domain to be applied to the first digital therapeutic content from a plurality of ML models. The computing system may determine the indication as one of compliance or non-compliance with respect to the domain. In some embodiments, the computing system may, in response to determining the indication of non-compliance of the first digital therapeutic content, generate one or more portions for modification in the first digital therapeutic content according to applying the ML model.

[0030] In some embodiments, the computing system may select one of the first content item or the second content item from the plurality of content items as the first digital therapeutic content to be provided based on the indication of conformity or non-conformity of each of the plurality of content items determined using the ML model. In some embodiments, the computing system may receive, via a user interface, a selection of a second indication identifying the first digital therapeutic content as conforming or non-conforming. The computing system may override the indication from the ML model with the second indication received via the user interface.

[0031] In some embodiments, each example of the set of examples of the training data set also identifies a first risk score that identifies a degree of compliance or noncompliance for providing the second digital therapeutic content. The computing system can determine the second indication by determining a second risk score that identifies a degree of compliance or noncompliance for providing the first digital therapeutic content. In some embodiments, the first digital therapeutic content may include at least one of textual content or visual content that is to be provided to a device for presentation in a session to address a condition of an audience, wherein the audience is using medication to at least partially address the condition in accordance with the session.

[0032] Various aspects of the present disclosure relate to systems, methods, and computer-readable media for adjusting digital therapeutic content for provision. A computing system may identify a training data set including a set of examples, each example of the set of examples identifying a corresponding first digital therapeutic content and a first indication for identifying one of compliance or non-compliance, to control the provision of the first digital therapeutic content via a network by the following steps: (i) limiting the provision of the first digital therapeutic content when non-compliant, and (ii) allowing the provision of the first digital therapeutic content when compliant. The computing system may apply the first digital therapeutic content of the example set from the training data set to a machine learning (ML) model including a set of weights to generate an output. The computing system may determine a second indication for controlling one of compliance or non-compliance of the first digital therapeutic content provided. The computing system may compare the first indication of the example from the training data set with the second indication determined by the ML model. The computing system may update at least one of the set of weights of the ML model based on the comparison between the first indication and the second indication. The computing system may store a set of weights of the ML model for determining compliance or non-compliance of the second digital therapeutic content.

[0033] In some embodiments, the computing system may receive, via a user interface, a selection of an indication identifying the second digital therapeutic content as compliant or non-compliant. The computing system may compare the indication determined from the output of the ML model with the indication determined from the user interface. The computing system may retrain the ML model based on the comparison. In some embodiments, the computing system may, in response to determining the indication of non-compliant second digital therapeutic content, generate one or more portions for modification in the second digital therapeutic content based on applying the ML model.

[0034] In some embodiments, the computing system may retrain the ML model using one or more parts to modify the second digital therapeutic content. In some embodiments, in response to determining non-conformity of the second digital therapeutic content, the computing system may modify a prompt applied to a generative transformer model used to output the first digital therapeutic content, wherein the generative transformer model is part of the ML model or is separate from the ML model. In some embodiments, the computing system may retrain the generative transformer model used to generate the first digital therapeutic content based on an indication of one of non-conformity or conformity of the second digital therapeutic content.

[0035] In some embodiments, each example of the example set of the training data set may also identify a first risk score that identifies a degree of compliance or non-compliance for providing the first digital therapeutic content. The computing system may apply the first digital therapeutic content of the example from the example set of the training data set to the ML model to determine a second risk score that identifies a degree of compliance or non-compliance for providing the first digital therapeutic content. The computing system may update at least one of the set of weights of the ML model based on a comparison between the first risk score and the second risk score of the example.

[0036] In some embodiments, each example of the example set of training data sets may include identifying the first digital therapeutic content as a domain of a set of domains and identifying a first indication of one of compliance or non-compliance with the domain. The computing system may identify an ML model corresponding to the domain to be applied to the first digital therapeutic content from a set of ML models. The computing system may determine a second indication by determining a second indication of one of compliance or non-compliance with the first digital therapeutic. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects, aspects, features and advantages of the present disclosure will become more apparent and more easily understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0038] Figure 1 depicts a block diagram of a system for generating and adapting the content of messages directed to addressing a condition in a user in accordance with an illustrative embodiment;

[0039] Figure 2 depicts a block diagram of a process for identifying content in a system for generating and adapting content for targeted messages in accordance with an illustrative embodiment;

[0040] Figure 3 depicts a block diagram of a process for selecting one or more models for use in determining a risk score in a system for generating and adapting content of a targeted message in accordance with an illustrative embodiment;

[0041] Figure 4depicts a block diagram of a process for selecting content items to be sent in a message to an administrator device in a system for generating and adapting a targeted message in accordance with an illustrative embodiment;

[0042] FIG. 5A to FIG. 5C depicts a block diagram of a process for updating a generative transformer model in a system for adapting and generating content for a target message in accordance with an illustrative embodiment;

[0043] Figure 6 depicts a block diagram of the architecture of a system for generating content items in a system for adapting and generating content for a targeted message in accordance with an illustrative embodiment;

[0044] Figure 7 depicts a flow chart of a method of generating and adapting content for a targeted message in accordance with an illustrative embodiment;

[0045] Figure 8 depicts a block diagram of a process for training a validation model and applying the validation model to content in accordance with an illustrative embodiment;

[0046] Fig. 9 depicts a block diagram of a process for training a validation model using a training data set in a system to train the validation model and apply the validation model to content in accordance with an illustrative embodiment;

[0047] Fig.10 depicts a block diagram of a process for incorporating feedback for a validation model into a system to train the validation model and apply the validation model to content in accordance with an illustrative embodiment;

[0048] Fig.11 depicts a block diagram of a process for providing content items to users in a system for use in training a validation model and applying the validation model to the content in accordance with an illustrative embodiment;

[0049] FIG. 12A to FIG. 12C describes an exemplary user interface for a system for training a validation model and applying the validation model to content in accordance with an illustrative embodiment;

[0050] Fig.13 depicts a flow chart of a method for training a validation model and applying the validation model to content according to an illustrative embodiment;

[0051] Fig.14 A flowchart depicting a method of providing content to users in a system for use in training a validation model and applying the validation model to the content according to an illustrative embodiment; and

[0052] Fig.15 is a block diagram of a server system and a client computer system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0053] In order to read the description of the various embodiments below, it will be useful to list the various parts of the specification and their respective contents below:

[0054] Section A describes systems and methods for generating and tailoring the content of messages directed to addressing a condition in a user;

[0055] Section B describes systems and methods for training and applying content validation models; and

[0056] Section C describes a network and computing environment that can be used to practice the embodiments described herein.

[0057] A. Systems and methods for generating and tailoring the content of messages targeted at addressing pathologies in users

[0058] Reference now Figure 1 , which depicts a block diagram of a system 100 for generating and adjusting content of messages directed to addressing a condition in an end user. In overview, the system 100 may include at least one data processing system 105, a set of end-user devices 110A-N (hereinafter collectively referred to as end-user devices 110), and at least one administrator device 190, which are communicatively coupled to each other via at least one network 115. The administrator device 190 may include at least one application 120. The application 120 may include or provide at least one user interface 125 having one or more UI elements 130A-N (hereinafter collectively referred to as UI elements 130). The data processing system 105 may include at least one content handler 140, at least one content controller 145, at least one risk assessor 150, at least one feedback handler 155, at least one model trainer 160, at least one generative transformer model 165A-N, and at least one risk model 185A-N, etc. The data processing system 105 may include or have access to at least one database 135. Database 135 may store, maintain or otherwise include one or more end-user profiles 170A-N (hereinafter collectively referred to as end-user profiles 170) and one or more content items 180A-N, etc. The functionality of application 120 on administrator device 190 may be partially performed on data processing system 105, and vice versa.

[0059] Within the data processing system 105, the content handler 140 may receive input to generate content items 180 through generative transformer models 165A-N (hereinafter collectively referred to as generative transformer models 165), and may provide selected content items 180 related to a session initiated by an administrator of the application 120 on an administrator device 190. The content controller 145 may identify the content items 180. The risk assessor 150 may apply the risk models 185A-N to determine a risk score for each content item 180. The feedback handler 155 may use the response from the administrator device 190 to generate feedback to update the generative transformer model 165, the content item 180, or the risk model 185A-N (hereinafter collectively referred to as the risk model 185). The model trainer 160 may train, improve, or update the generative transformer model 165 or the risk model 185 related to the session initiated by the administrator of the application 120.

[0060] In more detail, data processing system 105 may be any computing device that includes one or more processors coupled to memory and software and is capable of performing the various processes and tasks described herein. Data processing system 105 may communicate with one or more user devices 110, administrator device 190, and database 135 via network 115. Data processing system 105 may be located, located at, or otherwise associated with at least one computer system. The computer system may correspond to a data center, a branch office, or a site where one or more computers corresponding to data processing system 105 are located.

[0061] The end-user device 110 may be any computing device that includes one or more processors coupled with memory and software and is capable of performing the various processes and tasks described herein. The end-user device 110 may communicate with the data processing system 105, the administrator device 190, and the database 135 via a network 115. The end-user device 110 may be a smart phone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses), or laptop computer. The end-user device may be provided with one or more content items 180 via the data processing system 105, or the end-user device may request one or more content items 180 via interaction 105 with the data processing system, such as via an application associated with the end-user device 110.

[0062] Administrator device 190 can be any computing device that includes one or more processors coupled with memory and software and is capable of performing the various processes and tasks described herein. Administrator device 190 can be associated with entities that interact with data processing system 105 to control and adjust the generation of content items 180. Administrator device 190 can communicate with data processing system 105, user device 110, and database 135 via network 115. Administrator device 190 can be a smart phone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses), or laptop computer. Administrator device 190 can be used to access application 120. In some embodiments, application 120 can be downloaded and installed on administrator device 190 (e.g., via a digital publishing platform). In some embodiments, application 120 can be a web application with resources accessible via network 115.

[0063] Application 120 executing on administrator device 190 may interact with data processing system 105 to generate or modify one or more content items 180. In some embodiments, application 120 may be an application for generating one or more messages for providing to an audience of end users associated with end user device 110 in conjunction with a digital therapeutic application to address at least one condition of the end user. End-user conditions may include, for example, chronic pain (e.g., related to or including arthritis, migraines, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive stress injury, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin diseases (e.g., atopic dermatitis, psoriasis, pruritus, and eczema), cognitive disorders (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis, and schizophrenia), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, borderline personality disorder, and attention-deficit / hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, tobacco use disorder, or hallucinogenic disorder), and other diseases (e.g., narcolepsy and oncology or cancer).

[0064] The terminal user can take medicine at least partially at the same time to solve the disease. For example, if the medicine is used for pain, the terminal user can take acetaminophen, non-steroidal anti-inflammatory composition, antidepressant, anticonvulsant or other compositions, etc. For skin diseases, the terminal user can take steroids, antihistamines or major preservatives, etc. For cognitive impairment, the terminal user can take cholinesterase inhibitors or memantine, etc. For narcolepsy, the terminal user can take stimulants or antidepressants, etc. The terminal user can also participate in other psychotherapy for these diseases. In some embodiments, content items 180 can be provided to the terminal user in the digital treatment application to realize the endpoint of the terminal user. The endpoint can be, for example, the physical or psychological goal of the terminal user, the completion of the drug treatment plan, or the endpoint indicated by a doctor or terminal user. At least one of the terminal user devices 110 can have a digital treatment application, and a session (sometimes referred to as a treatment session in this article) can be provided to solve at least one disease of the terminal user.

[0065] In some embodiments, the application 120 may be an application for generating one or more content items 180 for submission to a medical journal, a government agency, or a subject matter expert. The application 120 may be an application for generating or modifying summaries associated with one or more of a clinical trial, a test trial, a journal entry or publication, etc. For example, the application 120 may be an application for generating a clinical summary report on a topic related to a patient condition, a medication, a psychotherapy or treatment, etc.

[0066] Depending on the configuration on application 120, application 120 may include, be present, or otherwise provide a user interface 125 including one or more UI elements 130A-N (hereinafter collectively referred to as UI elements 130) to an administrator of administrator device 190. UI elements 130 may correspond to visual components of user interface 125, such as command buttons, text boxes, check boxes, radio buttons, menu items, sliders, etc. In some embodiments, the administrator may interact with UI elements 130 to provide feedback, responses, or other interactions to generate and modify content items 180 when interacting with data processing system 105.

[0067] Database 135 may store and maintain various resources and data associated with data processing system 105 and application 120. Database 135 may include a database management system (DBMS) to arrange and organize the data maintained thereon, such as end-user profile 170, content item 180, risk model 185, or generative transformer model 165, etc. Database 135 may communicate with data processing system 105, administrator device 190, and one or more end-user devices 110 via network 115. When executing various operations, data processing system 105 and application 120 may access database 135 to retrieve identified data therefrom. Data processing system 105 and application 120 may also write data to database 135 in accordance with executing such operations.

[0068] On the database 135, each end-user profile 170 (sometimes referred to herein as an end-user account or end-user information) can store and maintain information related to the end-user through the end-user device 110. Each end-user profile 170 can be associated with or correspond to a corresponding end-user to which a content item 180 is provided. The end-user profile 170 can identify various information about the end-user, such as an end-user identifier, a condition to be addressed, information about a session conducted by the end-user (e.g., a completed activity or course, or other content item 180 generated or modified by an administrator), preferences, user characteristic information, and progress status in addressing the condition (e.g., completion of an endpoint), etc. The information about the session can include various parameters of a previous session performed by the end-user and can be initially empty. The preferences can include message preferences. The message preferences can include treatment preferences and end-user input preferences, such as the type of message or the preferred timing of the message. The message preferences can also include preferences determined by the data processing system 105, such as the type of message to which the end-user can respond. The preferences can include profile preferences, such as words, phrases, or content items 180 preferred by the administrator for inclusion in the content item 180. End user profile 170 may be continuously updated by application 120 and data processing system 105 .

[0069] In some embodiments, the end-user profile 170 may identify or include information about a treatment regimen conducted by the end-user, such as the type of treatment (e.g., therapy, medication, or psychotherapy), duration (e.g., days, weeks, or years), and frequency (e.g., daily, weekly, quarterly, annually), etc. The end-user profile 170 may include at least one activity log of messages provided to the end-user, interactions by the end-user that identify the performance of the particular end-user, and responses from the end-user device 110 associated with the end-user, etc. The end-user profile 170 may be stored and maintained in the database 135 using one or more files (e.g., extensible markup language (XML), comma value (CSV) delimited text files, or structured query language (SQL) files). The end-user profile 170 may be iteratively updated as the end-user performs additional sessions, provides input, or responds to content items 180.

[0070] The content item 180 may be in any form, such as text, image, audio, video or multimedia content, etc., or any combination thereof. The content item 180 may be stored and maintained in the database 135 using one or more files. For example, for text, the content item 180 may be stored as a text file (TXT), a rich text file (RTF), an extensible markup language (XML), and a hypertext markup language (HTTP), etc. For images, the content item 180 may be stored as a Joint Photographic Experts Group (JPEG) format, a Portable Network Graphics (PNG) format, a Graphics Interchange Format (GIF), or a Scalable Vector Graphics (SVG) format, etc. For audio, the content item 180 may be stored as a waveform audio file (WAV), a Moving Picture Experts Group format (e.g., MP3 and MP4), and an OggVorbis (OGG) format, etc. For video, the content item 180 may be stored as a Movie Experts Group format (e.g., MP3 and MP4), a QuickTime movie (MOV), and a Windows Movie Video (WMV), etc. For multimedia content, the content item 180 may be Audio Video Interleave (AVI), Movie Experts Group formats (eg, MP3 and MP4), QuickTime Movie (MOV), Windows Movie Video (WMV), and the like.

[0071] Each content item 180 may identify or include information to be presented via an end-user device 110 or an administrator device 190. For example, the content item 180 may be presented to an end-user or administrator via a message sent to the end-user device 110 or administrator device 190, respectively. The message may be in any format, such as a short message / messaging service (SMS), a multimedia messaging service (MMS), or as instructions presented via a display associated with the end-user device 110, the administrator device 190, or the like.

[0072] Content item 180 of the message may include a reminder to perform the task of the session. The message may be derived from a pre-generated psychotherapy message library or a pre-generated engagement (reminder) message library. The message may include a reminder for the end user to complete the therapy session, take medication, or complete the task of the regimen. The message may include an activity for the end user to perform or a course for the end user to participate in. Content item 180 may also include a mechanism for responding, such as a link, chat box, or instructions for responding to the message.

[0073] Content item 180 may include or correspond to one or more texts, such as articles, summaries, or publications. For example, content item 180 may include a research article, a review article, a case report, a clinical trial protocol, or an editorial, etc. Content item 180 may include text submitted to a government agency, a subject matter expert, a scientific journal, or a conference, etc. For example, content item 180 may include a clinical trial protocol related to a treatment provided for an end-user's condition, for submission to the Food and Drug Administration (FDA), a medical journal, or for internal distribution.

[0074] Content items 180A-N may be generated and stored in the database 135 prior to the [X] period to generate and modify content items 180 operating on the administrator device 190. Content items 180 may be manually created, computer generated, or a combination thereof. In some embodiments, an administrator may provide content items 180 through an application 120 operating on the administrator device 190. For example, an administrator may upload, provide, or transmit one or more content items 180 to an application 120 during a session to be stored in the database 135, thereby generating and modifying content items 180. Content items 180 may be computer generated, such as generated by a generative transformer model 165. In some embodiments, an administrator may provide input through an application 120 operating on the administrator device 190 to create one or more content items 180 using a generative transformer model 165. For example, an administrator may provide text, images, videos, or other presentations as input to generate content items 180. One or more generative transformer models 165 may generate one or more content items 180 based on prompts created by input from the administrator. In some embodiments, the data processing system 105 may modify the input to include additional information to generate prompts to provide to the generative transformer models 165. Each generative transformer model 165 may generate one or more content items 180 based on the provided prompts.

[0075] Generative transformer model 165 may receive input in the form of a set of strings (e.g., from text input) and output content (e.g., content item 180) in one or more modes (e.g., in the form of text strings, audio content, images, videos, or multimedia content). Generative transformer model 165 may be a machine learning model based on a transformer model (e.g., a generative pre-trained model or a bidirectional encoder representation from a transformer). Generative transformer model 165 may be a large language model (LLM), a text-to-image model, a text-to-audio model, or a text-to-video model, etc. In some embodiments, generative transformer model 165 may be part of data processing system 105 (e.g., as shown). In some embodiments, generative transformer model 165 may be part of a server that is separate from data processing system 105 and communicates with data processing system 105 via network 115.

[0076] One or more generative transformer models 165 may be trained and maintained by the data processing system 105. The generative transformer model 165 may include a set of weights arranged across a set of layers according to a transformer architecture. Under this architecture, the generative transformer model 165 may include at least one word segmentation layer (sometimes referred to herein as a tokenizer), at least one input embedding layer, at least one position encoder, at least one encoder stack, at least one decoder stack, and at least one output layer, etc., wherein these layers are interconnected to each other (e.g., via forward, backward or skip connections). In some embodiments, the generative transformer layer 165 may lack an encoder stack (e.g., for an encoder-only architecture) or a decoder stack (e.g., for a decoder-only model architecture). The word segmentation layer may convert the original input in the form of a string group into a corresponding word vector group (also referred to herein as a token or vector) in an n-dimensional feature space. The input embedding layer may generate a set of embeddings using the set of word vectors. Each embedding may be a low-dimensional representation of a corresponding word vector, and may capture the semantic and grammatical information of a string associated with a word vector. The position encoder may generate a position encoding for each input embedding according to the position of the corresponding word vector or by expanding the string within the input string set.

[0077] Next, in the generative transformer model 165, the encoder stack may include a set of encoders. Each encoder may include at least one attention layer and at least one feedforward layer, etc. The attention layer (e.g., a multi-header self-attention layer) may calculate an attention score for each input embedding to indicate the degree of attention to be focused on the embedding, and generate a weighted sum of the input embedding set. The feedforward layer may apply a linear transformation with a nonlinear activation (e.g., a reconciled linear unit (ReLU)) to the output of the attention layer. The output may be fed to another encoder in the encoder stack in the generative transformer layer 165. When the encoder is a terminal encoder in the encoder stack, the output may be fed to a decoder stack.

[0078] The decoder stack may include at least one attention layer, at least one encoder-decoder attention layer, and at least one feedforward layer, among others. In the decoder stack, an attention layer (e.g., a multi-header self-attention layer) may calculate an attention score for each output embedding (e.g., an embedding generated from a target or expected output). The encoder-decoder attention layer may combine inputs from an attention layer within the decoder stack and outputs from one encoder in the encoder stack, and may calculate an attention score from the combined inputs. The feedforward layer may apply a linear transformation with a nonlinear activation (e.g., a reconciled linear unit (ReLU)) to the output of the encoder-decoder attention layer. The output of the decoder may be fed to another decoder within the decoder stack. When the decoder is a terminal decoder in the decoder stack, the output may be fed to an output layer.

[0079] The output layer of the generative transformer model 165 may include at least one linear layer and at least one activation layer, etc. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to calculate the word-unit score. The activation layer may apply an activation function (e.g., Softmax, sigmoid, or adjusted linear unit) to the output of the linear function to convert the word-unit score into a probability (or distribution). The probability may represent the likelihood of the output word-unit occurring for a given input word-unit. The output layer may use the probability to select the output word-unit (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability). Repeating this on a set of input tags, the resulting set of output tags may be used to form the output of the entire generative transformer model 165. Although this article is primarily described in terms of a transformer model, the data processing system 105 may also use other machine learning models to generate and output content. In some implementations, the data processing system 105 may generate and output content using one or more models maintained by an external system. For example, the data processing system may generate content using one or more models such as ChatGPT produced by OpenAI, BARD produced by Google, or LLaMA produced by Meta.

[0080] Each generative transformer model 165 can generate one or more of content items 180 based on the prompts provided to the generative transformer model 165. Each content item 180 generated according to the prompts created from the text input provided to the data processing system 105 can be different due to the differences in each generative transformer model 165. In this way, content item 180A can be more suitable for providing to the end user (first by the administrator via the user interface 125) than other content items 180B-N. For example, the content item 180 generated by the generative transformer model 165 may include inaccurate, irrelevant content or illusion. In some embodiments, the content item 180A generated by the generative transformer model 165A may be irrelevant to a specific user due to the information in the content item 180A, the disease solved by the content item 180A, the presentation style of the content item 180A, or the authoritative assertion provided by the content item 180A. For example, the content item 180A can assert that it is the "best" treatment method for a given disease; however, this cannot be asserted and provides wrong information. For example, content item 180A may recommend to the user that a meat-based dish be consumed without recognizing that the user was previously indicated as a vegetarian. For example, content item 180A may be a style of text presentation, despite the user's prior behavior from prior sessions indicating that the user more consistently relies on the session when video content is presented. For example, content item 180A may generate data that is not verified or proven to be true. In order to mitigate the content items 180 generated by the generative transformer model 165, the data processing system 105 may train and use one or more risk models 185 to the content items 180 to determine a risk score associated with each content item 180.

[0081] Each risk model 185 can be a machine learning model trained to determine a risk score associated with a content item 180, a prompt, or a combination thereof. The risk model 185 can be trained as described herein (e.g., in conjunction with the training of a validation model) to calculate a risk score for a content item 180, a prompt, or a combination thereof. The risk scores generated by the risk model 185 can also be used to continuously train the generative transformer model 165 to provide more relevant, more accurate, or less risky content items 180 over time. The risk model 185 can include one or more natural language models, including the generative transformer model 165 described herein. The risk model 185 can include one or more classifier models, such as a naive Bayes classifier, a support vector machine (SVM) ensemble classifier, a kernel approximation, a k-nearest neighbor classifier, or a decision tree, among others.

[0082] One or more risk models 185 may accept a prompt as input. By accepting the prompt as input, one or more risk models 185 may generate a risk score associated with the likelihood of a particular prompt generating a desired content item 180. The desired content item 180 may include a content item 180 in a format specified by the prompt with desired accuracy (e.g., correct information, a relevant data set) or with desired relevance (e.g., to an end user receiving the content item 180 as part of a digital therapy session or an administrator receiving text of a desired article type), etc. One or more risk models 185 may accept a generated content item 180 as input. By accepting the content item 180 as input, one or more risk models 185 may generate a risk score associated with the likelihood that the content item 180 is a desired content item 180, as described above.

[0083] Reference now Figure 2 , which depicts a block diagram of a process 200 for identifying content in a system 100 for generating and adapting content for a targeted message. The process 200 may include or correspond to operations performed in a content system 100 for generating and adapting a targeted message. Under the process 200, the content handler 140 may receive, retrieve, or recognize text input, utilize the text input to create a prompt 230 for providing to the generative transformer model 165. The risk assessor 150 may generate a risk score 235 for the prompt 230 using the risk model 185.

[0084] The administrator 205 may provide an interaction 210 with the user interface 125 via the UI element 130 to generate or modify the content item 180. The interaction 210 may be part of a request to generate or modify a presented activity or message. For example, the interaction 210 may be provided in response to the presentation of the application 120 to generate or modify the content item 180 on the administrator device 190. The interaction 210 may include an interaction with the UI element 130. In some embodiments, the interaction 210 may include the administrator 205 providing text via the user interface 125 via a text box, a drop-down menu, or speech to text, etc. In some embodiments, the interaction 210 may include the administrator 205 making a selection via a drop-down box, a button, or another UI element 130. For example, the interaction 210 may include the administrator 205 making a selection of a type of content item 180 (e.g., a journal article, a clinical research report, etc.).

[0085] The application 120 operating on the administrator device 190 can accept the interaction 210 and can generate a text input 225 based on the interaction 210. In some embodiments, the interaction 210 includes a text input, which the application 120 can utilize in creating the input 225. In some embodiments, the application 120 generates the input 225 from a non-text interaction, such as a selection of a button or a drop-down, etc. The input 225 (also referred to as text input 225) can include one or more parameters for defining the generation of a message to be presented to the administrator 205, such as an audience 215A-N or a domain identifier 220A-N.

[0086] Application 120 may identify audiences 215A-N (collectively referred to below as audiences 215) indicated in input 225. Audiences 215 may refer to or include one or more individuals or organizations to which content item 180 is directed. For example, audiences 215 may include a grouping of patients with similar demographic characteristics, such as patients with similar conditions, with similar message preferences, condition severity levels, or medication prescriptions, etc. For example, audiences 215 may include a grouping of subject matter experts in similar fields of study, similar educational backgrounds, or similar locations, etc. For example, audiences 215 may include entities such as government organizations, educational institutions, or non-profit organizations.

[0087] Application 120 may identify domain identifiers 220A-N (hereinafter collectively referred to as domain identifiers 220) indicated in input 225. Domain identifiers 220 may correspond to or indicate one or more domains of input 225. A domain may correspond to an intent to generate content item 180. A domain may correspond to one or more of generative transformer models 165. For example, generative transformer model 165A may correspond to a first domain, such that generative transformer model 165A generates content item 180 using, training, or otherwise based on the domain. A domain may include a library of words, phrases, or rules for generating content item 180, such as generating content item 180 for information about a particular digital therapeutic treatment. Domains may include domains such as an audience experience domain, a regulatory domain, a compliance domain, a scientific or medical domain, or a product domain.

[0088] The domain identifier 220 may correspond to a domain used to test the risk of the content item 180, such as a scientific or medical domain (e.g., standards related to scientific research or medical literature, such as whether the information is medically or scientifically accurate, clinically and statistically relevant, and presented in a scientifically balanced manner), a regulatory domain (e.g., standards related to regulatory guidance, such as not claiming safety or effectiveness before the product has been cleared or approved by a regulatory agency), an audience experience domain (e.g., standards related to user experience, such as clear and understandable instructions, easily accessible technical support units, more attractive reward systems), a compliance domain (e.g., standards related to compliance requirements), a product domain (e.g., standards related to product requirements, such as including certain features within the scope of the project), a commercial domain (e.g., standards related to commercial release requirements), or a marketing domain (e.g., standards related to marketing products), etc. The domain may be derived from end-user behavior, end-user preferences, or profile information of a given audience 215, etc. The domain may be identified by an end-user profile 170 associated with the audience 215. For example, audience 215 may include end-user profiles 170 related to conditions such as chronic pain, skin pathology, cognitive impairment, mental health conditions, or substance use disorders.

[0089] In some embodiments, the profile of the administrator 205 may indicate the domain. For example, the settings (or profile) of the administrator 205 may indicate that the administrator 205 is a product manager, and the application may therefore identify the product domain for input 225. In some embodiments, the application 120 may include a preference selected by the administrator 205 or determined according to the settings of the administrator 205 to identify the domain. In some embodiments, the administrator 205 may identify the domain through interaction 210. For example, the administrator 205 may select the domain from a drop-down list of domains, or may provide text indicating the domain with interaction 210.

[0090] In some embodiments, the science domain or the medical domain may define a selection of content items 180 for end users suffering from a particular condition. In some embodiments, the medical domain may define appropriate psychoeducational sessions for a particular medical condition, appropriate activities for an end user to participate in as part of treatment for a particular medical condition, or text to satisfy a particular reading level of a particular patient subgroup for generating content items 180. In some embodiments, the medical domain may define a template for content items 180, such as the format of a research paper or journal publication, etc. In some embodiments, the audience experience domain may include end-user preferences derived from profile information in an end-user profile 170. The end-user preferences may correspond to the type of messages that the end-user prefers, as identified in the end-user profile 170 of the end user corresponding to the audience 215. The profile information may include other data points about the end user, such as the progress of treatment, the degree of difficulty, the stage on the path to achieve an endpoint, or end-user behavior, etc. For example, the end-user behavior may include the type of activity performed by the end user identified in the end-user profile 170.

[0091] In some embodiments, the product domain may include standards for user experience for a specific audience. The marketing domain may define a library of marketing phrases for generating content items 180. In some embodiments, the marketing domain may define templates for content items 180, such as product launch statements, product advisories, press releases, or marketing materials for products. In some embodiments, the regulatory domain may define regulatory guidelines for establishing and recruiting for clinical trials to provide various situations for the generative transformer model 165 to generate desired content items 180 (e.g., promotional materials may not reference safe or effective devices until approved or ratified by a regulatory agency). In some embodiments, the regulatory domain may provide templates or lists of content items 180 related to submissions to regulatory agencies, clinical trial protocols, press releases, or recall statements, etc.

[0092] Content handler 140 executing on data processing system 105 may retrieve, obtain, or otherwise receive input 225 from end-user device 110. Content handler 140 may perform any function of application 120 in generating input 225. In some embodiments, content handler 140 may retrieve information from one or more end-user profiles 170 to generate input 225. In some embodiments, content handler 140 may identify a profile corresponding to administrator 205 through input 225. Content handler 140 may also identify an identifier (e.g., a name), information about sessions conducted by administrator 205, activity logs, message preferences, etc. Content handler 140 may identify parameters of input 225. For example, content handler 140 may identify audience 215 or domain identifier 220 of input 225.

[0093] In some embodiments, upon receiving input 225, risk assessor 150 may determine risk score 235 for input 225. Risk assessor 150 may determine input 225 based at least on a parameter of input 225 (e.g., audience 215 or domain identifier 220 of input 225). Risk assessor 150 may apply one or more risk models 185 to input 225 (or a prompt derived from input 220) to generate risk score 235 for input 225. Upon applying one or more risk models 185 to the input, risk assessor 150 may generate risk score 235 for the input.

[0094] Applying the risk model 185 to the input 225 may include selecting one or more risk models 185. The risk assessor 150 may select one or more risk models 185 to apply to the input 225 based on the parameters of the input 225. For example, the risk assessor 150 may select a first risk model 185A for a domain identifier 220 corresponding to a medical domain and a second risk model 185B for a domain identifier 220 corresponding to a product domain. In some embodiments, the parameters of the input 225 may identify or include one or more domain identifiers 220 or an audience 215. The risk assessor 150 may select one or more risk models 185 corresponding to each parameter identified in the input 225.

[0095] Applying the risk model 185 may include providing the input 225 as an input to one or more selected risk models 185. Providing the input 225 may include the risk assessor 150 applying one or more parameters of the input 225. For example, the risk assessor 150 may provide as input the domain identifier 220 of the input 225, the audience 215 of the input 225, other parameters of the input 225, or a combination thereof. In some embodiments, the risk assessor 150 may provide a first parameter of the input 225 to a first risk model 185A and a second parameter of the input 225 to a second risk model 185B. For example, the risk assessor 150 may apply different, the same, or overlapping risk models 185 to each parameter (e.g., the audience 215, the domain identifier 220, the format type of the content item, etc.). In this manner, one or more risk scores 235 may be generated for each parameter of the input 225.

[0096] The risk score 235 generated by the risk assessor 150 based on the input 225 may be a number, score, rating, or other identifier that indicates the likelihood that the input 225 corresponds to generating the desired content item 180. In some embodiments, a lower risk score may indicate a higher likelihood that the content item 180 generated based on the input 225 corresponds to the desired audience 215, domain, format type, or other parameters of the input 225. In some embodiments, a lower risk score may indicate a higher likelihood that the content item 180 generated based on the input 225 meets a threshold of compliance with a rule or accuracy (e.g., the accuracy of the data, information, citation, or other facts contained within the content item 180). In some embodiments, a lower risk score may indicate a higher likelihood that the content item 180 generated based on the input 225 meets a threshold relevance (such as relevance to the audience 215 or compliance with the preferences of an end user or administrator). For example, the relevance may relate to information in the end user profile 170, such as a condition experienced by the end user or audience 215, the preferences of the administrator 205 for the type of content item 180, and the like. In some embodiments, a lower risk score may indicate a higher likelihood that the input 225 does not contain the subset of words or phrases or that the input 225 will not generate a content item 180 that contains the subset of words or phrases.

[0097] The content handler 140 may generate a prompt 230 based on the input 225 including the audience 215 and the domain identifier 220. In some embodiments, the content handler 140 may generate the prompt 230 in response to a risk score 235 that meets a threshold. For example, if the risk score 235 is below a threshold risk score, the content handler 140 may generate the prompt 230. In some embodiments, when the risk score 235 is at or above the threshold risk score, the content handler 140 may not generate the prompt 230 to generate the content item 180. For presentation on the user interface 125, the content handler 140 may provide an indication that the risk score 235 is at or above a threshold risk level. For example, the content handler 140 may generate a presentation to be displayed identifying the risk score 235. The indication may include an identification of the risk score 235 for each parameter of the input 225, an overall risk score 235 for the input 225, and the like.

[0098] The content handler 140 may generate the prompt based on information from the profile identification of the administrator 205, such as previous sessions for generating or modifying content items for the administrator 205, the occupation of the administrator 205, preferred content items for the administrator 205, domains associated with or selected by the administrator 205, etc. The content handler 140 may generate the prompt 230 based on a string of characters within the input 225. For example, the text input 225 may indicate or include a word, phrase, or string of characters that identifies the parameters of the content item 180 to be generated. The qualities may include the audience 215, the domain identifier 220, the file type 180 of the content item, the format type (e.g., journal article, image, animation, clinical trial protocol, etc.), or the size of the content item (e.g., file size, word count, page count), etc. The content processor 140 can generate prompts based on information identified from the end-user profile 170 corresponding to the audience 215, wherein the information such as implementation for the audience 215 to resolve the condition, the condition to be resolved, activity logs, end-user characteristics, end-user identifiers, message preferences, information about sessions conducted by the end-user, and progress in resolving the condition.

[0099] The content handler 140 may write, generate, or otherwise generate at least one prompt 230 using information from an end-user profile and one or more parameters. When generating, the content handler 140 may add, insert, or otherwise include information from an end-user profile 170 and one or more parameters in a template for generating the prompt. The template may include a set of defined strings and one or more placeholders that will be inserted into the string for input to the generative transformer model 165. The string may correspond to predetermined, fixed text to be included as part of the input prompt for the generative transformer model 165. By traversing the template and inserting data, the content handler 140 may form and construct the prompt 230.

[0100] Reference now Figure 3 , which depicts a block diagram for a process 300 for selecting one or more risk models 185 for use in determining risk scores 305A-N in a system 100 for generating and adjusting content for a target message. The process 300 may include or correspond to operations performed in the system 100 to generate and adjust content items 180 for a target message. Under the process 300, a content controller 145 may identify one or more content items 180. A risk assessor 150 executing on a data processing system 105 may identify one or more risk models 185 based on a domain 220 of the content item 180. The risk assessor 150 may apply the model 185 to the content item 180 to determine one or more risk scores 305A-N for the content item 180.

[0101] As prompts 230 are generated, content processor 140 may feed or apply prompts 230 to generative transformer model 165. In application, content processor 140 may process prompts 230 using a set of layers in generative transformer model 165. As described above, generative transformer model 165 may include a word segmentation layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer, among others. Content processor 140 may process prompts 230 using the word segmentation layer of generative transformer model 165 to generate a set of word vectors (sometimes referred to herein as word units or word units) for an input set. Each word vector may be a vector representation of at least a portion of prompt 230 in an n-dimensional feature space (e.g., using a word embedding table). Content processor 140 may apply the set of word vectors to an input embedding layer to generate a corresponding set of embeddings.

[0102] Through this processing, the content processor 140 can calculate the probability of each embedding. The probability can represent the likelihood of the output of a given input word element occurring. Based on these probabilities, the content processor 140 can select the output word element (e.g., output text, image, audio, video, or at least a portion of multimedia content with the highest probability) to form, generate, or otherwise generate at least a portion of the content item 180. The content item 180 output by the generative transformer model 165 may include text content, image content, audio content, video or multimedia content, etc., or any combination thereof.

[0103] In some embodiments, content items 180 generated by generative transformer model 165 may include text content, image content, audio content, video or multimedia content, etc., or any combination thereof. For example, the presentation of content item 180 may include an image of a smiling face to further encourage the user to complete their goal. In addition, content item 180 may include a video clip of a loved one, a motivational speaker, a medical professional, influencer or therapist, etc., or any combination thereof.

[0104] The content controller 145 may identify a content item 180 for providing to the administrator 205 via the application 120 operating on the administrator device 190. Identifying the content item 180 may include retrieving, receiving, or otherwise obtaining the content item 180 from one or more sources, such as from the generative transformer model 165 or the database 135. The content controller 145 may identify the content item 180 within the database 135 that corresponds to the input 225. For example, the content controller 145 may identify a previously generated, created, or uploaded content item 180 stored within the database 135.

[0105] The content controller 145 may determine that the stored content item 180 corresponds to the input 225 provided by the administrator 205. The content controller 145 may identify strings within the input 225 or parameters of the input 225 that are related to, correspond to, or are associated with the stored content item 180. For example, the input 225 may indicate the audience 215A that is also identified in the stored content item 180A. The content controller 145 may determine that one or more strings of the input 225 match or are similar to one or more strings of the content item 180 stored in the database 135. In this manner, the content controller 145 may identify the content item 180 that corresponds to the input 225 within the database 135, and may retrieve one or more content items 180 that were previously generated and stored on the database 135.

[0106] The content controller 145 may prevent the content handler 140 from generating the prompt 230 in response to determining that the one or more content items 180 in the database 135 correspond to the input 225. In some embodiments, the content controller 145 may access the database 135 to determine whether the one or more content items 180 in the database 135 correspond to the input 225 before the prompt 230 is generated by the content handler 140 or before the generated prompt 230 is provided to the generative transformer model 165. The content controller 145 may instruct the content handler 140 to avoid generating the prompt 230 or avoid providing the generated prompt 230 to the generative transformer model 165. In some embodiments, the content controller 145 may prevent the content handler 140 from providing the generated prompt 230 to the generative transformer model 165 in response to identifying the one or more corresponding content items 180 in the database 135.

[0107] The content controller 145 may determine that no previously generated content item 180 is stored on the database corresponding to the input 225. The content controller 145 may access the database 135 and determine that no previously generated content item 180 corresponding to the input 225 is stored in the database 135. In some embodiments, the content controller 145 may access the database 135 to make a determination before the prompt 230 is generated by the content handler 140 or before the content handler 140 provides the generated prompt to the generative transformer model 165 to generate the content item 180. In response to determining that the previously generated content item 180 is not stored in the database 135, the content controller 145 may call the generative transformer model 165 to generate the content item 180 using the prompt 230 created based on the input 225. Calling the generative transformer model 165 may include the content controller 145 providing instructions to the content handler 140 to generate the prompt 230 or providing the generated prompt 230 to the generative transformer model 165 to generate the content item 180.

[0108] The content controller 145 may rank the identified content items 180 according to one or more criteria. The content controller 145 may rank the identified content items 180 when the content items 180 are identified, or the content controller 145 may rank the identified content items 180 before providing the content items 180 to the administrator device 190. The one or more criteria used by the content controller 145 to rank the content items 180 may include feedback from the audience 215 to previous digital therapeutic content. For example, the data processing system 105 may receive feedback from the audience 215 about content items previously presented to the audience 215. The content controller 145 may rank the generated content items 180 based on the feedback from the audience 215. The feedback may include a numerical rating of the content item 180, text feedback describing an opinion of the content item 180, or a binary indicator of the content item (such as "thumbs up" or "thumbs down").

[0109] The content controller 145 may sort the generated content items 180 based on criteria including audience preferences for previous digital therapeutic content. For example, the audience may indicate a preferred type of content item 180, such as a preferred format (video, text, audio), a preferred duration for presentation of the content item 180, a preferred style of the content item 180 (color, font type, paragraph format, etc.), etc. The criteria used by the content controller 145 to sort the content items 180 may include audience behavior in response to previous digital therapeutic content. For example, an end-user profile 170 indicated in or included in the audience 215 may indicate one or more content items 180 or types of content items 180 that obtain a higher response, engagement, or compliance from the audience 215 participating in the digital therapeutic program.

[0110] The content controller 145 may rank the content items 180 based on the reading levels of the content items 180. For example, the content controller 145 may determine the reading level of each of the identified content items 180, and may rank the content items 180 based on the reading level. The reading level may refer to the age, education, or literacy level of reading or understanding a text, such as a text-based content item 180. In some embodiments, the content controller 145 may rank the content items 180 based on the association of the reading levels of the content items 180 with the reading levels indicated by the audience 215. For example, if the content item 180A includes the same or similar reading level as the audience 215, the content controller 145 may rank the content item 180A higher or more favorably than the content item 180B.

[0111] The one or more criteria may include an identification of a corresponding model used to generate the content items 180. For example, each content item 180 may be generated by a corresponding generative transformer model 165. The content controller 145 may rank the content items 180 generated by the generative transformer models 165 according to which the corresponding content items 180 were generated. In some embodiments, a content item 180A generated by a generative transformer model 165A maintained, trained, or associated with the data processing system 105 may be ranked higher or more favorably than a content item 180B generated by a generative transformer model 165B that is not maintained or trained by the data processing system 105.

[0112] The risk assessor 150 may select, choose, or identify a risk model 185 from the database 135 to apply to the identified content item 180. The risk assessor 150 may identify the risk model 185 in response to receiving an input 225 from an application operating on the administrator device 190. The risk assessor 150 may retrieve the risk model 185 in response to the content handler 140 generating a prompt 230. The risk assessor 150 may access the database 135 at any time to retrieve a risk model 185 to apply to the identified content item 180.

[0113] The risk assessor 150 may select a model 185 based on one or more domains 310 associated with the input 225. The risk assessor 150 may identify the one or more domains 310 associated with the input 225 based on the domain identifier 220 of the input 225. In some embodiments, one or more risk models of the risk model 185 may correspond to a specific domain in the domains 310. For example, the first risk model 185A may correspond to the medical domain and the second risk model 185B may correspond to the audience experience domain. The risk assessor 150 may select one or more risk models 185 corresponding to the domain 310 of the input 225. The domain identifier 220 corresponding to the input 225 may refer to the domain identifier 220 of the input 225 that matches the domain on which the one or more risk models 185 were trained. The one or more risk models 185 may be trained for one or more specific domains 310 by having a training set provided to the one or more risk models 185, the training set being trained for the specific domain, the specific domain including phrasing, phrases, or content items of the specific domain.

[0114] The risk assessor 150 may select a plurality of models 185 corresponding to one or more domains 310. In some embodiments, the input 225 may identify a plurality of domains 310. For example, the input 225 may identify a plurality of domains 310, such as a product domain, a medical domain, and a regulatory domain. The risk assessor 150 may select a risk model 185 corresponding to each domain 310 identified in the input 225. For example, for an input 225 identifying a product domain, a medical domain, and a regulatory domain, the risk assessor 150 may identify a first risk model 185A corresponding to the product domain, a second risk model 185B corresponding to the medical domain, and a third risk model 185C corresponding to the regulatory domain. In some embodiments, the risk model 185 may correspond to a plurality of domains. For example, the risk model 185A may correspond to the medical domain and the scientific domain. The risk assessor 150 may select a risk model 185 corresponding to one or more of the domains identified by the domain identifier 220 of the input 225.

[0115] The risk assessor 150 may apply the selected risk model 185 to the identified content items 180 to generate one or more risk scores 305A-N (hereinafter collectively referred to as risk scores 305). In some embodiments, applying the risk model 185 to the content items 180 may include the risk assessor 150 providing the identified content items 180 as input to the model 185 to generate the risk score 305 associated with each content item. In some embodiments, a subset of the content items 180 may be applied to the risk model 185. For example, content items generated by the first generative transformer model 165A may not be provided to the first risk model 185A. In this way, the content items 180 generated by the generative transformer model 165 maintained or trained by the data processing system 105 may not be subject to one or more risk models 185.

[0116] The risk assessor 150 may provide the content item 180 to a plurality of risk models 185 corresponding to each domain identifier 220 of the input 225 to generate a corresponding risk score 305 according to each risk model 185 applied to the content item 180. In some embodiments, the risk assessor 150 may generate a risk score 305 for each domain 310 associated with the input 225. The risk score 305 may be a score identifying a risk level associated with the content item 180 or the input 225. In some embodiments, the risk score 305 may define a risk level relative to the domain 310. The risk score 305 may be similar to or include the risk score 235.

[0117] The risk score 305 may be used to determine whether to allow or restrict the content item 180 from being provided to the audience 215 (e.g., directly or by the administrator 205 via the application 120 operating the administrator device 190). In some embodiments, a first risk score 305A that is lower than, less than, or less than a second risk score 305B may indicate that the first risk score may be more appropriate for provision. In general, a lower risk score 305 may indicate a higher likelihood that the content item 180 generated based on the input 225 corresponds to the desired audience 215, domain, format type, or other parameters of the input 225. Generating the risk score 305 may include the risk assessor 150 applying the risk model 185 to determine the extent to which each content item 180 conforms to the domain 310.

[0118] Using the risk model 185, the risk assessor 150 may determine a risk score 305 to indicate the extent to which the information of the generated content item 180 satisfies the criteria of the domain 310 associated with the risk model 185. For example, the risk score 305 may indicate the conformity or satisfaction of a phrase, word, or portion of the criteria of a particular domain 310 associated with the risk model 185. For example, the regulatory domain may specify that "will" or "yes" should be avoided from being included in the efficacy. If the content contains such a statement (e.g., "this product will improve your condition"), the corresponding risk model 185 may output a high risk score 305 indicating that the criteria of the regulatory domain are not met.

[0119] The risk score 305 may correspond to the compliance or non-compliance of the content item 180 with the standards set for a particular domain. The compliance of the content item 180 may mean that the content item 180 is suitable for provision. Suitable for provision may include the content item 180 being above a threshold corresponding to a parameter identified in the input 225 (such as a domain or audience 215). In some embodiments, a risk score 305 below a threshold risk score may indicate that the content item 180 corresponding to the risk score 305 is compliant. In some embodiments, a risk score 305 greater than or equal to a threshold risk score may indicate that the content item 180 corresponding to the risk score 305 is non-compliant.

[0120] Reference now Figure 4 , which depicts a block diagram of a process 400 for selecting content items 180A to be sent in a message to an administrator device 190 in the system 100 for generating and adapting a targeted message. The process 400 may include or correspond to operations performed in the system 100 to generate and adapt content items 180 for a targeted message. Under the process 400, a content controller 145 executing on the data processing system 105 may select one or more content items 180 based on their corresponding risk scores 305. The content controller 145 may provide the one or more content items 180 to the content handler 140 for providing to the administrator device 190.

[0121] The content controller 145 may select one or more content items 180A based on the risk score 305 associated with the content item 180A. When the risk assessor 150 generates the risk score 305, the content controller 145 may select one or more content items 180A based on their respective risk scores. In some embodiments, the content controller 145 may select the content item 180A with the lowest risk score. The content controller 145 may select one or more content items 180A with a risk score below a threshold risk score, or the content controller 145 may select a content item 180A or a group of content items 180A with the lowest risk score. In some cases, the content controller 145 may select one or more content items 180A with a risk score at or above a threshold risk score or a group of content items 180A with the highest risk score. The highest or lowest risk score 305 may be determined as the highest or lowest risk score 305 in a group of risk scores 305 generated by the risk assessor 150 for the input 225, respectively.

[0122] The content controller 145 may select the content item 180A based on the ranking of the content items by the content controller 145. As described herein, the content controller 145 may rank the content items 180 based on various criteria, including, but not limited to, feedback from the audience to previous digital therapeutic content, audience preferences for previous digital therapeutic content, identification of the corresponding generative transformer model 165 used to generate the content item 180, or audience behavior in response to previous digital therapeutic content, etc. The content controller 145 may select the content item 180A with the most favorable or highest ranking ranked by the content controller 145. In some embodiments, the content controller 145 may select the content item 180A based on the highest ranked content item 180A of a set of content items having a risk score below a threshold risk score. For example, the content controller 145 may select the content item 180A with the highest likelihood of increasing the audience's compliance with the digital therapeutic regimen, which digital therapeutic regimen also has a risk score 305 below the threshold risk score.

[0123] Content handler 140 may provide selected content item 180A to administrator device 190. Content handler 140 may provide selected content item 180A in a message sent to administrator device 190. The message may include instructions for application 120 operating on administrator device 190 to present content item 180A on user interface 125. User interface 125 may present content item 180A according to the instructions in the message.

[0124] In order to send, the content handler 140 can generate at least one instruction for presenting the content item 180A to send to the administrator device 190. The instruction may include an identifier of the message or the content item 180A. In some embodiments, the instruction may be a code, a data packet, or a control for presenting the message to the administrator 205. The instruction may include a processing instruction for displaying the message on the application 120 through the UI element 130 of the user interface 125. The instruction may also include instructions for the administrator 205 to perform actions related to its session to generate and modify the content item. For example, the instruction may display a message including the content item 180A, instructing the end user to perform a specific activity associated with its session. In some embodiments, the instruction may be based on a messaging protocol, such as a short message service (SMS) or a multimedia messaging service (MMS). The instruction may identify the administrator device 190 to which the message is to be sent (e.g., using a phone number or a network address) and the content item 180A of the message in the payload. When generated, the content handler 140 may send the instruction to the administrator device 190 via the network 115.

[0125] Upon receipt, the administrator device 190 may render, display, or otherwise present the message via a display such as a user interface 125. In some embodiments, the application 120 on the administrator device 190 may render, display, or otherwise present the message via the user interface 125. For example, when presenting the message via the user interface 125 of the application 120, the instructions for the message may specify, identify, or define the layout (e.g., positioning, size, and color) for individual UI elements 130. In some embodiments, the application 120 on the administrator device 190 may provide a sandbox environment to simulate the presentation of the message to an end user. For example, when delivering the message according to a messaging protocol (e.g., SMS and MMS), the application 120 on the administrator device 190 may present a messaging application to mimic various SMS messaging applications to present the message.

[0126] In some embodiments, the message may include information related to the risk score 305 of the content item 180. For example, the message may identify the risk score 305 associated with each content item 180. The message may indicate whether the content item 180B corresponds to a risk score 305B that satisfies the risk threshold. The application 120 may display a list, table, or other presentation of the risk scores 305 associated with their respective content items. In some embodiments, the application 120 may display a list, map, or other presentation describing the generative transformer model 165 associated with each content item 180. For example, the application may display the associated risk score 305B and the associated generative transformer model 165B of the content item 180B. The application 120 may display information related to the risk score 305 via the user interface 125. For example, the application 120 may display a mapping between any of the content items 180 and their respective risk scores 305, generative transformer models 165, or domains 310 according to instructions.

[0127] In some embodiments, the message may include information related to the risk score 235 corresponding to the input 225. The message may include an indication that the risk score 235 corresponding to the input 225 exceeds a threshold risk score. For example, the indication may include that the risk score 235 associated with the input 225 exceeds the threshold risk score. The application 120 may display the indication as a list, table, or other presentation using the UI element 130 of the user interface 125. In some embodiments, the application 120 may display a list, map, or other presentation depicting parameters associated with the input 225. For example, the application may display the associated risk score 235, the associated domain identifier 220, or the associated audience 215, etc., with the input 225 as an indication.

[0128] Reference now FIG. 5A to FIG. 5C , which depicts a block diagram of a process 500 for updating a generative transformer model in a system 100 for adapting and generating content for a target message. The process 500 may include or correspond to operations performed in the system 100 to generate and adapt a content item 180 for a target message. Under the process 500, an application 120 on an administrator device 190 may receive an interaction 210 from an administrator 205. A feedback handler 155 may receive a response 515 generated from the administrator device 190. The feedback handler 155 may receive a response 515' generated from a presentation of a content item on an end-user device 110. The feedback handler 155 may provide feedback data 505 based on the response to the model trainer 160 to train the generative transformer model 165.

[0129] Reference now Figure 5A, by presenting content item 180A to administrator 205, administrator device 190 may monitor at least one interaction 210 of administrator 205 in response to presenting a message including at least one of content item 180A or information related to risk score 305 or 235. Interaction 210 may include data (e.g., text) input by administrator 205 in response to the message. In some embodiments, interaction 210 may include data identifying one or more portions of content item 180A to be maintained or modified. For example, administrator 205 may indicate selection of one or more portions of content item 180A through UI element 130 through interaction 210. The one or more portions may correspond to portions of content item 180A to be modified. In some embodiments, administrator 205 may provide modification of portions of content item 180A through interaction 210 with UI element 130. For example, by providing interaction 210, administrator 205 may edit (e.g., delete, add, format) text of content item 180A, edit the color or style of content item 180A, or edit other presentations of content item 180A on user interface 125.

[0130] Upon detecting interaction 210, administrator device 190 may output, generate, or otherwise produce at least one administrator response 515 for transmission to data processing system 105. Administrator response 515 may indicate, include, or otherwise identify the portion of content item 180A to be maintained or modified. For example, administrator response 515 may include an association of content item 180A to be modified with one or more portions of content item 180A to be modified. Administrator response 515 may include a time for response 515.

[0131] Reference now Figure 5B , the end user 505 may provide an interaction 510 to the presented content item. In some embodiments, the content item 180A may be presented to the end user 505 on the end user device 110. Upon approval by the administrator 205, the content item 180A may be stored in the database 135 to be provided to the end user 505. In some implementations, the end user device 110 may access the database 135 to retrieve, obtain, or access the content item 180A stored thereon. The end user device 110 may present the content item 180A to the end user 505 via the user interface 125 operating on the end user device 110. The end user device 110 may present the content item 180A as part of a digital therapy session to address a condition of the end user 505, present the content item 180A as public text (e.g., a publication or journal article), or present the content item 180A as an advertisement on a web browser or other application operating on the end user device 110, etc.

[0132] End-user device 110 may monitor interactions 510 with content item 180A on user interface 125. Interactions 510 may include some functionality of interactions 210 of administrator 205. For example, interactions 510 may be concurrent with the presentation of a message including content item 180A. For example, interactions 510 may correspond to an interaction of playing a video clip included in the message via a play button presented by an application operating on end-user device 110. In some embodiments, interactions 510 may be subsequent to presentation of the message on end-user device 110. For example, interactions 510 may correspond to a set of interactions that record end-user feedback related to content item 180A within the message after the message is presented to the end user.

[0133] Interaction 510 may also include data (e.g., text) input by terminal user 505 in response to a message. In some embodiments, interaction 510 may include data for identifying one or more parts of content item 180A to be modified. For example, terminal user 505 may indicate a selection of one or more content items 180A presented to terminal user 505 through interaction 510. Terminal user 505 may be prompted to provide feedback based on content item 180A, such as whether terminal user 505 enjoys the content or participates in an activity indicated by the content. In some cases, terminal user device 110 may monitor interaction 510 with one or more UI elements 130. For example, terminal user device 110 may monitor the time between presentation and interaction 510 of a message including content item 180A, compliance with a treatment regimen at the time of presentation of content item 180A, or subsequent activities performed by terminal user 505 related to presentation of content item 180A.

[0134] Upon detecting interaction 510, end-user device 110 may output, produce or generate at least one end-user response 515' for transmission to data processing system 105. End-user response 515' may indicate, include or otherwise identify a UI element 130 indicated in end-user interaction 510 to present a message. End-user response 515' may include a time for response 515', such as a time between presentation of a message including content item 180A and receipt of interaction 510. End-user response 515' may indicate other times associated with presentation of a message including content item 180A. For example, end-user response 515' may include a read time at which end-user 505 reads content item 180A.

[0135] In some embodiments, the application on the end-user device 110 can use the detected interaction 510 to generate an end-user response 515'. The response 515' can identify information such as an event associated with the interaction 510 of the end-user 505, a timestamp for presenting the message to the end-user 505, a timestamp for the event, and an identifier for the end-user 505. The end-user response 515' can also include data entered by the end-user 505 via the user interface 125 operated on the end-user device 110. In some embodiments, the application operating on the end-user device 110 can keep a timer to track the time elapsed since the message was presented to the end-user 505. The application can compare the elapsed time with the time limit of the message. When the elapsed time exceeds the time limit, the application can generate an end-user response 515' to indicate that the end-user does not interact with the message. Through this generation, the application or the end-user device 110 can provide, send or otherwise send the response 515' to the feedback processor 155.

[0136] Reference now Figure 5C , the feedback handler 155 may generate, create, or otherwise generate feedback data 505 based on the administrator response 515 and the end-user response 515′. The feedback data 505 may be used to train the generative transformer model 165. The feedback data 505 may include information derived from the administrator response 515. For example, the feedback data 505 may include an identification of one or more portions to be maintained or modified. The feedback data 505 may include information derived from the end-user response 515. The feedback data 505 may include an indication of whether the end-user interacted with the content item when it was presented.

[0137] In some embodiments, feedback handler 155 may generate feedback data 505 for subsequent generation of content items by generative transformer model 165. In some embodiments, feedback data 505 may identify or include information to be used as one or more parameters to define subsequent content items generated and presented to end user 505. When generated, feedback handler 155 may store and maintain an association between feedback data 505 and end user profile 170 on database 135. Feedback data 505 may indicate one or more portions to be modified in content item 180. When generated, feedback handler 155 may store and maintain an association of the one or more portions with content item 180A in database 135.

[0138] In some embodiments, the feedback handler 155 may generate feedback data 505 to include information for updating weights associated with the generative transformer model 165. The feedback data 505 may be generated to include information of the content item 180A of the message, the input 225, or the response 515 from the administrator 405. The feedback data 505 generated from the end-user response 515 may indicate the degree to which the presented message elicited interaction from the end-user 505. The number or duration of interactions from the end-user 505 with the content item 180A may be recorded and included in the feedback data 505. When generated, the feedback handler 155 may include the end-user interaction information and the content item 180A of the message in the feedback data 505.

[0139] The model trainer 160 may use the feedback data 505 to modify, adjust, or otherwise update the weights of the generative transformer model 165. The model trainer 160 may identify one or more of the generative transformer models 165 for updating based on the feedback data 505. The feedback data 505 may be aggregated across multiple responses 515 from an administrator 205 or from multiple administrators 205. For example, the content item 180A may be reviewed multiple times by various administrators or the same administrator 205. The feedback data 505 may also be aggregated across multiple responses 515′ from an end user 505 or from multiple end users 505. For example, the content item 180A may be presented to one or more end users 205 on one or more user devices 110 during each digital therapy session. In response to the presentation of the content item 180A, each end user 205 that presents the content item 180A may provide an interaction 510.

[0140] In general, the model trainer 160 can update the weights to trust the generation of messages with high performance metrics or low risk scores 305 or 235, and penalize message outputs with low performance metrics. The training or fine-tuning of the generative transformer model 165 using the feedback data 505 can be similar to the training or fine-tuning of the risk model 185 described herein. For training, the model trainer 160 can define, select or otherwise identify at least a portion of each feedback data 505 as training data, and identify at least a portion of each feedback data 505 as test data. The training data can be used to input to the generative transformer model 165 to generate outputs to be compared with the test data. Portions of each feedback data 505 can overlap at least partially and can correspond to a subset of the text strings within the feedback data 505.

[0141] The model trainer 160 may feed or apply a training data string from the feedback data 505 to the generative transformer model 165. In application, the model trainer 160 may process the input string according to a set of layers in the generative transformer model 165. As described above, the generative transformer model 165 may include a word segmentation layer, an input embedding layer, a position encoder, an encoder stack, a decoder stack, and an output layer, etc. The model trainer 160 may process the input string of the training data (a word or phrase in the form of alphanumeric characters) using the word segmentation layer of the generative transformer model 165 to generate a set of word vectors for the input set. Each word vector may be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a word embedding table). The model trainer 160 may apply the set of word vectors to the input embedding layer to generate a corresponding set of embeddings. The model trainer 160 may identify the position of each string within the set of strings of the source set. By identifying, the model trainer 160 may apply a position encoder to the position of each string to generate a position encoding corresponding to each embedding of the string, and expand the embedding.

[0142] The model trainer 160 can apply the set of embeddings together with a corresponding set of positional encodings generated from the input set of feedback data 505 to the encoder stack of the generative transformer model 165. In application, the model trainer 160 can process the set of embeddings and the corresponding set of positional encodings according to the layers (e.g., attention layers and feedforward layers) in each encoder in the encoder block. From this processing, the model trainer 160 can generate another set of embeddings to feed forward to the encoders in the encoder stack. The model trainer 160 can then feed the output of the encoder stack to the decoder stack.

[0143] In combination, the model trainer 160 can use a separate word segmentation layer of the generative transformer model 165 to process the data of the test data (e.g., text, images, audio, video, or multimedia content) to generate a set of word vectors for the test data. Each word vector can be a vector representation of at least one corresponding string in an n-dimensional feature space (e.g., using a word embedding table). The model trainer 160 can apply the set of word vectors to the input embedding layer to generate a corresponding set of embeddings. The model trainer 160 can identify the position of each string within a set of strings of the target set. By identification, the model trainer 160 can apply a position encoder to the position of each string to generate a position encoding of each embedding corresponding to the string, and expand the embedding.

[0144] The model trainer 160 can apply the set of embeddings and the corresponding set of positional encodings generated from the destination set of feedback data 505 to the decoder stack of the generative transformer model 165. The model trainer 160 can also combine the outputs of the encoder stack in the process through the decoder stack. In application, the model trainer 160 can process the set of embeddings and the corresponding set of positional encodings according to the layers in each decoder in the decoder block (e.g., attention layer, encoder-decoder attention layer, feedforward layer). The model trainer 160 can combine the output from the encoder with the input of the encoder-decoder attention layer in the decoder block. Based on this processing, the model trainer 160 can generate an output set of embeddings to be fed forward to the output layer.

[0145] Next, the model trainer 160 can feed the output from the decoder block into the output layer of the generator transformer layer 165. In the feed, the model trainer 160 can process the embedding from the decoder block according to the linear layer and the activation layer of the output layer. Through this processing, the model trainer 160 can calculate the probability of each embedding. The probability can represent the likelihood of the output of a given input word element. Based on the probability, the model trainer 160 can select an output word element (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability) to form, generate, or otherwise generate a feedback output. The feedback output may include text content, image content, audio content, video or multimedia content, etc., or any combination thereof.

[0146] By generation, the model trainer 160 can compare the output from the generative transformer model 165 with the feedback data 505 used to generate the feedback output. The comparison can be between the probability (or distribution) of various word-grams from the content of the output (e.g., words for the text output) and the probability of the word-grams in the feedback data 505. For example, the model trainer 160 can determine the difference between the probability distribution of the output and the feedback data 505 to be compared. The probability distribution can identify the probability of each candidate word-gram in the output or the word-gram in the feedback data 505. Based on the comparison, the model trainer 160 can calculate, determine or otherwise generate a loss metric. The loss metric can indicate the degree of deviation of the output from the expected output defined by the feedback data 505 used to generate the output. The loss metric can be calculated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean square error (MSE), quadratic loss, cross entropy loss, Huber loss, etc.

[0147] Using the loss metric, the model trainer 160 may update one or more weights in the set of layers of the generative transformer model 165. The update of the weights may be based on back propagation and an optimization function (sometimes referred to herein as an objective function) having one or more parameters (e.g., a learning rate, momentum, weight decay, and a number of iterations). The optimization function may define one or more parameters in which the weights of the generative transformer model 165 will be updated. The model trainer 160 may iteratively train the generative transformer model 165 until convergence. Upon convergence, the model trainer 160 may store and maintain a set of weights for the set of layers of the generative transformer model 165.

[0148] In this manner, the data processing system 105 may iteratively and continuously consider responses 515 from presentation of content items 180 to administrators 205 and end users 505 to improve generation of messages that conform to specific domains 310. Furthermore, the data processing system 105 may iteratively and continuously consider responses 515' from end users 505 to improve generation of new and effective content for subsequent messages within each domain 310. The incorporation of feedback data 505 may enable generation of new content items that are more targeted and relevant to the preferences of the audience 215, domain 310, and end users 505 than content generated by a general generative transformer model 165. In the context of digital therapeutics, newly generated content items may take into account changes in use, such as improvement or deterioration of an end user's condition or progression through a treatment regimen. By using a generative transformer model 165, content may be flexibly generated for end users, and personalization of content may be extended to large audiences. The implementation of flexibility, scalability, and specificity can optimize or reduce the consumption of computing resources (e.g., processors and memory) and network bandwidth that would otherwise be wasted by providing ineffective content. From a human-computer interaction (HCI) perspective, the content generated by utilizing the generative transformer model 165 can produce higher quality interactions by end users with the end user device 110. In addition, the increase in engagement can lead to a higher level of compliance with the treatment regimen by the end user. Higher compliance can also have more possibilities in preventing, alleviating, or treating the end user's condition.

[0149] Reference now Figure 6, which depicts a flow diagram of the architecture of a system 600 for generating content items in a system 100 for generating and adjusting content for a targeted message. The system 600 may include or correspond to operations performed in the system 100 for generating and adjusting content items 180 for a targeted message. The system 600 may include a moderator to accept input from an internal administrator, provide content items from a model or previously generated content, and receive any feedback from the administrator when generating, and receive any feedback from the end user when presenting. Once the input is received, the moderator may forward the input to a guardrail. The input may include text data entered by the administrator to create or search for digital therapeutic content. The guardrail may use one or more natural language processing (NLP) models and risk models to assess the risk of the input. If the input is determined to be high risk (e.g., a risk score above a threshold), the guardrail may return an indication that the input has a high risk to the moderator for presentation to the administrator. On the other hand, if the input is determined to be low risk (e.g., a risk score below a threshold), the guardrail may pass the input to the generator.

[0150] By receiving input, the generator can use NLP techniques (e.g., automatic summarization) to search for previously generated digital therapeutic content on the database. The database can store and maintain content across multiple domains, such as audience experience, regulatory, compliance, scientific or medical, and product domains. In combination, the generator can pass input in the form of prompts to a set of generative models. Each generative model can be a generative transformer model and can use the prompts to generate new digital therapeutic content. The generator can aggregate content items found in the database and output by the generative model, and provide the content items to the guardrail. The guardrail can evaluate each content item in turn using the risk model. For each content item, the guardrail can determine a risk score indicating the degree of risk or compliance with a specific domain of the risk model. Using the risk score, the guardrail can select a subset of content items to be provided to the arbitrator.

[0151] The arbitrator may use a ranking algorithm to rank the content items based on other information such as end-user profile information and domains. Based on the ranking, the arbitrator may select one or more content items to be presented to the end-user. The content items may be presented to an administrator to enter revisions. The content items may also be provided to the end-user to measure the audience's response to the information contained therein. The arbitrator may receive feedback from the end-user identifying audience responses or administrator modifications, etc. The arbitrator may provide feedback to the trainer, and the trainer may in turn use the data in the feedback to improve the generative model and the content items stored and maintained on the database.

[0152] Reference now Figure 7, which depicts a method 700 for generating and adjusting the content of a target message. The method 700 may be implemented or performed using any of the components detailed herein, such as the data processing system 105, the administrator device 190, the end-user device 110, and the database 135. Under the method 700, a computing system (e.g., the data processing system 105 or the administrator device 190 or both) may receive a text input (e.g., the input 225) (705). The computing system may determine a risk score for the input (e.g., the risk score 235) (710). The computing system may determine whether the risk score satisfies a threshold (715). In response to determining that the risk score does not satisfy the threshold, the computing system may terminate processing (720). In response to determining that the risk score satisfies the threshold, the computing system may search for a content item (e.g., the content item 180) for the text input (725). The computing system may determine whether a content item corresponding to the text input exists in a database (e.g., the database 135) (730). In response to determining that there are no content items for the text input, the computing system may apply the text input to a generative model (e.g., generative transformer model 165) (735). In response to determining that there are one or more content items for the text input or in response to applying the text input to the generative model, the computing system may identify the content item (740). The computing system may determine a risk score for the output (e.g., risk score 305) (745). The computing system may determine whether the risk score satisfies a threshold (750). If the risk score does not satisfy the threshold, the computing system may exclude the content item (755). In response to the risk score satisfying the threshold, the computing system may select the content item (760). The computing system may provide the content item (765).

[0153] B. Systems and methods for training and applying validation models for content

[0154] Reference now Figure 8, which depicts a block diagram of a process 800 for training a validation model and applying the validation model to content. In overview, the system 800 may include at least one data validation system 805, an administrator device 810, model services 865A-865n (hereinafter collectively referred to as model services 865), a database 820, and an end-user device 825 that are communicatively coupled to each other via at least one network 815. The model service 865 may include at least one generative model 855. The data validation system 805 may include at least one model trainer 830, at least one model applier 835, at least one feedback handler 840, at least one policy enforcer 845, and at least one validation model 850, among others. The data validation system 805 may include at least one database 820 or may have access to at least one database 820. The database 820 may store, maintain, or otherwise include one or more content items 860A-N (hereinafter collectively referred to as content items 860), among others. Within the data validation system 805, the model trainer 830 may train the validation model 850. Model applicator 835 may apply validation model 850 to generate an output data set. Feedback handler 840 may receive feedback from administrator device 810. Policy enforcer 845 may provide content items to end users. In some embodiments, data validation system 805 may be part of data processing system 105 as described in detail herein.

[0155] In more detail, the data verification system 805 can be any computing device including one or more processors coupled with memory and software and capable of performing the various processes and tasks described herein. The data verification system 805 can communicate with the end-user device 825, the administrator device 810, the model service 865, and the database 820 via the network 815. The data verification system 805 can be located, located, or otherwise associated with at least one computer system. The computer system can correspond to a data center, a branch office, or a site, and one or more computers corresponding to the data verification system 805 are arranged on the data center, the branch office, or the site.

[0156] The end-user device 825 may be any computing device including one or more processors coupled to memory and software and capable of performing the various processes and tasks described herein. The end-user device 825 may be similar to or include the end-user device 110. The end-user device 825 may communicate with the data verification system 805, the administrator device 810, the model service 865, and the database 820 via the network 815. The end-user device 825 may be a smart phone, other mobile phones, tablet computers, wearable computing devices (e.g., smart watches, glasses), or laptop computers. The end-user device 825 may be provided with one or more content items 860 via the data verification system 805, or the end-user device 825 may request one or more content items 860 via the interaction 805 with the data verification system, such as via an application associated with the user device 825.

[0157] Administrator device 810 (sometimes referred to herein as an administrator device) can be any computing device that includes one or more processors coupled with memory and software and is capable of performing the various processes and tasks described herein. Administrator device 810 can be similar to or include administrator device 190. Administrator device 810 can communicate with data validation system 805, end-user device 825, model service 865, and database 820 via network 815. Administrator device 810 can be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses), or laptop computer. Administrator device 810 can be used to access applications.

[0158] The database 820 can store and maintain various resources and data associated with the data verification system 805 and the model service 865. The database 820 can include a database management system (DBMS) to arrange and organize the data maintained thereon, such as content items 860A-N, etc. The database 820 can communicate with the data verification system 805, the administrator device 810, the end-user device 825, and the model service 865 via the network 815. When running various operations, the data verification system 805, the model service 865, and the administrator device 810 can access the database 820 to retrieve identified data therefrom. The data verification system 805, the model service 865, and the administrator device 810 can also write data to the database 820 from running such operations.

[0159] On the database 820, the content item 860 may be in any form, such as text, image, audio, video or multimedia content, etc., or any combination thereof. The content item 860 may be similar to or include the content item 180. The content item 860 may be stored and maintained in the database 820 using one or more files. For example, for text, the content item 860 may be stored as a text file (TXT), a rich text file (RTF), an extensible markup language (XML), and a hypertext markup language (HTTP), etc. For images, the content item 860 may be stored as a Joint Photographic Experts Group (JPEG) format, a Portable Network Graphics (PNG) format, a Graphics Interchange Format (GIF), or a Scalable Vector Graphics (SVG) format, etc. For audio, the content item 860 may be stored as a waveform audio file (WAV), a Moving Picture Experts Group format (e.g., MP3 and MP4), and an Ogg Vorbis (OGG) format, etc. For video, the content item 860 may be stored as a Movie Experts Group format (e.g., MP3 and MP4), a QuickTime movie (MOV), and a Windows Movie Video (WMV), etc. For multimedia content, content item 860 may be Audio Video Interleave (AVI), Movie Experts Group formats (eg, MP3 and MP4), QuickTime Movie (MOV), Windows Movie Video (WMV), and the like.

[0160] Each content item 860 may identify or include information to be presented via the end-user device 825 or the administrator device 810. For example, the content item 860 may be presented to the end-user or administrator via a message sent to the end-user device 825 or the administrator device 810, respectively. The message may be in any format, such as a short message / messaging service (SMS), a multimedia messaging service (MMS), or as instructions presented via a display associated with the user device 825 or the administrator device 810, etc.

[0161] Content item 860 of the message may include a reminder to perform the task of the session. The message may be derived from a pre-generated psychotherapy message library or a pre-generated engagement (reminder) message library. The message may include a reminder for the end user to complete the therapy session, take medication, or complete the task of the regimen. The message may include an activity for the end user to perform or a course for the end user to participate in. Content item 860 may also include a mechanism for responding, such as a link, chat box, or instructions for responding to the message.

[0162] Content item 860 may include or correspond to one or more texts, such as articles, summaries, or publications. For example, content item 860 may include a research article, a review article, a case report, a clinical trial protocol, or an editorial, etc. Content item 860 may include text submitted to a government agency, a subject matter expert, a scientific journal, or a conference, etc. For example, content item 860 may include a clinical trial protocol related to a treatment provided for an end-user's condition, for submission to the Food and Drug Administration (FDA), a medical journal, or for internal distribution.

[0163] End-user conditions may include, for example, chronic pain (e.g., related to or including arthritis, migraines, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive stress injury, irritable bowel syndrome, inflammatory bowel disease, and cancer pain), skin diseases (e.g., atopic dermatitis, psoriasis, dermatitis, and eczema), cognitive disorders (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis, and schizophrenia), mental health conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, borderline personality disorder, and attention-deficit / hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, tobacco use disorder, or hallucinogenic disorder), and other conditions (e.g., narcolepsy and oncology or cancer).

[0164] When provided with content item 860 generated during the session to generate or modify content item 860, the end user may be taking medication at least in part at the same time to address the condition. For example, if the medication is for pain, the end user may be taking acetaminophen, a nonsteroidal anti-inflammatory composition, an antidepressant, an anticonvulsant, or other compositions, etc. For skin diseases, the end user may be taking steroids, antihistamines, or a topical preservative, etc. For cognitive disorders, the end user may be taking a cholinesterase inhibitor or memantine, etc. For narcolepsy, the end user may be taking a stimulant or an antidepressant, etc. The end user may also participate in other psychotherapy for these conditions.

[0165] The content items 860 may be manually created, computer generated, or a combination thereof. In some embodiments, a person may provide content items 860 via an administrator device 810. For example, the administrator device 810 may upload, provide, or transmit one or more content items 860 for storage in the database 820. The content items 860 may be computer generated, such as generated by a generative model 855. In some embodiments, the administrator device 810 may provide input to a model service 865 using a generative model 855 to create one or more content items 860. For example, the administrator device 810 may provide text, images, videos, or other presentations as input to generate content items 860. The model service 865 may generate one or more content items 860 based on prompts created using a generative model 855 by input from the administrator device 810.

[0166] Model service 865 can be or include any computing device capable of hosting one or more generative models 855 to generate content items 860. Model service 865 can be a local service of system 800, or can be remote from system 800 and accessed by data validation system 805 via network 815. In some embodiments, data validation system 805 can maintain one or more model services 865. In some embodiments, data validation system 805 can access remote model services 865 to provide input to generate one or more content items 860. Model service 865 can maintain one or more generative models 855.

[0167] Generative model 855 (sometimes referred to herein as generative transformer model 855) can receive input in the form of a set of strings (e.g., from text input) and output content (e.g., content item 860) in the form of one or more patterns (e.g., in the form of text strings, audio content, images, videos, or multimedia content). Generative model 855 can be similar to or include generative transformer model 165. Generative model 855 can be a machine learning model based on a transformer model (e.g., a generative pre-trained model or a bidirectional encoder representation from a transformer). Generative transformer model 165 can be a large language model (LLM), a text-to-image model, a text-to-audio model, or a text-to-video model, etc. In some embodiments, generative model 855 can be part of data verification system 805, or data verification system 805 can include model service 865. In some embodiments, generative model 855 can be part of a server that is separate from data verification system 805 and communicates with data verification system 805 via network 815.

[0168] One or more generative models 855 may be trained and maintained by the data processing system 105. The generative model 855 may include a set of weights arranged across a set of layers according to a transformer architecture. Under this architecture, the generative model 855 may include at least one word segmentation layer (sometimes referred to herein as a word segmenter), at least one input embedding layer, at least one position encoder, at least one encoder stack, at least one decoder stack, and at least one output layer, wherein these layers are interconnected to each other (e.g., via forward, backward, or skip connections). In some embodiments, the generative transformer layer may lack an encoder stack (e.g., for an encoder-only architecture) or a decoder stack (e.g., for a decoder-only model architecture). The word segmentation layer may convert the original input in the form of a string set into a corresponding word vector set (also referred to herein as a word unit or vector) in an n-dimensional feature space. The input embedding layer may generate a set of embeddings using the set of word vectors. Each embedding may be a low-dimensional representation of a corresponding word vector, and may capture the semantic and grammatical information of a string associated with a word vector. The position encoder may generate a position encoding for each input embedding according to the position of the corresponding word vector or by extending the string within the input string set.

[0169] Next, in the generative model 855, the encoder stack may include a set of encoders. Each encoder may include at least one attention layer and at least one feedforward layer, etc. The attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each input embedding to indicate the degree to which the embedding is to focus attention, and generate a weighted sum of the input embedding set. The feedforward layer may apply a linear transformation (e.g., a reconciled linear unit (ReLU)) with nonlinear activation to the output of the attention layer. The output may be fed to another encoder in the encoder stack in the generative transformer layer. When the encoder is a terminal encoder in the encoder stack, the output may be fed to the decoder stack.

[0170] The decoder stack may include at least one attention layer, at least one encoder-decoder attention layer, and at least one feedforward layer, among others. In the decoder stack, an attention layer (e.g., a multi-head self-attention layer) may calculate an attention score for each output embedding (e.g., an embedding generated from a target or expected output). The encoder-decoder attention layer may combine inputs from an attention layer within the decoder stack and outputs from one of the encoders in the encoder stack, and may calculate an attention score from the combined inputs. The feedforward layer may apply a linear transformation (e.g., a reconciled linear unit (ReLU)) with a nonlinear activation to the output of the encoder-decoder attention layer. The output of the decoder may be fed to another decoder within the decoder stack. When the decoder is a terminal decoder in the decoder stack, the output may be fed to the output layer.

[0171] The output layer of the generative model 855 may include at least one linear layer and at least one activation layer, etc. The linear layer may be a fully connected layer that performs a linear transformation on the output from the decoder stack to calculate the word-unit score. The activation layer may apply an activation function (e.g., Softmax, sigmoid, or adjusted linear unit) to the output of the linear function to convert the word-unit score into a probability (or distribution). The probability may represent the likelihood of the output word-unit occurring for a given input word-unit. The output layer may use the probability to select the output word-unit (e.g., at least a portion of the output text, image, audio, video, or multimedia content with the highest probability). Repeating this operation on a set of input tags, the resulting set of output tags may be used to form the output of the entire generative model 855. Although this article is mainly described in terms of a transformer model, the model service 865 may use other machine learning models to generate and output content. In some implementations, the model service 865 may use one or more models maintained by an external system to generate and output content. For example, a data processing system may use one or more models such as ChatGPT produced by OpenAI, BARD produced by Google, or LLaMA produced by Meta to generate content.

[0172] Each generative model 855 can produce one or more of content items 860 based on the prompts provided to the generative model 855. Each content item 860 produced according to the prompts created from the text input provided to the data verification system 805 or the model service 865 can be different due to the differences in each generative model 855. Like this, content item 860A can be more suitable for providing to the terminal user or administrator than other content items 860B-N. For example, the content item 860 generated by the generative model 855 can include inaccurate, irrelevant content or illusion. In certain embodiments, the content item 860A generated by the generative model 855 may be irrelevant to a specific terminal user due to the information in the content item 860A, the disease solved by the content item 860A, the presentation style of the content item 860A or the large assertion provided by the content item 860A. For example, the content item 860A can assert that it is the "best" treatment method for a given condition; However, this cannot be asserted and provides wrong information. For example, content item 860A may recommend to the end user that a meat-based dish be consumed without recognizing that the user was previously indicated as a vegetarian. For example, content item 860A may be a text presentation style, although the end user's previous behavior from a previous session indicates that the end user more consistently adheres to the session when video content is presented. For example, content item 860A may generate data that is not verified or proven to be true. In order to adjust the content item 860 generated by the generative model 855, the data verification system 805 may train and apply one or more verification models 850 to the content item 860 to determine the compliance of the content item 860.

[0173] Each verification model 850 can be a machine learning model trained to determine whether a content item 860 is compliant or non-compliant with a set of standards. The verification model 850 can be trained to calculate a risk score corresponding to a compliant or non-compliant content item 860. Compliance can refer to or include a content item 860 that is below a threshold risk score, and thus can be provided. Compliance can refer to or include a content item 860 that is above a threshold accuracy, or corresponds to a standard for a domain, an audience, or a combination thereof as described herein. Therefore, non-compliance can refer to or include a content item 860 that is at or above a threshold risk score, and thus can not be provided by the data verification system 805. The determination of compliance or non-compliance of each content item 860 by the data verification system 805 can also be used to continuously train the generative transformer model 165 to provide more accurate or less risky content items 860 over time. The verification model 850 can include one or more natural language models, including the generative transformer model 165 or the generative model 855 described herein. The validation model 850 may include one or more classifier models, such as a naive Bayes classifier, a support vector machine ensemble classifier, a kernel approximation, a k-nearest neighbor classifier, or a decision tree.

[0174] One or more validation models 850 may accept content items 860 as input. By accepting content items 860 as input, one or more validation models 850 may generate a risk score corresponding to a degree of compliance, or one or more validation models 850 may generate an indication of compliance or non-compliance. The degree of compliance or the indication of compliance may be associated with the likelihood that the content item 860 is a desired content item 860. The desired content item 860 may include content items 860 that have a desired accuracy (e.g., correct information, a relevant data set) or have a desired relevance (e.g., to a user receiving the content item 860 as part of a digital therapy session or an administrator receiving text of a desired article type), are specified for a group of people or audiences, are specified for a domain, and the like.

[0175] Reference now Fig. 9 , which depicts a block diagram of a process 900 for training a validation model using a training data set in a system 800 to train the validation model to content. The process 900 may include or correspond to operations performed in the system 800 to train and apply the validation model to content. Under the process 800, a model trainer 830 may use a training data set 905 to train a validation model 850. A model applier 835 may apply content items of the training data set 905 to determine a loss 930 of the validation model 850 to iteratively train the validation model 850.

[0176] Model trainer 830 executing on data validation system 805 may retrieve, obtain, or otherwise receive training data set 905 from database 820. Training data set 905 may include a set of examples for training validation model 850. Training data set may include content item 910. Content item 910 may be similar to content item 860 or content item 180. Content item 910 may be an example content item input by administrator device 810 to train validation model 850. For example, content item 910 may include text content, visual content, tactile content, auditory content, or a combination thereof to provide to a user device.

[0177] The content item 910 may be associated with an indication 915. The indication 915 may indicate that the content item 910 meets or does not meet a set of criteria. The content item 910 may correspond to the indication 915, so that the indication 915 may define whether the content item 910 is compliant or non-compliant, as described herein. In some embodiments, the administrator device 810 may assign the indication 915 to the content item 910 of the training data set 905. In this way, the content item 910 may be marked as compliant or non-compliant by the indication 915.

[0178] In some embodiments, the indication 915 may correspond to or include a risk score, such as the risk score 305 described herein. The risk score may identify the degree of compliance or noncompliance with a set of prescribed criteria for the provision. For example, the risk score may identify a level, amount, or score corresponding to the degree of expectation (e.g., how accurate, how risky, how relevant, etc.) for the user-provided content item 1010 associated with the end-user device 110. In some embodiments, the administrator device 810 may assign a risk score as part of the assignment indication 915.

[0179] In some cases, content item 910 may include an identification of a domain and a corresponding standard for the domain. Indication 915 associated with content item 910 of training data set 905 may include a domain identifier, such as domain identifier 220, to identify a domain (e.g., domain 310) associated with content item 910. Content item 910 may be associated with, correspond to, or be identified by a domain such as the domains described herein. For example, content item 910 may be associated with indication 915 identifying a product domain, an audience experience domain, a medical domain, a scientific domain, or a regulatory domain, etc.

[0180] When training a model, the model trainer 830 may access the database 820 to retrieve, obtain, or otherwise identify a training data set 905 for training the validation model 850. In some embodiments, the model trainer 830 may retrieve or identify an example of a training data set 905 that includes content items 910 stored in the database 820. In some embodiments, the model trainer 830 may access the database 820 to retrieve or identify a set of responses (or feedback data derived therefrom) by an end user to a previously provided message or content item. Each response may define or identify an indication 915 as an activity performed by the end user in response to the presentation of the corresponding message. For example, the indication 915 may include a response that may include whether the end user performed a specified activity, an indication of a favorable reaction to the message, one or more interactions in response to the presentation of the message, and a timestamp identifying the execution of the response, etc.

[0181] In some embodiments, the model trainer 830 may initialize the validation model 850. For example, the model trainer 830 may instantiate the validation model 850 by assigning random values ​​to the weights of the validation model 850 within the layers of the validation model 850. In some embodiments, the model trainer 830 may fine-tune a pre-trained machine learning model using a training data set 905. For training or fine-tuning, the model trainer 830 may define, select, or identify a training data set 905 from the database 820. The training data set 905 may be used to input to the validation model 850 to generate an output data set 925. The output data set 925 may include an indication 915' for comparison with the indication 915. The content items 910 may overlap at least partially and may correspond to a subset of the text strings within the training data set 905. For example, when the content items 910 contain text from a message related to a specific condition, the training data set 905 may correspond to a text description of the condition, and the output data set 925 may correspond to a text string of an activity to be performed, a psychoeducational course to be participated in, a reminder to take medication, or a notification to perform a specific activity, etc. The training dataset 905 corresponding to the input dataset 925 and the output dataset 925 may lack overlap. For example, when the training dataset 905 contains an association between text and an image, the content item 910 used as input may correspond to the text, and the output dataset 925 used as the destination may correspond to an image associated with the text.

[0182] The model applicator 835 may apply the content item 910 to the validation model 850. For example, the model applicator 835 may feed or apply the content item 910 into the validation model 850. In application, the model applicator 835 may process the content item 910 to generate an output data set 925 including the indication 915'. In some embodiments, the model applicator 835 may select a validation model 850 to apply the content item 910 based on a domain associated with the content item 910. For example, the model applicator 835 may apply a content item associated with a medical domain to a first validation model and a content item associated with an adjustment domain to a second validation model. In this way, a set of validation models 850 may be trained for a specific domain.

[0183] The validation model 850 may output an output data set 925 including an indication 915'. In some embodiments, the model applier 835 may determine the indication 915' as one of compliance or non-compliance based on the output data set 925. The indication 915' may be generated by the validation model 850 based on the content item 910. In some embodiments, the model trainer 830 may determine the indication 915' regarding the domain indicated in the content item 915. For example, the indication 915' may correspond to compliance within a particular domain. The indication 915' may indicate whether the content item 910 corresponds to the domain to which the validation model 850 corresponds.

[0184] In some embodiments, the output data set 925 may include portions 920A-N (hereinafter collectively referred to as portions 920). Portions 920 may include sub-portions of the content item 910 to be modified or identified as sub-portions to be modified. Portions 920 may include indications of sub-portions to be modified. The verification model 850 may determine portions 920 using the output data set 925 in response to determining that the indication 915' corresponds to a non-conformity or a degree of less than conformity. Portions 920 may indicate portions of text, images, words, phrases, or other components of the content item 910 that are desired to be edited. In some embodiments, the administrator device 810 may edit portions 920 of the output data set 925. In some embodiments, the administrator device 810 may provide portions 920 within the output data set 925 when generating the output data set 925. In this way, the verification model 850 may accept edits to portions 920 or selections 920 of the portions themselves in further iterative training 850 of the verification model.

[0185] By determining the indication 915', the model trainer 830 can compare the indication 915' with the indication 915. The probabilities (or distributions) of various tokens (e.g., words for text output) of the content item 910 from the output data set 925 can be compared with the probabilities of the tokens in the content item 910. For example, the model trainer 830 can determine the difference between the probability distribution of the indication 915' and the indication 915 to be compared. Based on the comparison, the model trainer 830 can calculate, determine, or otherwise generate a loss 930. The loss 930 can indicate the degree of deviation of the indication 915' from the indication 915. The loss 930 can be calculated according to any number of loss functions, such as norm loss (e.g., L1 or L2), mean squared error, quadratic loss, cross entropy loss, Huber loss function, etc.

[0186] In some embodiments, the model trainer 830 may determine the loss 930 of the output data set 925 based on the data retrieved from the database 820. When determined, the model trainer 830 may compare the output data set 925 with the content item 910 to calculate similarity. The similarity may measure, correspond to, or indicate, for example, semantic similarity (e.g., using knowledge mapping when comparing the text of the content item 910 and the output data set 925), visual similarity (e.g., using pixel-to-pixel value comparison when comparing an image or frame of a video of the content item 910 and the output data set 925), or audio similarity (e.g., using correlation or cosine similarity metrics between the audio of the content item 910 and the output data set 925). The loss 930 may be a function of the similarity, domain, or response of the content item 910 compared to the output data set 925. In general, the higher the loss 930, the more the generated output data set 925 may deviate from the preferences established by a given administrator device 810 in the design of the indication 915. Conversely, the lower the loss 930, the less likely the generated output data set 925 is to deviate from and be consistent with the indication 915 established by the administrator device 810. The loss 930 can be calculated to train the validation model 850 to generate a risk score or compliance indication for the indication 915 for messages with a higher probability of user engagement, content items corresponding to a particular domain or audience, and the like.

[0187] Using the loss 930, the model trainer 830 can update one or more weights in the set of layers of the validation model 850. The update of the weights can be based on back propagation and an optimization function (sometimes referred to herein as an objective function) having one or more parameters (e.g., a learning rate, momentum, weight decay, and number of iterations). The optimization function can define one or more parameters that will update the weights of the validation model 850. The optimization function can be based on stochastic gradient descent and can include, for example, adaptive moment estimation (ADAM), implicit update (ISGD), and adaptive gradient algorithm (AdaGrad), etc. The model trainer 830 and the model applier 835 can iteratively train and apply the validation model 850 until convergence. Upon convergence, the model trainer 830 can store and maintain a set of weights for the set of layers of the validation model 850.

[0188] Reference now Fig.10 , which depicts a block diagram of a process 1000 that incorporates feedback of a validation model into a system for training and applying a validation model to content. Process 1000 may include or correspond to operations performed in system 800 to train and apply a validation model to content. Under process 1000, model applier 835 may apply content items 1010A-N (hereinafter collectively referred to as content items 1010) to trained validation model 850. Validation model 850 may generate output data sets 1025A-N (hereinafter collectively referred to as output data sets 1025). Feedback handler 840 may present output data sets 1025 to administrator device 810 and may receive feedback data 1035 from administrator device 810.

[0189] When the loss 930 converges to a threshold level, the model applicator 835 may identify a content item 1010 from the database 820 or the model service 865. The content item 1010 may be a previously generated content item stored in the database 820. The content item 1010 may include a content item generated by one or more of the model services 865. The model applicator 835 may provide the identified content item 1010 as an input to the validation model 850. In some embodiments, the model applicator 835 may provide the content item 1010 based on a domain corresponding to the content item 1010 or a domain of each content item 1010. For example, the first content item 1010A may correspond to a product domain, and the second content item 1010B may correspond to an audience experience domain. The model applicator 835 may select a first validation model corresponding to the product domain to be applied to the first content item 1010A and a second validation model corresponding to the audience experience domain to be applied to the second content item 1010B.

[0190] By applying the validation model 850, the model applier 835 can generate at least one output data set 1025. The output data set 1025 can be similar to the output data set 925. The output data set 1025 can indicate whether a content item in the content items 1010 is compliant for provision. The output data set 1025 can include an indication 1015. The indication 1015 can indicate whether a particular content item in the content items 1010 is compliant or non-compliant, as described herein. When the content item 1010 is identified as compliant, the content item 1010 can be allowed to be provided (e.g., by the data validation system 805). On the other hand, when the content item 1010 is identified as non-compliant, the content item 1010 can be restricted (e.g., by the data validation system 805).

[0191] The validation model 850 may generate an indication 1015 based on the content item 1010 applied by the model applier 835. In some embodiments, the validation model 850 may generate an indication of one of compliance or non-compliance with respect to a domain identified in the content item 1010. For example, the validation model 850 may determine that the first content item 1010A is non-compliant (e.g., the indication 1015 indicates non-compliance) with respect to a particular domain, such as a medical or scientific domain, a regulatory domain, a product domain, or an audience experience domain, among others.

[0192] Output data set 1025 may include portions 1020A-N (hereinafter collectively referred to as portions 1020). Portion 1020 may be similar to or include portion 920. Portion 1020 may indicate a sub-portion of content item 1010 to be modified, or a sub-portion of content item 1010 that may not conform. In some embodiments, portion 1020 may indicate a sub-portion of content item 1010 that may be edited to make content item 1010 conform.

[0193] The feedback handler 840 may generate instructions to display the output data set 1025 to the administrator device 810 via the user interface 1030 of the administrator device 810. The user interface 1030 may be similar to or include the user interface 125, and may include user interface elements such as the UI element 130. The user interface 1030 may be any input / output device for displaying the output data set 1025 and accepting interactions from the administrator device 810 regarding the output data set 1025. The feedback handler 840 may generate instructions for displaying, presenting, or otherwise presenting the output data set 1025 to the administrator device 810 via the user interface 1030.

[0194] Feedback handler 840 may display output data set 1025 including indication 1015. In some embodiments, feedback handler 840 may display indication 1015 corresponding to each content item 1010. For example, feedback handler 840 may generate instructions for user interface 1030 to display each content item 1010 as corresponding to an indication of compliance, an indication of non-compliance, or a risk score indicating a degree of compliance. Feedback handler 840 may generate instructions to display portion 1020. In some embodiments, feedback handler 840 may generate instructions to display each content item in content items 1010 corresponding to its corresponding portion 1020. Feedback handler 840 may generate instructions to display portion 1020 in response to a determination that corresponding indication 1015 indicates that corresponding content item 1010 is non-compliant.

[0195] While displaying output data set 1025, feedback handler 840 may receive an interaction via user interface 1030. The interaction may be similar to interaction 210. The interaction may include actuating one or more user interface elements of user interface 1030. The interaction may indicate one or more of portions 1020 or indications 1015. Administrator device 810 may select one or more of portions 1020 from the display on user interface 1030. The interaction may indicate one or more portions 1020 to be edited. For example, the interaction may indicate portion 1020A of content item 1010, or the interaction may deselect or de-indicate portion 1020B determined by validation model 850.

[0196] Interaction can include editing one or more parts, such as editing other such attributes of text, display, image, audio or content item 1010. For example, interaction can be to change the volume, tone or duration of part 1020A of audio content item 1010. For example, interaction can be to change the color, size, image or display duration of video or image content item 1010B. For example, interaction can be to change the string, word, format, style or font of text content item 1010C.

[0197] The interaction may include a modification of the indication 1015. The administrator device 810 may change, modify, or overwrite the indication 1015 of the output data set 1025. In some embodiments, the administrator device 810 may determine that the indication 1015 assigned by the validation model 850 is erroneous, incorrect, or (in the case of a risk score indicating a degree of compliance) an erroneous value. The administrator device 810 may provide different indications via interactions that identify the content item 1010 as compliant or non-compliant. For example, the validation model 850 may determine that the content item 1010 is compliant. The administrator device 810 may view the indication 1025 and determine that the content item 1010 is actually non-compliant. The administrator may overwrite the indication 1015 with a different indication via a user interface 1030 associated with the administrator device 810 to determine the indication 1015.

[0198] The administrator device 810 may send the interaction to the feedback handler 840. The feedback handler 840 may generate feedback data 1035 from the interaction. The feedback data 1035 may include information related to the portion 1020. For example, the feedback data 1035 may include a portion selected by the administrator device 810, a portion edited by the administrator device 810, or an edit of the portion 1020 by the administrator device 810. The feedback data 1035 may include information related to the indication 1015 overridden by the administrator device 810. For example, the feedback data 1035 may include a confirmation of the indication 1015 generated by the validation model 850 of the content item 1010, a change to the indication 1015 generated by the validation model 850 of the content item 1010, an assignment of an indication 1015 or a risk score to the content item 1010, or a selection of a domain or multiple domains of the content item 1010.

[0199] The feedback handler 840 may provide the feedback data 1035 to the validation model 850 to retrain the validation model 850. In some embodiments, the validation model 850 may be continuously retrained by the model applier 835 and the model trainer 830 using the feedback data 1035 to further reduce the loss 930. The feedback data 1035 may be used to generate a new training data set that includes one or more of instructions from the administrator device 810, portions from the administrator device 810, portions edited by the administrator device 810, domains provided by the administrator device 810, etc. In this way, the validation model 850 may continue to learn and improve across different content items and domains.

[0200] Feedback handler 840 can determine the association between content item 1010 and indication 1015. Feedback handler 840 can store the association in database 820. The association between content item 1010 and indication 1015 can define content item 1010A as conforming and content item 1010B as non-conforming. Conforming content item 1010A can allow content item 1010A to be provided to the user via user equipment. Non-conforming content item 1010B can limit providing content item 1010B to the user. In some embodiments, at least one of the associations can include an association between text and content item 1010 in another modality (such as image, audio, video or multimedia content, etc.). The association between content item 1010 in text and other forms can come from a broad source. For example, a broad source association can be obtained from a large, predefined corpus of associations between identifiers and images. The association between content item 1010 in text and other modalities can come from a knowledge domain specific source. For example, the association may be taken from clinical studies, medical journals or web pages as well as text and other forms of content. Each content item of content items 1010 may be stored in database 820 in association with a data set as part of the data set.

[0201] Reference now Fig.11 , which depicts a block diagram of a process 1100 for providing content items to an administrator in a system to train and apply a validation model to the content. The process 1100 may include or correspond to operations performed in the system 800 to train and apply a validation model to content. Under the process 1100, the policy executor 845 may select content items 1010 to be provided to the administrator device 810. The administrator device 810 may present the content items 1010 on a user interface 1115 in a digital therapy session to address the condition of the end user.

[0202] When the association between the content items 1010 and the indications 1015 is stored, a policy executor 845 operating on the data verification system 805 may select one or more content items 1010 to provide to the administrator 1120. The policy executor 845 may select the one or more content items 1010 based on a profile associated with the administrator 1120. In some embodiments, the policy executor 845 may select the one or more content items 1010 based on preferences indicated by the profile, an audience 1120 indicated by the administrator, and the like. The policy executor 845 may select the content items 1010 corresponding to the indications 1015 that indicate compliance of the content items 1010. For example, the policy executor 845 may select for providing one or more content items 1010 that indicate compliance, or indicate a degree of compliance that is at or above a threshold degree of compliance. If the association includes an indication 1015 of non-compliance, the policy executor 845 may restrict providing the content item 1010. Conversely, if the association includes an indication of compliance 1015, the policy enforcer 845 may allow the content item 1010 for provision.

[0203] Upon selecting the content item 1010 based at least on the indication 1015, the policy enforcer 845 may provide the content item 1010 to the administrator device 810. The administrator device 810 may include an application 1110. The application 1110 operating on the administrator device 810 may be or include a testing digital therapeutic application to review content items provided to the end user in conjunction with a solution to address the end user's condition. The application 1110 operating on the administrator device 810 may generate instructions for displaying the content item 1010 on a user interface 1115 generated by the application 1110. The user interface 1115 may be similar to or include the user interface 1030, 125, such as including user interface elements and accepting interactions from an administrator 1120 of a device presenting the user interface.

[0204] By using the verification model 850, the data verification system 805 can control the provision of content items generated by the generative transformer model or created manually. This can enable content items to be automatically allowed or automatically restricted, thereby reducing the manual workload in reviewing the compliance of any violations of the content. The provision of content items can expand the functionality provided to the end user, thereby improving the utility of the end user device that provides digital treatment, and reducing the waste of computing resources (e.g., processing power and memory) that would otherwise be spent on invalid content. In the context of digital treatment, the control of content items can prevent incorrect or inappropriate information from being provided to the end user, thereby potentially improving the effectiveness of the digital treatment content. In addition, the increase in engagement can lead to a higher level of user compliance with the treatment plan. Higher compliance can in turn lead to a greater possibility in preventing, alleviating or treating the end user's illness.

[0205] Reference now FIG. 12A to FIG. 12C , which depicts an exemplary user interface of a system for training a validation model and applying the validation model to content. The exemplary user interface may be implemented or executed using any of the components detailed herein, such as a data validation system 805, an administrator device 810, and a database 820. Now referring to Fig. 12A , which depicts user interfaces 1200 and 1255 presenting an administrator overview for content item A. User interface 1200 can be presented in conjunction with the presentation of a message including content items as described herein. User interface 1200 can include information 1205 and user interface elements 1210 and 1215. Information 1205 can include information related to generated content items that have been input into one or more validation models as described herein. Information 1205 can include the source of the associated risk score of each domain of the content item, the audience of the content item, the content type of the content item, or the model item (e.g., which generative model or content item originates from which database). User interface 1200 can include UI element 1210 to view all generated content. When interacted by an administrator, user interface element 1210 can cause user interface 1200 to change to display a list of all content items generated from a prompt or retrieved from a database. When interacted by an administrator, user interface element 1215 can display content similar to content item A. For example, interaction with user interface element 1215 may cause user interface 1200 to display content items used to train a generative model that generated content item A.

[0206] As an alternative embodiment of user interface 1200, user interface 1255 can be presented in conjunction with the presentation of a message including content items as described herein. User interface 1255 can include information 1260 and user interface elements 1265 and 1270. Information 1260 can include information related to the generated content items that have been input into one or more validation models as described herein. Information 1260 can include an associated indication of the compliance of each domain of content items, the audience of content items, the content type of content items, or the source of model items (e.g., which generative model or content item originates from which database). User interface 1255 can include UI element 1265 to view all generated content. When interacted by an administrator, user interface element 1265 can cause user interface 1255 to change to display a list of all content items generated from a prompt or retrieved from a database. When interacted by an administrator, user interface element 1270 can display content similar to content item A. For example, interaction with user interface element 1270 can cause user interface 1255 to display content items for training the generative model that generates content item A.

[0207] Reference now Fig. 12B, which depicts a set of user interfaces 1220 and 1275. User interface 1220 can depict a possible risk score for each domain associated with a content item. For example, the modified possible risk score associated with the medical domain of a particular content item is 3. User interface 1220 can include user interface button 1255. When an administrator interacts with user interface button 1255, user interface 1220 can change to display user interface 1225, such as Fig. 12C As an alternative embodiment of user interface 1220, user interface 1275 can depict possible compliance indications for each domain associated with the content item. For example, a possible indication of compliance with a modification associated with the medical domain is compliance. User interface 1275 can include user interface button 1280. When the administrator interacts with user interface button 1280, user interface 1275 can change to display user interface 1225, such as Fig. 12C Depicted.

[0208] Reference now Fig. 12C , which depicts user interface 1225. User interface 1225 may display suggested modifications to obtain a possible risk score as depicted in user interface 1220 or a possible compliance indication as depicted in user interface 1275. User interface 1225 may display content item 1230. User interface 1225 may display portion 1235 of content item 1230 to be modified. User interface 1225 may include user interface button 1245 and user interface button 1250, among others. User interface 1225 may display alert 1240 regarding content item 1230. Alert 1240 may indicate that portion 1235 highlighted by alert 1240 may not comply with one or more domains. Portion 1235 may display suggested edits or modifications to content item 1230. User interface element 1245 may accept modifications presented by portion 1235. User interface button 1250 may enable an administrator to edit the modifications shown in portion 1235.

[0209] Reference now Fig.13, which depicts a method 1300 for training a validation model and applying the validation model to content. The method 1300 may be implemented or performed using any of the components detailed herein, such as the data validation system 805, the administrator device 810, and the database 820. Under the method 1300, a computing system (e.g., the data validation system 805 or the administrator device 810 or both) may identify a training data set (e.g., the training data set 905) (1305). The computing system may apply content (e.g., content item 910) on a model (e.g., the validation model 850) (1310). The computing system may determine an indication (e.g., indication 915') (1315). The computing system may determine a loss metric (e.g., loss 930). The computing system may determine whether it has received an input (1325). In response to receiving the input, the computing system may generate feedback (e.g., feedback data 1035) (1330). In response to not receiving the input, the computing system may determine a loss metric (1335). The computing system may update the weights of the model (1340).

[0210] Reference now Fig.14 , which depicts a method 1400 for providing a verified content item. The method 1400 may be implemented or performed using any of the components detailed herein, such as the data verification system 805, the administrator device 810, and the database 820. Under the method 1400, a computing system (e.g., the data verification system 805 or the administrator device 810 or both) may identify a content item (e.g., the content item 1010) (1405). The computing system may apply the content on a model (1410). The computing system may determine an indication (1415). The computing system may store the indication (1420). The computing system may determine whether the indication indicates compliance of the content (1425). In response to determining that the indication does not indicate compliance, the computing system may restrict provision (1430). In response to determining that the indication indicates compliance, the computing system may allow provision (1435). The computing system may provide the content item (1440).

[0211] C. Network and computing environment

[0212] Various operations described herein may be implemented on computer systems. Fig.15A simplified block diagram of a representative server system 1500, a client computer system 1514, and a network 1526 that can be used to implement certain embodiments of the present disclosure is shown. In various embodiments, the server system 1500 or a similar system can implement the services or servers described herein or portions thereof. The client computer system 1514 or a similar system can implement the clients described herein. The system 1500 described herein can be similar to the server system 1500. The server system 1500 can have a modular design including multiple modules 1502 (e.g., blades in a blade server embodiment); although two modules 1502 are shown, any number can be provided. Each module 1502 can include one or more processing units 1504 and local storage 1506.

[0213] One or more processing units 1504 may include a single processor, which may have one or more cores or multiple processors. In some embodiments, one or more processing units 1504 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processor, a digital signal processor. In some embodiments, some or all of the processing units 1504 may be implemented using custom circuits, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such an integrated circuit executes instructions stored on the circuit itself. In other embodiments, one or more processing units 1504 may execute instructions stored in local storage 1506. Any type of processor in any combination may be included in one or more processing units 1504.

[0214] The local storage 1506 may include volatile storage media (e.g., DRAM, SRAM, SDRAM, etc.) and / or non-volatile storage media (e.g., magnetic or optical disks, flash memory, etc.). The storage media incorporated in the local storage 1506 may be fixed, removable, or upgradeable as needed. The local storage 1506 may be physically or logically divided into various subunits, such as system memory, read-only memory (ROM), and permanent storage devices. The system memory may be a read-write memory device or a volatile read-write memory, such as a dynamic random access memory. The system memory may store some or all of the instructions and data required by one or more processing units 1504 at runtime. The read-only memory may store static data and instructions required by one or more processing units 1504. The permanent storage device may be a non-volatile read-write memory device that may store instructions and data even when the module 1502 is powered off. The term "storage media" as used herein includes any media in which data can be stored indefinitely (subject to coverage, electrical interference, power loss, etc.) and excludes carrier waves and transient electronic signals that propagate wirelessly or through wired connections.

[0215] In some embodiments, local storage 1506 may store one or more software programs to be executed by one or more processing units 1504, such as an operating system and / or programs that implement various server functions, such as functions of system 100, 800, or any other system described herein, or any other server associated with system 100, 800, or any other system described herein.

[0216] "Software" generally refers to a sequence of instructions that, when executed by one or more processing units 1504, causes the server system 1500 (or portions thereof) to perform various operations, thereby defining one or more specific machine embodiments that execute and perform the operations of the software program. Instructions may be stored as firmware residing in read-only memory and / or program code stored in non-volatile storage media, which may be read into volatile working memory for execution by one or more processing units 1504. Software may be implemented as a single program or a collection of independent programs or program modules that interact as needed. One or more processing units 1504 may retrieve program instructions to be executed and data to be processed from local storage 1506 (or non-local storage described below) to perform the various operations described above.

[0217] In some server systems 1500, multiple modules 1502 may be interconnected via a bus or other interconnect 1508 to form a local area network that supports communication between the modules 1502 and other components of the server system 1500. The interconnect 1508 may be implemented using a variety of technologies, including server racks, hubs, routers, etc.

[0218] Wide area network (WAN) interface 1510 can provide data communication capabilities between a local area network (e.g., via interconnect 1508) and a network 1526 (e.g., the Internet). Other technologies can be used to communicatively couple the server system to the network 1526, including wired (e.g., Ethernet, IEEE 802.3 standards) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standards).

[0219] In some embodiments, local storage 1506 is intended to provide working storage for one or more processing units 1504, providing fast access to programs and / or data to be processed while reducing traffic on interconnect 1508. Storage for larger amounts of data can be provided on a local area network through one or more mass storage subsystems 1512 that can be connected to interconnect 1508. Mass storage subsystem 1512 can be based on magnetic, optical, semiconductor or other data storage media. Direct attached storage, storage area networks, network attached storage, etc. can be used. Any data storage or other data collection described herein as being generated, used or maintained by a service or server can be stored in mass storage subsystem 1512. In some embodiments, additional data storage resources can be accessed via wide area network interface 1510 (potentially with increased latency).

[0220] The server system 1500 can operate in response to requests received via the WAN interface 1510. For example, one of the modules 1502 can implement monitoring functions and assign discrete tasks to other modules 1502 in response to received requests. Work distribution techniques can be used. When the request is processed, the result can be returned to the requester via the wide area network interface 1510. This operation can usually be automated. In addition, in some embodiments, the wide area network interface 1510 can connect multiple server systems 1500 to each other, providing an extensible system that can manage a large number of activities. Other technologies for managing server systems and server farms (a collection of collaborative server systems) can be used, including dynamic resource allocation and reallocation.

[0221] Server system 1500 can interact with various user-owned or user-operated devices via a wide area network such as the Internet. Examples of user-operated devices are shown in FIG. Fig.15 15 is shown as a client computing system 1514. The client computing system 1514 can be implemented as a consumer device such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses), desktop computer, laptop computer, etc.

[0222] For example, the client computing system 1514 may communicate via the wide area network interface 1510. The client computing system 1514 may include computer components such as one or more processing units 1516, storage 1518, a network interface 1520, a user input device 1522, and a user output device 1524. The client computing system 1514 may be a computing device implemented in a variety of form factors, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, other mobile computing devices, a wearable computing device, etc.

[0223] The processing unit 1516 and storage 1518 may be similar to the one or more processing units 1504 and local storage 1506 described above. Appropriate devices may be selected based on the requirements of the client computing system 1514; for example, the client computing system 1514 may be implemented as a "thin" client with limited processing capabilities or as a high-power computing device. The client computing system 1514 may be provided with program code executable by the processing unit 1516 to enable various interactions with the server system 1500.

[0224] The network interface 1520 may provide a connection to a network 1526, such as a wide area network (e.g., the Internet) to which the wide area network interface 1510 of the server system 1500 is also connected. In various embodiments, the network interface 1520 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various radio frequency data communication standards (e.g., Wi-Fi, Bluetooth, or a cellular data network standard (e.g., 3G, 4G, LTE, etc.)).

[0225] User input device 1522 may include any one or more devices via which a user may provide signals to client computing system 1514; client computing system 1514 may interpret the signals as indicating a user request or information. In various embodiments, user input device 1522 may include at least one of a keyboard, a touch pad, a touch screen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keyboard, a microphone, and the like.

[0226] User output device 1524 may include any device via which client computing system 1514 can provide information to a user. For example, user output device 1524 may include a display-to-display image generated by or delivered to client computing system 1514. The display may incorporate various image generation technologies, such as a liquid crystal display (LCD), a light emitting diode (LED) display including an organic light emitting diode (OLED), a projection system, a cathode ray tube (CRT), etc., and supporting electronic devices (e.g., digital-to-analog or analog-to-digital converters, signal processors, etc.). Some embodiments may include devices such as touch screens that serve as input and output devices. In some embodiments, other user output devices 1524 may be provided in addition to or in place of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, etc.

[0227] Some embodiments include electronic components, such as microprocessors, memories, and storage, that store computer program instructions in a computer-readable storage medium. Many features described in this specification may be implemented as processes designated as a group of program instructions encoded on a computer-readable storage medium. When one or more processing units execute these program instructions, they cause one or more processing units to perform the various operations indicated in the program instructions. Examples of program instructions or computer codes include machine codes such as those produced by a compiler and files comprising high-level codes executed by a computer, an electronic component, or a microprocessor using an interpreter. Through suitable programming, one or more processing units 1504 and 1516 may provide various functions for server system 1500 and client computing system 1514, including any functions or other functions described herein as being executed by a server or client.

[0228] It should be understood that server system 1500 and client computing system 1514 are illustrative, and variations and modifications are possible. The computer system used in conjunction with the embodiments of the present disclosure may have other capabilities not specifically described herein. In addition, although server system 1500 and client computing system 1514 are described with reference to specific blocks, it should be understood that these blocks are defined for the convenience of description, and these blocks are not intended to imply a specific physical arrangement of component parts. For example, different blocks can be but do not need to be in the same facility, the same server rack, or the same motherboard. In addition, blocks do not need to correspond to physically different components. Blocks can be configured to perform various operations, for example, by programming a processor or providing appropriate control circuits, and various blocks may or may not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present disclosure can be implemented in various devices including electronic devices implemented using any combination of circuits and software.

[0229] Although the present disclosure has been described with respect to specific embodiments, those skilled in the art will recognize that many modifications are possible. Various computer systems and communication technologies can be used to implement embodiments of the present disclosure, including but not limited to the specific examples described herein. Any combination of dedicated components and / or programmable processors and / or other programmable devices can be used to implement embodiments of the present disclosure. The various processes described herein can be implemented on the same processor or different processors in any combination. In the case where a component is described as being configured to perform certain operations, this configuration can be implemented, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation or any combination thereof. In addition, although the above-mentioned embodiments may refer to specific hardware and software components, it will be appreciated by those skilled in the art that different combinations of hardware and / or software components may also be used, and specific operations described as being implemented in hardware may also be implemented in software, and vice versa.

[0230] Computer programs incorporating various features of the present disclosure may be encoded and stored on a variety of computer-readable storage media; suitable media include disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, and other non-transitory media. Computer-readable media encoded with program code may be packaged with compatible electronic devices, or the program code may be provided separately from the electronic device (e.g., via Internet download or as a separately packaged computer-readable storage medium).

[0231] Therefore, although the disclosure has been described with respect to particular embodiments, it should be understood that the disclosure is intended to cover all modifications and equivalents within the scope of the appended claims.

Claims

1. A method of adapting digital therapeutic content for provision, comprising: identifying, by one or more processors, first digital therapeutic content to be provided via a network; Applying, by the one or more processors, the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output, wherein the ML model is trained by: identifying a training data set comprising a plurality of examples, each example of the plurality of examples identifying a corresponding second digital therapeutic content for providing and a first indication for identifying one of compliance or noncompliance, applying the second digital therapeutic content of an example from the plurality of examples in the training dataset to the ML model to generate a second output, determining a second indication for controlling one of compliance or non-compliance of provided second digital therapeutic content based on the second output, comparing the first indication of the example from the training dataset with the second indication determined by the ML model, and updating at least one of the set of weights of the ML model based on the comparison; determining, by the one or more processors, an indication as one of compliance or noncompliance based on applying the first digital therapeutic content to the ML model; as well as One or more data structures are used by the one or more processors to store an association between the first digital therapeutic content and the indication, and the indication is used to control the provision of the first digital therapeutic content via the network by the following steps: (i) limiting the provision of the first digital therapeutic content in response to determining the indication of non-compliance, and (ii) allowing the provision of the first digital therapeutic content in response to determining the indication of compliance.

2. The method according to claim 1, further comprising: receiving, by the one or more processors, an identification of the first digital therapeutic content associated with at least one domain of a plurality of domains; as well as selecting, by the one or more processors, an ML model corresponding to a domain to be applied to the first digital therapeutic content from a plurality of ML models, and Wherein, determining the indication further comprises determining the indication as one of compliance or non-compliance with the domain.

3. The method according to claim 1, further comprising: In response to determining the indication of non-compliance of the first digital therapeutic content, generating, by the one or more processors, one or more portions for modification in the first digital therapeutic content based on applying the ML model.

4. The method according to claim 1, further comprising: The one or more processors select one of a first content item or a second content item from a plurality of content items as the first digital therapeutic content to be provided based on an indication of conformance or non-conformance of each of the plurality of content items determined using the ML model.

5. The method according to claim 1, further comprising: receiving, by the one or more processors, via a user interface, a selection of a second indication identifying the first digital therapeutic content as compliant or noncompliant; as well as Overlaying, by the one or more processors, the indication from the ML model with the second indication received via the user interface.

6. The method according to claim 1, wherein: Each of the plurality of examples of the training data set further identifies a first risk score that identifies a degree of compliance or noncompliance for providing the second digital therapeutic content, Wherein determining the second indication further comprises determining a second risk score identifying a degree of compliance or non-compliance for providing the first digital therapeutic content.

7. The method according to claim 1, wherein: The first digital therapeutic content includes at least one of textual content or visual content to be provided to a device for presentation in a session to address a condition of an audience, wherein the audience is taking medication to at least partially address the condition in concert with the session.

8. A method of training a model to adjust the content of a provided digital therapeutic, comprising: identifying, by one or more processors, a training data set comprising a plurality of examples, each example in the plurality of examples identifying a corresponding first digital therapeutic content and a first indication for identifying one of compliance or non-compliance, to control provision of the first digital therapeutic content via a network by: (i) limiting provision of the first digital therapeutic content in the event of non-compliance, and (ii) allowing provision of the first digital therapeutic content in the event of compliance; applying, by the one or more processors, the first digital therapeutic content of an example from the plurality of examples in the training dataset to a machine learning (ML) model including a set of weights to generate an output; determining, by the one or more processors based on the output, a second indication for controlling one of compliance or noncompliance of provided first digital therapeutic content; comparing, by the one or more processors, the first indication of the example from the training dataset with the second indication determined by the ML model; updating, by the one or more processors, at least one of the set of weights of the ML model based on a comparison between the first indication and the second indication; as well as The set of weights of the ML model are stored, by the one or more processors, for use in determining compliance or non-compliance of second digital therapeutic content.

9. The method according to claim 8, further comprising: receiving, by the one or more processors via a user interface, a selection identifying an indication that the second digital therapeutic content is compliant or non-compliant; comparing, by the one or more processors, an indication determined from an output of the ML model with an indication from the user interface; as well as The ML model is retrained by the one or more processors based on the comparison.

10. The method according to claim 8, further comprising: In response to determining an indication of non-compliance of the second digital therapeutic content, generating, by the one or more processors, one or more portions for modification in the second digital therapeutic content based on applying the ML model.

11. The method according to claim 10, further comprising: The ML model is retrained, by the one or more processors, using the one or more portions to make modifications in the second digital therapeutic content.

12. The method according to claim 8, further comprising: In response to determining non-compliance of the second digital therapeutic content, modifying, by the one or more processors, prompts applied to a generative transformer model used to output the first digital therapeutic content, wherein the generative transformer model is part of or separate from the ML model.

13. The method according to claim 8, further comprising: A generative transformer model used to generate the first digital therapeutic content is retrained, by the one or more processors, based on an indication of one of non-compliance or compliance of the second digital therapeutic content.

14. The method according to claim 8, wherein: Each example in the plurality of examples of the training data set further identifies a first risk score that identifies a degree of compliance or noncompliance for providing the first digital therapeutic content, wherein applying further comprises applying the first digital therapeutic content of an example from the plurality of examples in the training dataset to the ML model to determine a second risk score identifying a degree of compliance or noncompliance for providing the first digital therapeutic content, and wherein at least one of the set of weights of the ML model is updated based on a comparison between the first risk score and the second risk score of the example.

15. The method according to claim 8, wherein: Each of the plurality of examples in the training dataset also includes an identification of the first digital therapeutic content as a domain of a plurality of domains and the first indication identifying one of compliance or non-compliance associated with the domain, and further includes: identifying, by the one or more processors, from a plurality of ML models, an ML model corresponding to a domain to be applied to the first digital therapeutic content; and Wherein determining the second indication further comprises determining the second indication as one of compliance or non-compliance with the first digital therapy.

16. A system for adapting digital therapeutic content for provision, comprising: One or more processors coupled to the memory and configured to: identifying first digital therapeutic content to be provided via a network; Applying the first digital therapeutic content to a machine learning (ML) model having a set of weights to generate a first output, wherein the ML model is trained by: identifying a training data set comprising a plurality of examples, each example of the plurality of examples identifying a corresponding second digital therapeutic content for providing and a first indication for identifying one of compliance or noncompliance, applying the second digital therapeutic content of an example from the plurality of examples in the training dataset to the ML model to generate a second output, determining a second indication for controlling one of compliance or non-compliance of provided second digital therapeutic content based on the second output, comparing the first indication of the example from the training dataset with the second indication determined by the ML model, and updating at least one of the set of weights of the ML model based on the comparison; determining an indication as one of compliance or noncompliance based on applying the first digital therapeutic content to the ML model; as well as One or more data structures are used to store an association between the first digital therapeutic content and the indication, and the indication is used to control the provision of the first digital therapeutic content via the network by the following steps: (i) limiting the provision of the first digital therapeutic content in response to determining the indication of non-compliance, and (ii) allowing the provision of the first digital therapeutic content in response to determining the indication of compliance.

17. The system of claim 16, wherein: The one or more processors are further configured to: receiving an identification of the first digital therapeutic content associated with at least one domain of a plurality of domains; selecting an ML model from a plurality of ML models that corresponds to a domain to be applied to the first digital therapeutic content, and The indication is determined as one of compliance or non-compliance associated with the domain.

18. The system of claim 16, wherein: The one or more processors are further configured to, in response to determining the indication of non-compliance of the first digital therapeutic content, generate one or more portions for modification in the first digital therapeutic content based on applying the ML model.

19. The system of claim 16, wherein: The one or more processors are also configured to select one of a first content item or a second content item from a plurality of content items as the first digital therapeutic content to be provided based on an indication of conformance or nonconformance of each of the plurality of content items determined using the ML model.

20. The system of claim 16, wherein: The first digital therapeutic content includes at least one of textual content or visual content to be provided to a device for presentation in a session to address a condition of an audience, wherein the audience is taking medication to at least partially address the condition in concert with the session.

Images