Automatic data integration for multiple individual digital transmission performance measurements with continuous optimization.

Abstract

To provide a method for measuring effectiveness of online healthcare advertising campaigns, a system, and a storage medium.SOLUTION: A method includes: obtaining, from a demand-side platform (DSP), impression data specifying service providers and consumer tokens representing consumers who have received digital impressions of a set of advertising campaigns; receiving a set of tokenized claims data records related to a prescription of a product from a database server; receiving a result set of integrated measurement records specifying measured campaigns linking the tokenized claims data records with impression data associated with consumer tokens and / or service provider identifiers from the database server; and generating and presenting aggregated analytics reports based on the integrated measurement records.SELECTED DRAWING: Figure 7A

Description

[Technical Field]

[0001] One technical field of this disclosure is computer-implemented demand-side platform (DSP) systems used in digital advertising technology. Another technical field is the use of relational databases, and specifically the automatic joining of tables that store diverse data sets under stored program control. [Background technology]

[0002] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be presumed that any of the approaches described in this section qualify as prior art solely by virtue of their inclusion in this section.

[0003] Digital advertising technology (ad tech) uses distributed computer systems under stored program control to determine what media or content user computers are accessing and which digital advertising units to select, deliver, or place on the media, content, or other locations. Ad tech systems have developed sophisticated means for real-time bidding on the placement of electronic ad units within websites, mobile device feeds, and other applications. However, ad tech systems still suffer from many limitations.

[0004] Many advertising agencies, pharmaceutical companies, medical device companies, insurance companies, and other healthcare-related companies want to increase advertising impressions of their healthcare products and services to eligible healthcare professionals (HCPs) and consumers. Impression deployment necessarily involves a demand-side platform (DSP) system for targeted delivery of product information. Determining appropriate online HCP identities, target consumers, and where to deliver specific product and service information is challenging given the myriad types of medical conditions, HCPs and their practice histories, patient privacy requirements under the Health Insurance Portability and Accountability Act (HIPAA), and the diversity of products in the healthcare industry. Clinical medical data, prescribing behavior data, National Healthcare Professional Identifier (NPI) data, demographic data, authentication, appointment scheduling, payment data, and other information related to NPI are generally unavailable to agencies and advertisers for use in determining which HCPs are best suited to deliver information about specific products, or the information on NPI is not sufficiently comprehensive or consistent with other data, thereby limiting its usefulness. Thus, DSP systems often deliver product information to HCPs whose patients would not benefit from such delivery, and / or miss many HCPs whose patients would benefit.

[0005] Data vendors can often sell data defining audience segments to DSPs. These approaches typically allow for minimal customization of target audiences or incur significant delays in customization that significantly reduce relevance, relying on buckets or segments of cookie or device data that are manually tagged to indicate specific audience characteristics. Other data providers present data through platforms that provide counting and aggregation of how many users with various attributes are recorded in HCP databases; these platforms do not have a DSP and therefore require an intermediary to transfer audience data to the DSP. This approach's lack of integration prevents it from providing real-time HCP-specific reports on ad engagement. Furthermore, existing systems use individual data stores based on browser cookie limitations and do not provide a secure method for integrating digital identity data with third-party data to enable more real-time customization and ad relevance.

[0006] Furthermore, existing technology does not provide an effective means of digitally combining or merging ad serving data with healthcare data (real-world evidence), thereby not providing an effective means of measuring the impact of healthcare advertising on patient outcomes and healthcare provider behavior change, and currently customizing and optimizing messaging based on campaign measurements of promotional responses and / or near-real-time clinical events. There is also no effective way to measure how certain advertising leads to improved health outcomes, including interactions between consumers and service providers and minimal ability to link offline behavioral changes with online campaign impression data. Integration of campaigns targeted to HCPs and patients is impractical or nonexistent. Even if this data could be determined, there is no practical or effective way to optimize DSPs or bids based on campaign measurements that account for this outcome. Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention has been made to solve the problems in the prior art described above. [Means for solving the problem]

[0008] The appended claims may serve as a summary of the invention. [Brief explanation of the drawings]

[0009] [Figure 1] An example system for secure training and deployment of machine learning systems using protected data is depicted. [Figure 2] An example message diagram is depicted for securely using protected information to generate and train machine learning systems. [Figure 3] An example method for building and validating machine learning systems in a secure environment is described. [Figure 4] An example method for building and validating machine learning systems using a secure environment is described. [Figure 5] 1 is a block diagram illustrating a computer system in which one embodiment may be implemented. [Figure 6A] 1 illustrates an example distributed computer system in which some embodiments may be implemented. [Figure 6B] The system of FIG. 6A is shown including some optimization elements. [Figure 7A] ~ [Figure 7B] An example measurement management process for a given campaign is illustrated. [Figure 7C] An example of functional elements and data flow in a DSP optimized embodiment is shown. [Figure 8] An example data flow for a given campaign is illustrated. [Figure 9] An example GUI for presenting an analytical report of a measured campaign is shown. [Figure 10] An example artificial neural network is shown. DETAILED DESCRIPTION OF THE INVENTION

[0010] Below, example embodiments are described in each section following this outline. 1.Overview 2. Direct-to-Consumer Modeling 2.1 Structure example 2.2 Process Overview 2.3 Implementing a Protected Environment 2.4 Media Server Implementation 2.5 Modeling Implementation 2.6 Advantages of Some Embodiments 3. Campaign performance measurement and optimization 4. Implementation Example 4.1 Hardware Overview 4.2 Artificial Neural Networks 5. Extensions and Substitutions

[0011] The embodiments disclosed herein are merely examples, and the scope of the disclosure is not limited thereto. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments described herein. Embodiments in accordance with the present invention are specifically disclosed in the accompanying claims for methods, storage media, systems, and computer program products, and any feature recited in one claim category, e.g., a method, may also be claimed in another claim category, e.g., a system. Dependencies or references in the accompanying claims are selected for formality reasons only. However, subject matter resulting from intentional reference (e.g., multiple references) to the preceding claims is also claimed, and therefore any combination of claims and their features is disclosed and may be claimed regardless of the references selected in the accompanying claims. Subject matter that may be claimed encompasses not only combinations of features presented in the accompanying claims, but also other combinations of claim features, and each feature recited in a claim may be combined with other features or combinations of features in the claims. Furthermore, any of the embodiments and features described or illustrated herein may be claimed in separate claims and / or in any combination with other embodiments or features described or illustrated herein or with any of the features of the accompanying claims.

[0012] 1.Overview Computer systems used in healthcare marketing have historically relied on a variety of technologies to measure and optimize the transmission of digital data, including advertisements. Healthcare data requires consideration of latency and security. Therefore, existing solutions for measuring the effectiveness of online healthcare advertising campaigns are generally inconsistent and not integrated into a single media platform. Some measurement systems have long data refresh delays, questionable accuracy, insufficient transparency, and operate independently of the media platform. Additionally, changes in browser cross-domain tracking policies pose numerous challenges for third-party measurement solutions to measure campaigns. As a result, healthcare marketers struggle to perform real-time optimization of online campaigns based on offline health outcomes data.

[0013] Furthermore, it has not been possible to measure the collaborative and interactive effects of integrated campaigns targeting both service providers and consumers. An integrated campaign involves intentionally coordinating advertising communications to service providers and consumers through similar messaging. In one embodiment, "service provider" refers to a healthcare provider ("HCP"), and "consumer" refers to a patient in the same embodiment. However, in other embodiments, service providers and consumers other than patients may be implemented in fields other than healthcare, but privacy concerns may also arise. Integrated campaigns are valued in the industry because they improve the shared understanding between consumers and providers prior to visits such as outpatient visits. Evidence suggests that more conversions occur when consumers and providers receive the same messaging regarding drug safety or drug efficacy. Conversion, in this context, may include prescription filling. In an embodiment of the present disclosure, a first distributed computer system or platform is provided that is programmed to plan, launch, and measure integrated campaigns within a DSP, allowing multiple brands to manage, measure, and optimize all of their campaigns within a single platform. "DSP" in this context refers to a demand-side platform or ad server computer, and all examples in this disclosure where a "DSP" is mentioned may be implemented using an ad server that does not act as a DSP, for example, by a publisher that serves ads but does not use a DSP.

[0014] Embodiments are constructed to use medical claims data that represents an accurate accounting of health care activities, complying with legal regimes such as HIPAA that prevent the direct disclosure and use of personal health information for marketing purposes. In embodiments, some data is anonymized to protect privacy while allowing legitimate uses for both ad targeting and campaign effectiveness measurement.

[0015] In certain embodiments, ad serving data is combined with healthcare data to measure the impact of healthcare advertising on patient outcomes and HCP behavior change. This combined data can be further used to measure interactions between consumers and service providers who receive these ads to quantify how a particular ad campaign leads to improved health outcomes. Thus, DSPs provide the ability to plan, launch, and measure healthcare campaigns that deliver timely information to consumers and their service providers to help them make more informed decisions about their health. The measured results can then be used to adjust and optimize various parameters of a particular ad campaign to enable control, customization, and delivery of the integrated campaign for service providers and consumers to achieve further desired outcomes.

[0016] In certain embodiments, the method includes, by the measurement server computer, receiving from the DSP, impression data specifying a plurality of first anonymized consumer tokens representing consumers who received digital impressions of a first campaign set associated with a first set of one or more healthcare attributes. Further, impression data specifying a plurality of second HCP identifiers representing HCPs who received digital impressions of the first campaign set associated with the first set of healthcare attributes is received from the DSP. The first campaign set may be included in a plurality of diverse campaigns executed by the DSP. A set of anonymized tokenized claims data records is received based on analytical instructions executed on the database server, each data record relating to at least one claim for a prescription for a designated product, and execution of these instructions may be performed in a secure environment with data privacy safeguards. The method further includes receiving from the database server a result set of one or more aggregated measurement records specifying one or more measured campaigns among the plurality of diverse campaigns. The one or more measured campaigns may be associated with at least one of the consumer tokens and / or prescriptions for a designated product in at least one claims data record associated with at least one of the HCP identifiers. Finally, the method includes generating and presenting one or more analytical reports based on the consolidated measurement records.

[0017] There are technical challenges associated with running, measuring, and optimizing ad campaigns, such as those run by pharmaceutical companies. For example, healthcare marketers often must rely on several different technologies to run their digital ad campaigns. However, given the latency of healthcare data and the unique challenges surrounding managing consumer data, solutions for measuring the effectiveness of healthcare campaigns are often inconsistent and not fully integrated into a single media buying platform. Traditional measurement products have long data refresh delays, questionable accuracy, limited transparency, and often operate independently of the media buying platform used. Furthermore, the lack of a platform for measuring the collaborative and interactive effectiveness of integrated campaigns for service providers and consumers presents another technical challenge. Yet another technical challenge lies in the fact that browser cross-domain tracking policies create challenges for third-party measurement solutions attempting to measure campaigns. These limitations can make it difficult for healthcare marketers to perform real-time optimization of campaigns based on offline health outcomes data.

[0018] Certain embodiments disclosed herein may provide one or more solutions to address these challenges, as well as various technical advantages. As an example, a disclosed DSP may provide an integrated platform that enables targeted delivery of advertisements and intentional tailoring of campaigns to select relevant healthcare providers and consumers with timely, similar messaging. For example, a DSP may enable key considerations highlighting the benefits of a particular drug to be communicated to HCPs and patients prior to a clinical encounter or interaction between the two. Such an integrated campaign can "synchronize" patients and providers with relevant messaging about a particular drug prior to the encounter / visit. This can result in increased patient conversions when both patients and HCPs are prepared to consider the same messages (such as drug safety, its effectiveness, etc.) prior to this interaction.

[0019] This unified approach results in various technical advantages. By way of example, one technical advantage of embodiments may include the ability to administer, manage, and measure integrated campaigns within a single, unified platform. Another technical advantage of embodiments may include the ability for an entity to receive data representing interactions between qualified patients and HCPs and evaluate the impact of this marketing campaign on driving conversions or changing patient behavior. Yet another technical advantage of embodiments may include automating campaign optimization in response to this data to improve the effectiveness and targeting of future campaigns. Certain embodiments described herein may provide none, some, or all of the above technical advantages. One or more other technical advantages will be readily apparent to those skilled in the art upon consideration of the figures, descriptions, and claims of the present disclosure.

[0020] 2. Direct-to-Consumer Modeling 2.1 Structure example FIG. 1 illustrates an example system for secure training and distribution of machine learning systems using protected data. A server computer 110, a billing processor 130, an attribute database 140, a media server 150, and a client computing device 160 are communicatively coupled via one or more networks. A network broadly represents any combination of one or more data communication networks, including a local area network, a wide area network, an interconnected network, or the Internet, using either wired or wireless links, including terrestrial or satellite links. A network may be implemented by any medium or mechanism for exchanging data between the various elements of FIG. 1. The various elements of FIG. 1 may also have direct (wired or wireless) communication links. Each of the server computer 110, the billing processor 130, the attribute database 140, the media server 150, the client computing device 160, and other elements of the system is equipped with a network-compatible interface and is programmed or configured to use standardized communication protocols over the network, such as TCP / IP, Bluetooth, the CAN protocol, and higher-layer protocols such as HTTP, TLS, and others.

[0021] The claims processor 130 includes one or more computing systems configured to accept and store claims data. The claims processor 130 stores claims data 132 and identification information 134. The claims data 132 includes data identifying one or more status values ​​for a plurality of personal data records. For example, the claims data may include medical claims records identifying International Statistical Classification (ICD) codes for diseases and related health problems, procedure codes such as Physician Profession Terminology (CPT) codes, codes associated with healthcare providers (HCPs), Healthcare Common Procedure Coding System (HCPCS) codes, and diagnostic codes such as J-codes or NDC codes for prescriptions. The status values ​​may include the presence or absence of a particular code, such as an ICD-10 code for the diagnosis of type 2 diabetes. The claims data 132 may be associated with identification information 134, such as the name, address, date of birth, or other identifying information of the personal data records. The claims processor 130 uses the identification information 134 to generate a cryptographic token 136 using the methods described herein. The billing processor 130 sends the billing data 132, including the encrypted token 136, to the server computer 110. Additionally or alternatively, the billing processor 130 sends the billing data 132 and the identification information 134 to a tokenization server, which generates the encrypted token 136 from the identification information using methods described herein and sends the encrypted token and the billing data to the server computer 110.

[0022] The attribute database 140 comprises a data store, such as a relational database or other structured data storage device, configured to store attribute information for a plurality of personal data records. The attribute database 140 stores attribute data 142 and identification information 144. The attribute data 142 may include distinct values ​​for a plurality of values. For example, the attribute database 140 may store multiple rows, each corresponding to a different personal data record, and multiple columns, each corresponding to a different attribute. The attributes may include personal information such as age, physical activity level, weight, hair color, and / or eye color; data related to online search history, such as the presence of particular search terms, websites visited, or other internet history; or data related to one or more online accounts, such as social network accounts or other memberships. The attribute data 142 may be associated with identification information 144, such as the name, address, date of birth, or other identification information of the personal data record. The attribute database 140 uses the identification information 144 to generate a cryptographic token 146 using the methods described herein. Attribute database 140 sends attribute data 142, including encrypted token 146, to server computer 110. Additionally or alternatively, attribute database 140 sends attribute data 142 and identification information 144 to a tokenization server, which generates encrypted token 146 from the identification information using methods described herein and sends the encrypted token and attribute data to server computer 110.

[0023] The server computer 110 comprises one or more computing devices configured to generate and train one or more machine learning systems. The server computer 110 can be a physical server computer and / or a virtual server instance stored in a data center, such as through cloud computing. The server computer 110 can be configured to generate and train machine learning systems within a protected environment 112. The protected environment 112 encompasses a hardware or software environment that can include one or more server computers, such as the server computer 110, one or more local networks, load balancers, and / or data storage devices. The protected environment 112 is configured to protect data stored within the environment, such as through firewalls or other network security systems that restrict access to various systems or devices within the protected environment through a network, such as the Internet. The protected environment 112 can be configured to prevent data from being released from the environment that does not meet certain criteria, as described further herein. In this manner, the protected environment can be used as a barrier to protect certain types of information, such as confidential or restricted-use data, such as medical claims protected by HIPAA.

[0024] The server computer 110 stores anonymized attribute data 122 received from the attribute database 140 and anonymized billing data 124 received from the billing processor 130. The anonymized attribute data 122 and the anonymized billing data 124 may include attributes and billing data, respectively, that are mapped to cryptographic tokens but do not include identifying information. Methods for generating anonymized data are further described herein. The server computer 110 uses the anonymized attribute data 122 and the anonymized billing data 124 to create anonymized training data 114 that the server computer 110 stores. The server computer 110 also stores training data validation instructions 115, machine learning generation training instructions 116, and machine learning validation instructions 118. The anonymized training data 114 is stored as multiple rows, each row corresponding to a different personal data record. The multiple data rows may include columns corresponding to different attributes of the personal data record and columns corresponding to status values ​​of the personal data record, such as a diagnostic code.

[0025] The training data validation instructions 115 include computer-readable instructions that, when executed by one or more processors of the server computer 110, cause the server computer 110 to determine whether a training data set meets one or more criteria and to perform a responsive action depending on whether the training data set meets the one or more criteria. The machine learning generation training instructions 116 include computer-readable instructions that, when executed by one or more processors of the server computer 110, cause the server computer 110 to generate a machine learning system based on the one or more instructions and train the machine learning system using the anonymized training data 114. The machine learning validation instructions 118 include computer-readable instructions that, when executed by one or more processors of the server computer 110, cause the server computer 110 to determine whether a machine learning system meets one or more criteria and to perform a responsive action depending on whether the training data set meets the one or more criteria.

[0026] The computer-executable instructions described herein are machine-executable code of a CPU's instruction set, which may be compiled based on source code written in JAVA, C, C++, OBJECTIVE-C, or other human-readable programming languages ​​or environments, alone or in combination with JAVASCRIPT scripts, other scripting languages, and other programming source text. In another embodiment, the programmatic instructions also represent one or more files or projects of source code digitally stored in a mass storage device, such as non-volatile RAM or disk storage, in the system of FIG. 1 or another repository system that, when compiled or interpreted, generates executable instructions at run time that cause a computer to perform the functions or operations described herein for the executable instructions. In other words, the diagram represents how a programmer or software developer organizes and arranges source code for later compilation into an executable file or interpretation into bytecode or the like for execution by server 110.

[0027] The server computer 110 uses the machine learning generation training instructions 116 and the anonymized training data 114 to generate a trained machine learning instruction system 117. For example, the server computer 110 generates a training data set from the anonymized training data 114 based on the one or more instructions and uses the training data set to train a machine learning system generated by the server computer based on the one or more instructions. The server computer 110 sends the trained machine learning system 117 to the media server 150.

[0028] Media server 150 comprises one or more computers configured to receive requests and send media to one or more client computing devices. Media server 150 stores media items 152 and trained machine learning systems 156 received from server computer 110. Media items 152 include one or more images, videos, or other media items that can be provided to a client computing device. Media server 150 is configured to communicate with client computing device 160 to determine whether to send one of media items 152 to client computing device 160. Media server 150 determines whether to send a media item using client computing device attribute data 154 stored on media server 150.

[0029] Client computing device attribute data 154 includes one or more attributes corresponding to client computing device 160, such as attributes associated with a personal data record corresponding to the client computing device. Client computing device attribute data 154 may be received from client computing device 160, attribute database 140, and / or one or more other attribute sources. For example, a request for attribute data about client computing device 160 may cause media server 150 to receive identifying information from client computing device 160 that it sends to attribute database 140.

[0030] 1 depicts single instances of server computer 110, attribute database 140, billing processor 130, media server 150, and client computing device 160 for purposes of clarity, in some embodiments, the systems and devices of FIG. 1 may comprise multiple systems or devices. For example, server computer 110 may comprise multiple server computers and / or external storage devices that store attribute data, billing data, training data, and / or any other data stored within protected environment 112. As another example, server computer 101 may communicate with multiple media servers 150, each of which may communicate with multiple client computing devices 160.

[0031] 2.2 Process Overview Figure 2 depicts an example message diagram for securely using protected information to generate and use a trained machine learning system. Figure 2 and the other flowcharts described herein, alone or in combination with the process and functional descriptions in the text herein, may act as algorithms, plans, or instructions that can be used to program a computer or logic to implement the described functions. In other words, all written text and all drawings herein, together, are intended to provide a disclosure of algorithms, plans, or instructions that are sufficient for one skilled in the art to program a computer to perform the functions described herein, in combination with the skill and knowledge of a person with an appropriate level of skill for this type of invention and disclosure.

[0032] In step 202, the claims processor 130 stores claims data. Claim data, as used herein, refers to condition value data for one or more personal data records. Personal data record, as used herein, refers to an individual's record containing one or more values ​​related to an individual. Thus, an individual claims data record identifies an individual's condition and identifies the individual through identifying information such as name, date of birth, social security number, address, or other identifying information. An individual's condition may include a medical condition, personal condition, legal condition, or other data value related to a condition that may be stored in the claims data record. For example, an individual claims data record may include a medical diagnosis from a medical professional. An example of a claims processor may include an intermediary between a medical professional and an insurance agency that accepts medical records containing protected data such as diagnoses or prescriptions that are then forwarded to the insurance agency.

[0033] In step 204, the billing processor 130 anonymizes the billing data using a tokenization scheme. For example, the billing processor may create a data token by hashing certain identifying information, such as first name, last name, zip code, and date of birth, using a specific hash function and encrypting the hashed information. The billing processor may then create anonymized billing data that includes the data token and one or more status data values ​​for the data token. As a practical example, if a billing data record includes the name, zip code, date of birth, or medical diagnosis for a personal data record, the billing processor 130 generates a token using the identifying information and stores the anonymized data record that includes the token and the medical diagnosis. Because the token is generated from the identifying information for the personal data record, the token is unique for each personal data record. Although a data token generated through hashing the identifying information and encrypting the hashed information is described in this disclosure, any identifying algorithm scheme for generating a unique data token from identifying information may be used.

[0034] In step 206, the billing processor 130 sends the anonymized billing data to the server computer 110. For example, the billing processor 130 sends a plurality of tokens and a corresponding state value for each of the plurality of tokens to the server computer 110, which stores them as anonymized billing data. The billing processor 130 sends the anonymized billing data as a plurality of data records, each of which includes a unique token but does not contain any identifying information.

[0035] In step 208, attribute database 140 stores attribute data. Attribute data, as used herein, refers to multiple attribute data values ​​for one or more personal data records. Thus, a personal attribute data record identifies multiple attributes of an individual and identifies the individual through identifying information such as name, date of birth, social security number, address, or other identifying information. Attributes may include well-known information about the personal data record, such as personal information, internet history information, account information, or other storage information. In embodiments, attribute database 140 may store data for hundreds of attributes, including data records that store information about a subset of hundreds of attributes, such as when attribute data for one or more attributes is unavailable for a particular personal data record.

[0036] At step 210, the attribute database 140 anonymizes the attribute data using a tokenization scheme. In one embodiment, the tokenization scheme used by the attribute database 140 to anonymize the attribute data is the same tokenization scheme used by the claims processor 130 to anonymize the billing data. For example, if the tokenization scheme used by the claims processor involves using a particular hash function to hash a string containing a first name, a last name, and a zip code, and using a particular encryption key to encrypt the hashed string, the tokenization scheme used by the attribute database 140 may also use a particular hash function to hash the same string and encrypt the same hashed string using the same particular encryption key. In this way, the same token is created by both the claims processor 130 and the attribute database 140 for the same personal data record, even though both the claims processor 130 and the attribute database 140 anonymize the information separately. Additionally or alternatively, tokenization may be performed by a tokenization server that uses the same method to generate tokens for the claims processor 130 and the attribute database 140. The attribute database 140 may then create anonymized attribute data for each personal data record, including one or more attribute tokens and values.

[0037] In step 212, the attribute database 140 sends the anonymized attribute data to the server computer 110. For example, the attribute database 140 sends a plurality of tokens and corresponding attribute values ​​for each of the plurality of tokens to the server computer 110, which stores the anonymized attribute data. The attribute database 140 sends the anonymized attribute data as a plurality of data records, each data record containing a unique token but no identifying information.

[0038] In step 214, the server computer 110 joins the attribute data and the claim data to form a joined data set. For example, the server computer 110 generates multiple data rows, with each row corresponding to a particular personal data record. One example of a joining technique involves a left join of the claim data to the attribute data, thereby retaining all attribute data, but storing in the anonymized training data only the claim data stored with tokens corresponding to those in the attribute data. As another example, the server computer 110 may identify claim data containing a particular token and attribute data containing the same particular token. The server computer 110 generates a data row for the particular token, with the data row including multiple columns for multiple attributes based on the attribute data and one or more columns for one or more state values ​​based on the claim data. Thus, each row includes attribute data for a personal data record and claim data for the personal data record, and the row does not include identifying information for the personal data record.

[0039] 1 includes tokenizing the identification information, in other embodiments, the identification information is not tokenized and / or encrypted. For example, billing data 132, identification information 134, attribute data 142, and identification information 144 are sent directly to a server computer, which in a secure environment uses the identification information to join the two data sets together instead of using an encrypted token to match the billing data to the attribute data.

[0040] In step 218, the media server 150 sends a request to the server computer 110 for the machine learning system. The request is sent through an application programming interface of the server computer 110 and may include identification of input and output strings from the joint data. For example, the request may identify a subset of multiple attributes to use as inputs and the presence of a particular state value as an output. The request may additionally include parameters for the machine learning system, such as the number of nodes or layers.

[0041] At step 220, server computer 110 generates a machine learning system from the spliced ​​data based on the request. For example, server computer 110 may generate a machine learning system such as a random forest model, a neural network, a logistic regression, or a gradient boosting decision tree such as the XGBoost algorithm using stored parameters and / or parameters received from media server 150. Server computer 110 may then train the machine learning system using the attributes identified by media server 150 as inputs and the state values ​​of particular states as outputs.

[0042] As a practical example, the media server computer may identify five input attributes—age, gender, average number of tests, weight, and height—and an output status value indicating whether or not a diabetes diagnosis has been made. The server computer may identify corresponding columns of attribute data and billing data and generate a training data set using only these columns. Additionally or alternatively, the server computer may generate columns in which the data in the columns may be non-numeric or stored differently. For example, if a status value column stored in the server computer 110 contains one or more diagnostic codes for diagnoses corresponding to personal data records in each row, the server computer 110 may generate a column for a particular diagnostic code by including in each row a value of "0" if the row does not contain the particular diagnostic code and a value of "1" if the row contains the particular diagnostic code.

[0043] In step 222, server computer 110 sends the machine learning system to media server 150. In one embodiment, server computer 110 identifies one or more of the training datasets used to train the machine learning system, or a trained machine learning system, using methods described herein before sending the machine learning system to media server 150. The trained machine learning system may be sent in a form that is easily usable by media server 150, such as a weight matrix for the machine learning system.

[0044] At step 224, media server 150 uses a machine learning system to determine whether to send media to the client computing device. For example, media server 150 may receive attribute data about the client computing device. The attribute data may include a value for each attribute used to train the machine learning system. Media server 150 may use the machine learning system to calculate a likelihood of state existence from the attribute data about the client computing device. Based on the likelihood of state existence, media server 150 may send the relevant media item to the client computing device. For example, media server 150 determines whether the likelihood is greater than a threshold, and if the likelihood is greater than the threshold, sends the media item to be displayed on the client computing device.

[0045] Additionally, in one embodiment, after steps 214, 224, or other points in the process, input may be accepted to request one or more analytical reports based on the results of the preceding steps. Examples of reports that may be requested and generated at points in the process flow of Figure 2 are described in other sections of this specification in connection with Figures 8 and 9.

[0046] 2.3 Implementing a Protected Environment Figure 3 depicts an example method for building and validating machine learning systems in a secure environment.

[0047] In step 302, a server computer within the protected environment stores attribute data and status data. For example, the server computer may store multiple data columns, each column corresponding to a different attribute, with each row containing a value indicating the attribute value of a particular personal data record. The server computer may additionally store one or more data columns identifying a status value, such as an ICD-10 code.

[0048] At step 304, the server computer receives instructions to generate a machine learning system including specific inputs and outputs. The instructions may identify which attributes to use as inputs and the presence or absence of a state value as an output. For example, the instructions may specify inputs for age, sex, weight, and height, and an input for the presence or absence of an ICD-10 code for type 2 diabetes. The instructions may also identify parameters for the machine learning system, such as the number of layers or the number of nodes. Additionally or alternatively, the server computer may be configured to store parameters for the machine learning system and / or modify parameters for the machine learning system in response to the machine learning system failing to meet one or more criteria.

[0049] In step 306, the server computer generates a training data set from the stored data. For example, the server computer may first identify personal data records that have values ​​for each of the selected inputs. For example, some personal data records lack values ​​for "age" or "gender" and therefore would not be used to generate the training data set if the instructions identified age and gender as inputs. The server computer may generate training data sets for multiple personal data records that include attribute values ​​as inputs and the presence or absence of a state value as an output. For example, if the output is specified as indicating the presence of a particular ICD-10 code, the output of personal data records that include the particular ICD-10 code may be set to 1, while the output of personal data records that do not include the particular ICD-10 code may be set to 0.

[0050] In one embodiment, generating the training data set includes selecting a subset of stored data that can be used to generate the training data set. For example, if 3,000 data records contain the required attributes, the server computer may select fewer than 3,000 data records to train the machine learning system. The number of records to be used may be identified in the received instruction and / or a stored percentage value. For example, the server computer may be configured to use only half of the valid records. Additionally or alternatively, the server computer may select records such that a minimum number of records including outputs are used for training, and a minimum number of records including outputs are not used for training, thereby ensuring that the machine learning system does not have to memorize all stored personal data records.

[0051] In step 308, the server computer determines whether the dataset satisfies a first criterion. The first criterion may include a minimum number of instances of positive values ​​for the output. The server computer may be configured to determine whether at least a threshold number of instances of personal data records including a status value as an output are provided. For example, if the output value is a specific ICD-10 code, the server computer may determine whether at least a threshold number of data records in the stored data that can be used to construct the training dataset include the specific ICD-10 code. The threshold number may be a value stored on the server computer or identified in the received instructions. The first criterion may additionally or alternatively include a minimum number of instances of personal data records that do not include a status value as an output, a minimum and / or maximum ratio between personal data records that include a status value as an output and data records that do not include a status value as an output, and / or a minimum number of residual data records not used to generate the training dataset that include and / or do not include a status value.

[0052] Step 308 may be performed prior to generating a training data set, thereby determining whether a training data set generated from stored data satisfies a first criterion. For example, if the first criterion is a minimum number of instances of a particular ICD-10 code, the server computer may first identify each data record that can be used to generate the training data set and determine whether the number of data records meets or exceeds the minimum number. In one embodiment, the server computer generally determines whether the stored data that can be used to construct the training data set contains a minimum number of instances of a state value in addition to determining whether the stored data contains a minimum number of instances of a state value. Thus, the server computer may distinguish between whether a data set satisfies the first criterion when using the identified state value as an output and whether a training data set using the requested attribute as an input when using the identified state value as an output satisfies the first criterion.

[0053] If the dataset does not satisfy the first criterion, the server computer rejects the request for the machine learning system in step 316. For example, the server computer may send data to the requesting computing device rejecting the request for the machine learning system. The rejection may indicate that the first criterion was not met. In one embodiment, the rejection additionally identifies whether the first criterion is met for various inputs, such as when a minimum number of instances of an output state value are present but not present in the records containing the attribute values ​​for the request inputs.

[0054] If the dataset satisfies the first criterion, in step 310, the server computer trains a machine learning system using the training dataset. For example, the server computer may generate a new machine learning system using the received and / or stored values ​​for the machine learning system's parameters. The machine learning system may include a logistic regression model, a neural network, a random forest model, a gradient-boosted decision tree, and / or any other machine learning system that can be used to solve classification problems. In one embodiment, the received instructions specify a type of machine learning system to train from multiple types of machine learning systems. For example, the server computer may store instructions for generating one of multiple machine learning systems. The server computer may accept instructions specifying which of the multiple machine learning systems to generate and train. The server computer may generate the machine learning system using the stored and / or received parameters and train the machine learning system using attributes about the personal data record as inputs and values ​​indicating the presence or absence of a particular state as outputs.

[0055] In step 312, the server computer determines whether the machine learning system satisfies a second criterion. The second criterion relates to the accuracy of the machine learning system, which ensures that the machine learning system does not have to memorize all inputs. For example, the second criterion may be the maximum average percentage likelihood of a state value calculated when using the machine learning system to calculate an output for an input training data set that contains states as outputs.

[0056] In one embodiment, the second criterion includes a minimum percentage of the risk population based on the machine learning system. For example, the server computer may use the trained machine learning system to calculate outputs for multiple input datasets. The input datasets may include datasets generated from stored data not used to train the machine learning system, datasets used to train the machine learning system, and / or datasets received by the initial instructions to generate and train the machine learning system. The server computer may then calculate the percentage of the risk population based on the number of positive outputs from the multiple input datasets and / or the number of training dataset instances of positive state values. An example equation is: R=T / P In the above formula, R is the proportion of the risk population, T is the number of training dataset instances that are truly positive for the output value, and P is the number of positive predictions by using the machine learning system on multiple input datasets. The server computer may store a maximum threshold for R as the second criterion, such as 0.2. Therefore, if R is greater than 0.2, the server computer may determine that the machine learning system does not meet the second criterion.

[0057] If the machine learning system does not satisfy the second criterion, the server computer rejects the request for the machine learning system in step 316. For example, the server computer may send data to the requesting computing device rejecting the request for the machine learning system. The rejection may be that the second criterion was not met. In one embodiment, after sending the rejection, the server computer may accept another request to create a machine learning system. If the rejection is accepted based on the first criterion, the server computer proceeds to step 306. If the rejection is accepted based on the second criterion and the selected inputs and outputs remain the same, the server computer may skip checking the first criterion that is known to be satisfied. For example, the second request may specify the same inputs and outputs but may change parameters for training the machine learning system in an attempt to reduce the accuracy or percentage of the risk population. The server computer may generate a new machine learning system including new parameters, train the new machine learning system with the same training data set, and determine whether the new machine learning system satisfies the second criterion.

[0058] If the machine learning system satisfies the second criterion, the server computer sends the trained machine learning system to the requesting computing device in step 314. For example, the server computer may release the trained machine learning system from the protected environment to the requesting computing device upon determining that all criteria have been met. The trained machine learning system may not contain any of the training data used to create the trained machine learning system, but may include weight values ​​for each of the columns, thereby protecting personal data while still providing a machine learning system trained based on personal data. Because the server computer is configured to perform these tasks without allowing external access to data stored on the server computer, the server computer provides a means for accessing protected or personal information, but does not provide the content of the protected or personal information.

[0059] 2.4 Media Server Implementation Figure 4 depicts an example method for building and validating machine learning systems using a secure environment.

[0060] In step 402, the media server identifies client attributes, a target state, and machine learning system parameters. For example, the media server may accept input specifying client attributes for the input and a target state as the output. The media server may additionally accept input specifying machine learning parameters. Additionally or alternatively, the media server may store initial machine learning parameters. In one embodiment, the media server further accepts input specifying the type of machine learning system to build.

[0061] In step 404, the media server sends instructions to the secure environment to build a machine learning system that includes the identified client attributes as inputs, the target state as outputs, and the machine learning system parameters. For example, the media server sends instructions to build the machine learning system through an API of a server computer running in the secure environment, the instructions identifying the attributes to use as inputs and the state values ​​to use as outputs.

[0062] In optional step 406, if the media server receives a rejection, the media server responds by sending instructions including updated attributes or parameters. The media server may receive a rejection if the training data or the machine learning system fails to meet one or more criteria. The media server may display an error message and request various inputs, outputs, and / or parameters to be sent to the server computer. In one embodiment, the media server may be configured to change parameters for the machine learning system when an error is received based on the machine learning system failing to meet one or more criteria. For example, the media server may be configured to change the number of nodes or the number of layers pseudo-randomly and / or based on a stored second set of parameters.

[0063] At step 408, the media server receives the trained machine learning system. For example, the media server may receive the trained machine learning system from the secure environment when the machine learning system meets stored criteria. The media server may store the machine learning system, identifiers of the attributes used as inputs, and states used as outputs of the machine learning system.

[0064] At step 410, the media server accepts attributes of the client computing devices. The media server may be configured to determine whether to provide a particular media item to the client computing device. For example, the media server may be configured to determine which computing devices to send advertisements for diabetes medication. The media server may accept the attributes of the client computing devices prior to or after steps 402-408. For example, the media server may store attributes of multiple client computing devices prior to accepting a request for media to be sent to the client computing device.

[0065] Additionally or alternatively, the media server may request attribute data from an external source, such as an attribute database, based on information received from the client computing device. For example, the media server may receive a request to display media on the client computing device, such as in response to the client computing device directing the client computing device to a particular web page. The media server may additionally receive data from the client computing device or from an external source, to which the media server may send an attribute database upon request for attributes of the client computing device. The request may specify attributes used to train a machine learning system.

[0066] In step 412, the media server uses the received attributes and a machine learning system to determine the likelihood of the condition. The media server uses the attributes as input to the machine learning system to calculate a result value or target value indicative of the likelihood of the condition. Thus, if the machine learning system was trained using a diagnosis of type 2 diabetes as output, the media server may use the attributes to calculate the likelihood of type 2 diabetes based on the input attributes. The server computer may calculate the likelihood of the condition upon receiving a request for media and / or prior to receiving the request. For example, the server computer may calculate likelihoods for multiple client computing devices and store the likelihood values ​​for later use.

[0067] In one embodiment, the media server performs steps 402-412 multiple times for a single client computing device. For example, the media server requests multiple machine learning systems from the protected environment, each trained with different state values ​​as output. The media server uses the multiple trained machine learning systems to calculate multiple likelihood values, each corresponding to a different state. The media server may store the multiple likelihood values ​​for use in determining which media items to send to the client computing device.

[0068] In step 414, the media server determines whether to send a media item to the client computing device based on the likelihood of the state. For example, the media server may store media items corresponding to particular states. The media server may determine whether the likelihood of the state for the client computing device is greater than a stored threshold, such as 80%. If the likelihood is greater than the stored threshold, the media server may send the media item to the client computing device. If the likelihood is not greater than the stored threshold, the media server may send various media items to the client computing device.

[0069] In one embodiment, the media server selects one of a plurality of media items based on a plurality of likelihood values. For example, the media server stores a plurality of media items, each corresponding to one or more states. The media server may calculate a plurality of state likelihoods for the client computing device using a plurality of machine learning systems, each trained with one of the plurality of states as an output. The media server may identify the state with the highest likelihood and select the media item corresponding to the identified state. The media server may then send the selected media item to the client computing device.

[0070] In one embodiment, a media server may use the likelihood of a state to determine the value of one or more media items. For example, a media server may receive a request to send a plurality of media items, such as 1,000 media items, to a client computing device corresponding to a personal data record that includes a state value. If the likelihood of a state for a particular personal data record is 50%, the media server may evaluate the sending of the media items to the client computing device to be valued at one-half the value of the personal data record that corresponds to the state. Thus, if the request was for 1,000 media items to be sent to a client computing device corresponding to a personal data record that includes a state value, the media server may send media items to the client computing device until the value of the personal data record being sent corresponds to 1,000, such as 2,000 media items being sent to the client computing device corresponding to the personal data record with a 50% likelihood of the state value. Additionally or alternatively, the media server may use the likelihood of a state to dynamically determine the price of sending media items to the client computing device. For example, if the price to send a media item to a client computing device corresponding to a personal data record with a state value is $10, the media server may charge $5 to send a media item to a client computing device corresponding to a personal data record with a 50% likelihood of the state value.

[0071] 2.5 Modeling Implementation In one embodiment, the systems and methods described herein may be used to identify the effect of a particular action on the state of personal data records while protecting the information used. For example, a server computer may determine the percentage of identified personal data records in a protected environment that have a particular state and the percentage of identified personal data records that have received a benefit based on a request from an external computing device, such as a media server. Embodiments are further described herein.

[0072] In one embodiment, the server computer determines the percentage of identified personal data records that have a particular status. For example, after sending media items to multiple client computing devices, the media server may store identifiers, such as cookie identifiers, for the multiple personal data records that correspond to the computing devices that received the media items corresponding to the particular status. The media server may send the identifier and identification information for the particular status to the server computer. In one embodiment, the media server generates a unique token for the multiple personal data records using the methods described herein and sends the generated unique token to the server computer, including the identification information for the status. The server computer may match the received identifiers to personal data records stored in the secure environment, such as through mapping cookie identifiers to personal data records. The server computer may then determine for each identifier within the secure environment whether the identifier corresponds to a particular status. As an example, the server computer may determine whether a particular ICD-10 code is listed in a row corresponding to the personal data record. The server computer may determine the number and / or percentage of identifiers that correspond to a particular status and send this number and / or percentage to the media server.

[0073] In one embodiment, the server computer may be configured to send the number or percentage of identifiers from the protected environment only after determining that the number and / or percentage meets a third criterion. The third criterion may be a minimum number of total identifiers, a maximum number and / or percentage of identifiers having a particular state, or a minimum number or percentage of identifiers having a particular state. By using the third criterion, the server computer may ensure that protected information is not released to the media server.

[0074] In one embodiment, the server computer is configured to determine a benefit for one or more personal data records based on the additional received billing data. For example, a billing processor may send the additional billing data to the server computer. The server computer accepts the additional billing data and correlates the additional billing data with previously stored billing data, such as through a unique identifier generated by the billing processor. The server computer may additionally accept data from the media server including a plurality of identifiers of personal data records corresponding to computing devices that have received media items corresponding to a particular state. The server computer may determine the number and / or percentage of personal data records that have received a benefit from the plurality of identifiers of personal data records and the received additional billing data. Benefit, as used herein, encompasses a determination made by the server computer of a change in the state of a personal data record defined as beneficial. The definition of what is used by the server computer as a "benefit" is further described herein.

[0075] In one embodiment, a benefit is defined as an additional state corresponding to a personal data record. For example, the server computer may receive from the media server an identification of a prescription code for a medication corresponding to a sent media item. The server computer may determine from the additional billing data whether any of the personal data records corresponding to the identifier received from the media server include a prescription code for the medication. The server computer may calculate the number and / or percentage of identifiers in the additional billing data that correspond to personal data records that include a prescription code and send the number and / or percentage to the media server.

[0076] A benefit may also be defined as the elimination or change of a condition in a corresponding data record. For example, the server computer may be configured to determine that a benefit has occurred when a particular condition is listed for elimination in a future data record or is changed to a condition identified by the media server, such as a less severe illness, or when various conditions, such as prescriptions for pain medication, are eliminated, indicating that pain management is no longer needed. In one embodiment, a benefit may be defined by claims, such as fewer doctor visits or fewer filled prescriptions.

[0077] In one embodiment, the benefit is defined by a request from the media server. For example, a request including a plurality of identifiers and one or more states and / or state changes for the plurality of identifiers may be sent to the server computer. As a practical example, the media server may send a request for identifying the number and / or percentage of identifiers sent by the media server corresponding to personal data records that include the removal of a particular state in the additional billing data. The server computer may identify each identifier sent by the media server that initially corresponds to a particular state. The server computer may then identify which identifiers corresponding to a particular state will result in the removal of the particular state in future billing data. The server computer may then send to the media server the number or percentage of accepted identifiers that include the removal of the particular state in future billing data.

[0078] 2.6 Advantages of Some Embodiments The systems and methods described herein contribute to the technical features of using a machine learning system by being specifically adapted to a particular technical implementation in which instructions for generating a training data set and a machine learning system and training the machine learning system using the training data set are received from an external server computer, while the server computer within the protected environment is used to train and validate the machine learning system, and the system is then released from the protected environment for use by the external computing system. In a unique technical implementation of this machine learning system, additional data protection is provided for information stored on the server computer by performing the training and validation on the server computer so that the initial training data is not visible to users of the external device.

[0079] The systems and methods described herein further provide a practical application of machine learning systems through the generation and training of machine learning systems in the protected environment of a server computer. These systems and methods provide a specific means for solving the technical problem of using protected information without providing the protected information in an environment where users can see or use it. By validating machine learning systems in the protected environment using stored rules and providing a means for defining the generation and training of machine learning systems from outside the environment without access to the training data, the systems and methods described herein provide a technical solution to the technical problem of how to provide trained learning systems that protect training data without access to the training data.

[0080] 3. Campaign performance measurement and optimization In certain embodiments, ad serving data can be combined with healthcare data in a measurement server computer to measure the effectiveness of healthcare advertising on patient outcomes and HCP behavior change. This combined data can be further used to measure interactions between consumers and service providers to quantify how a particular ad campaign leads to improved health outcomes. Thus, the DSP provides the ability to plan, launch, and measure healthcare campaigns that present timely information to patients and their HCPs to help them make more informed decisions about their health. Measurement results can then be used to adjust and optimize various parameters of a particular ad campaign to achieve further desired results, thereby enabling control, customization, and communication of individual or integrated campaigns for HCPs and patients.

[0081] Figure 6A illustrates an example of a distributed computer system for implementing certain embodiments, and Figure 6B illustrates the system of Figure 6A including an optimization element.

[0082] In one embodiment, referring initially to FIG. 6A , a distributed computer system 600 includes a DSP environment 610, a medical data repository (MDR) environment 620, and an analysis code space 630. Generally, the DSP environment 610 and the medical data repository (MDR) environment 620, as well as the analysis code space 630, represent various computing domains of individual parties, although in some embodiments they may comprise the same computing device, virtual machine instance, domain, entity, or computing center. The DSP environment 610 and the medical data repository (MDR) environment 620 each include at least one computer, process, processor, or virtual machine instance in a public or private data center, programmed to perform functions described further in the following sections. In the DSP environment 610 and the medical data repository (MDR) environment 620, the computer or virtual machine instance is communicatively coupled to other functional elements depicted within each environment, directly or indirectly, via one or more network links. The analysis code space 630 contains a set of executable instructions that may be executed on the same processor or computer that implements the medical data repository environment 620. In some embodiments, analysis code space 630 is a domain for executing custom code that is initially created by entities associated with DSP environment 610 but that is securely executed within medical data repository environment 620 .

[0083] In one embodiment, the DSP environment 610 includes a measurement server computer that is directly or indirectly communicatively coupled to the server computer 110 and may be implemented using the same server as the server computer 110 or a different server. In one embodiment, the DSP environment 610 includes an impression database 612 that is programmed to store data describing impressions delivered by the DSP in an advertising campaign. In one embodiment, the impression database 612 stores records 613 that include column attributes such as a device identifier (DI), a campaign description, and timestamps for each portion of the campaign (e.g., the time of delivery of various impressions or the start / end times of a stage of the campaign). Each DI may be an anonymized token representing a consumer or other user of the device. In other embodiments, the records 613 may use hashed email or other identifiers linked to impression data associated with the consumer and / or HCP, and a DI is not required.

[0084] In one embodiment, consumer impression data from database 612 is processed using a data mapping process 614 that is programmed to map DI values, which in one example may be denoted as DV(DI)t, to anonymized user device tokens. Output from data mapping process 614, containing tokenized impression data representing online interactions with consumers, is programmatically sent to a data binding operation 632 in an analytics code space 630, further described in other sections herein. In database 612, HCP identifiers 615 are extracted from impressions communicated to HCPs and programmatically sent to data binding operation 632. Examples of HCP identifiers include NPIs, medical education (ME) numbers, or other unique identifiers for particular healthcare providers.

[0085] In some embodiments, MDR environment 620 is owned, operated, and / or managed by an entity separate from the second entity that owns, operates, and / or manages DSP environment 610. In one embodiment, DSP environment 610 may include a measurement database 616 that is programmed to store tracking measurements for advertising campaigns. The generation of measurement data for database 616 is described in other sections herein.

[0086] In one embodiment, the MDR environment 620 may include a claims database 622 that is programmed to store prescription dispensing records derived or replicated from HCPs' medical claims submissions to insurance companies, government agencies, or other payers. In some embodiments, the claims data records include patient personally identifying information, prescription and diagnostic (Rx and Dx) data for multiple patients. By way of example, this Rx and Dx data may include diagnostic (ICD-10), drug (NDC), or medical procedure (CPT) codes, or other values ​​indicative of provider and consumer claims for procedures, prescriptions, or other encounters. Because the database 622 stores personally identifiable information (PII) for the claims data records, the MDR 620 employs high-security techniques to prevent disclosure or unauthorized use of the data.

[0087] The MDR implements proprietary algorithms for mapping PII to de-identification token values, and a computer or process in the DSP environment 610 accesses or calls these algorithms via an application programming interface (API) implemented in the MDR environment. The MDR 620 also implements code to extract HCP identifiers, such as NPIs, from records in the claims database 622. These values ​​do not require tokenization because they identify healthcare providers and do not imply consumer data privacy, but they are useful in later steps for combining records to identify when the same campaign reaches the same provider and consumer and is associated with prescriptions or claims for products related to this campaign. The data mapping process 614 can be programmed to send an API call containing the device identifier value to the MDR environment 620 and to accept a response specifying the mapped de-identification token value. With this approach, in a downstream step, a given DI from the data mapping process 614 can be matched with a de-identification token value created and stored within the MDR environment 620 based on the PII, other valid data, and proprietary algorithms executed within the MDR environment. Additionally, the MDR environment 620 operates to discover the HCP identifier in the claims data and provide it to subsequent steps for use in record joining and matching.

[0088] In one embodiment, within the secure MDR environment 620, the claims database 622 is coupled to a token process 623 that is programmed to de-identify and tokenize claims data and store the de-identified tokenized records in another database 624. HCP identifiers are also extracted from the records in the claims database 622 and stored in the same database 624 as de-identified tokenized claims records, but without tokenization. For example, the records in database 624 may store a token that links the consumer associated with the claim to such claim on an anonymized basis, but may also store the HCP identifier directly without tokenization. Alternatively, the HCP identifier or other attributes of the records in the claims database 622, whether or not they comprise traditionally recognized PII, may be tokenized or otherwise anonymized before being stored in the records in database 624. In one embodiment, the records in database 624 do not contain PII but are encoded with the consumer's token, e.g., DV(D)t, as depicted in Figures 6A and 6B. Other values ​​in the records in database 624 may be the same as those in the claims database 622. The data may be tokenized so that patient information associated with a given code is anonymized to comply with HIPAA and / or other legal requirements.

[0089] In the analytical code space 630, the de-identified tokenized claims database 624 is coupled to a data join operation 632 that can be programmed to perform a join between records from the database 624, the output of the data mapping process 614, and the HCP identifier 615. For example, a data record 634 received from the database 624 via the data join operation 632 can be matched with a record 638 received from the data mapping process 614 via the data join operation 632 and joined based on matching values ​​of keys such as the de-identified token and the HCP identifier 615. The HCP identifier 615, whether or not comprised of conventionally recognized PII, can be tokenized or de-identified prior to joining and matching via the data join operation 632 with the HCP identifier extracted from the claims database 622. The result set is programmatically provided to an analytical operation 636 that can be programmed to perform one or more analytical operations on the result set. The output of the operation 636 is a data set 618 that specifies the campaigns and metadata about the campaigns that reached both the service provider device and the consumer device. The analysis in operation 636 may include calculating audience quality, lift analysis, or other metrics, and creating, training, and validating a machine learning model 640 for export to a DSP 660 as an optimization model 652 for use in the DSP's continuous optimization. In one embodiment, the optimization model 652 may be the same as the machine learning model, except that it has passed validation and has a logical location within the system that enables evaluation of bid request data and use in optimizing the bid portion, as further described in other sections.

[0090] The output database 618 may be exported or otherwise transmitted to a database 616 in the DSP environment 610 and stored as measurements. In the analysis code space 630, the measurements are used to generate an analysis report, which may be used to train a machine learning model 640 that can output predictions and adjust parameters of the associated advertising campaign, which may be loaded into the server computer 110 as an optimization model 652, as further described in other sections with respect to Figures 6B and 7C.

[0091] FIG. 7A illustrates an example of the measurement and management process for a given campaign. FIG. 7B illustrates an example of the measurement and management process for a given campaign. Each of FIGS. 7A and 7B represents a different embodiment of the following general process: A user or account sets up a campaign targeting a specific audience of healthcare providers and patients, or other types of service providers and consumers. The target audience is defined as clinical attributes. The audience is loaded into the DSP and attached to the ad campaign. In one embodiment, each campaign is represented using a data set where one attribute specifies whether the campaign should receive integrated measurement reports. In one embodiment, if the attribute is set to use a report, it is instantiated through the registration process. New report jobs are created and periodically refreshed to update key report statistics and deliver reports by fusing and combining DSP data with clinical data available via the architecture described in FIG. 6. When the campaign ends, a request to the registration API causes the system to stop collecting statistics and updating reports. DSP data is then populated into the environment during periodic process updates. Each report consists of several analytical dimensions ("report pivots"). The data is combined with patient and HCP clinical data, and the combined data set is run through various report analysis processes to update each report pivot for each report. Each time a report is created, it is populated into a working database. An automated process is notified and loads the data into the UI database. The data is made available in a dedicated graphical user interface. Figure 9 shows an example, which is further described in other sections of this specification. In one embodiment, the DSP automatically optimizes campaign targeting, paying more for ads and impressions that perform well, such as low cost per conversion or prescription.

[0092] Referring first to Figure 7A, in one embodiment, the steps or operations of Figure 7A may be implemented using stored program instructions of a computer implementing DSP environment 610 and analysis code space 630. In one embodiment, in step 702, a process is programmed to obtain impression data from a DSP specifying a first campaign set associated with a first set of one or more healthcare attributes, the first campaign set being included in a plurality of diverse campaigns executed by the DSP.

[0093] In step 704, the process is programmed to obtain a plurality of first records from the impression data, each record including anonymized consumer tokens representing consumers who received digital impressions of a first campaign set associated with the same first set of healthcare attributes.

[0094] In step 705, the process is programmed to obtain a second plurality of records from the impression data, the second plurality of records including HCP identifiers representing HCPs that received digital impressions of the first campaign set associated with the first healthcare attribute set.

[0095] In step 706, based on the analytical instructions executed on the data server, the process is programmed to accept a collection of de-identified tokenized patient claim data records, each data record relating to at least one claim for a prescription of a specified product, which are the basis for matching with the records of step 704.

[0096] In step 707, the process is programmed to match the HCP identifier of the record from step 705 with the HCP identifier of claims data records in the set of de-identified tokenized patient claims data records, each of which relates to at least one claim for a prescription for the specified product. The matching records may include some of the records selected in step 706. In this regard, the process matches records of campaigns in which the consumer received an impression with records of campaigns in which the HCP received an impression, for the same product, with claims records relating to the same patient and / or HCP, in a manner that fully protects consumer privacy. These matching steps were not available in previous practice. The ability to tokenize billing data and similarly tokenize impression data in a secure environment, coupled with analytics such as matching tokens associated with impression records to anonymized tokenized billing records, and / or matching HCP identifiers to impression records, optionally in combination with training machine learning models based on the analytical output to predict bids or other DSP parameters that can be automatically updated or modified at the DSP, represents a significant advancement over the technology conceived by the inventors at the time of the invention. The combination presented in this disclosure allows for the extraction of new meaning from data using previously unused computer-implemented operations.

[0097] In step 708, the process is programmed to receive from the database server a result set from one or more integrated measurement records specifying one or more measured campaigns from a diverse plurality of campaigns, the one or more measured campaigns being associated with prescriptions for the specified product in at least one claims data record associated with at least one of the consumer tokens and / or at least one of the HCP identifiers.

[0098] In step 710, the process is programmed to generate and present one or more analytical reports based on the aggregate measurement records. In certain embodiments, one or more steps of the method of Figure 7A may be repeated as appropriate.

[0099] As one specific example of this overall process, two advertising campaigns (one for HCPs and one for patients) may be linked and registered with the MDR environment 620. These campaigns are marketed to deliver ads for new medications to patients with a specific health condition (e.g., COPD) and HCPs who are experts in treating patients with this specific health condition (COPD). Impression data for the advertising campaigns for patients is input into the MDR environment, where analytics instructions may be executed to extract de-identified tokens for patients with a diagnosis code indicating COPD or a drug code / NDC indicating they are receiving medication for COPD. In contrast, because HCP information need not be de-identified, HCPs associated with the COPD condition may be directly identified in the measurement server computer 610 or DSP system 110 by their NPI or other identifier. As an example, the NPI of the targeted HCP may be identified by identifying their work experience / specialty or by prescription / drug codes filled by the NPI for COPD medication. Thus, impression data for an HCP's advertising campaign can be directly injected into the MDR environment 620 with all of the information already determined. As an example, the server computer 110 has already determined that the NPI served the targeted advertisement.

[0100] Once these HCP and patient campaigns are linked, the effectiveness of the two campaigns can be determined with respect to changes in patient / HCP behavior. As an example, the MDR environment may monitor the increase in the number of prescriptions associated with NDC codes for the newly advertised medication. In certain embodiments, this number may be measured at a very granular level. For example, the increase in the number of prescriptions / prescription fills for the newly advertised medication, as noted in the claims database 622 records, may be determined, and impressions may be shown to the patient only, the HCP only, both the patient and HCP, or either or both the patient and HCP, close to the time a particular encounter between these two parties occurred.

[0101] In certain embodiments, this measurement may be updated at regular intervals, such as weekly. In this situation, consumer data in the MDR environment 620 may be refreshed (e.g., new prescriptions with the desired NDC code, new medical codes indicating eligible encounters between the target patient and HCP, or new patient records with a diagnosis code indicating a COPD diagnosis may be determined). Ad coverage may also be determined. As an example, an ad with 62% coverage means that the ad was delivered to 62% of the patients or HCPs of the total recorded encounters. Similarly, impression data for an advertising campaign may be refreshed in the DSP environment 610 (e.g., records of NPIs that received the COPD ad, the number of ads displayed on various websites, and the total payout per impression may be determined). This data may be populated into the analytics code space 630 of the MDR environment 620, where it may be combined with the de-identified consumer data via operation 632. This data may then be aligned with refreshed data in the MDR environment 620, such as the number of prescriptions filled for an NDC code in the past week.

[0102] In one embodiment, data regarding the performance of the advertising campaign may be fed into a machine learning algorithm that determines how to best optimize and update the parameters of the advertising campaign. Referring now to FIG. 6B , in one embodiment, the analysis operation 636 may be coupled to a machine learning model builder that contains an executable script that trains a machine learning model 640 on all valid data in the analysis output of the analytical code space 630. The user data, bid request data, and results of the analysis operation 636 all contribute to the machine learning model builder and its output in the form of the resulting machine learning model 640, which may be initially built within the analytical code space 630 and exported to the DSP environment 610 as part of the optimization model 652.

[0103] In one embodiment, the optimization model 652 is programmed to automatically modify one or more parameters of the DSP 660's campaigns based on real-world clinical data and results derived from the campaigns via the analysis operations 636. The automated modification of campaign parameters results in more efficient media investments in the form of higher return on advertising spend (ROAS) or lower-cost conversions such as prescriptions or procedures. For example, the optimization model 652 may be programmed using a machine learning classifier coupled to a programmatic optimization function that accepts as input bid request data for the DSP 660's existing campaigns and demographic segment data associated with campaigns identified as successful in the analysis operations 636, and output parameter values ​​for the bids and / or segments. For example, the output of the model 652 may drive the pricing or fulfillment of ad requests configured by the DSP 660.

[0104] The optimization model 652, in conjunction with the UI 616, may present a graphical user interface that is programmed to accept optimization inputs, such as user input specifying optimization goals. Example goals include cost per entire prescription, cost per new prescription, cost per first-time prescription, or cost per action (CPA)-based optimization.

[0105] As one example, the change in the number of prescriptions containing a desired NDC code over the past week may be compared against various parameters, such as format, publisher / website, specialty, region, frequency, etc. As one example, if it is determined that impressions from a first website result in a relatively large number of prescriptions being filled for a new COPD medication, while impressions shown on a second website result in relatively few new prescriptions being filled, payments may be automatically shifted to reduce the number of impressions paid for on the second website, while the number of impressions purchased on the first website increases. Similarly, if it is determined that impressions shown in a particular geographic region result in more prescriptions being filled for a new COPD medication, the DSP system may refine the targeting parameters of an advertising campaign to focus the presentation of impressions to patients / HCPs in that geographic location, even when presented on the same website. Thus, the data used to target ads in a first location is the same data used to measure the performance of the campaign, thereby enabling improved precision and optimization of both the reach of desired patients / HCPs and the impact of the campaign itself.

[0106] FIG. 7C illustrates an example of functional elements and data flow in a DSP optimized embodiment.

[0107] In one embodiment, within the DSP environment 610, impression data 612 for impressions related to a campaign is processed through a data mapping operation 614 and communicated along with an HCP identifier 615 to a combine operation 632 within the analytics code space 630. For example, impression log files may be transferred to the analytics code space 630 for secure processing within this environment. In some embodiments, demographic data 772 is also transferred from the DSP environment 610 to the analytics code space 630 in combination with the impression data 612 and includes records describing demographic characteristics of the audience segment and / or consumer represented by the impression data 612. The demographic data 772 may also include values ​​for the publisher website, the time of day the ad is served, or the device type.

[0108] In some embodiments, the impression data 612 and demographic data 772 may be tokenized prior to transfer using commercially available anonymization tokenization software. The transfer of data from the DSP environment 610 to the medical data repository 630 may also include selecting specific fields that are required for model training.

[0109] In one embodiment, the impression data 612 and demographic data 772 that are combined and transferred to the combined data 632 include multiple records, each having values ​​or attributes for: Inventory type ('site', 'app') Type of device ("smartphone", "personal computer", "tablet") Local or non-local publisher (true or false) Creative type ("banner" or "video") Demographic segments (an array of demographic segments for a given impression. The sequence of segment values ​​is “ <segmentid>:1”s) GEO State Inventory ID ZIP3 (3-digit zip code)

[0110] Within the analytics code space 630, the impression data 612 and demographic data 772 are combined with the de-identified tokenized claims data 624 as described in a data combination operation 632. In one embodiment, the de-identified token and / or HCP identifier are used as a common key in the combination operation. The combination operation 632 may result in the creation and storage of a results file.

[0111] In one embodiment, the results file is prepared for training in operation 776. The results file is used as a training dataset to train the machine learning model 640 in operation 778. In some embodiments, operations 776 and 778 form part of the perform analysis operation 636 of Figures 6A and 6B.

[0112] At operation 780, the trained machine learning model 640 undergoes a validation process that may be programmed to verify the trained model's compliance with data privacy or de-identification requirements. For example, validation may address whether the trained model is HIPAA compliant when patient or healthcare data is involved.

[0113] If the validation is successful, the trained machine learning model 640 is exported or transferred to the DSP environment 610, as represented by the optimization model 652 and path 784 in FIG. 7C . In one embodiment, the DSP environment 610 includes a bidding component 774 implemented as a functional element of software and programmed to perform in a manner further described in this and other sections. The bidding component 774 loads the optimization model 652 and uses the trained model to evaluate bid request data 790, resulting in a prediction of a bid value or other parameter of the bidding component 774 that can be used to modify the DSP's operation in connection with a campaign. For example, assume that a consumer or service provider user computer 786 interfaces with a website 788, or other device on which a digital advertisement is placed, in the course of serving a digital advertisement to the website 788, which communicates bid request data 790 to the bidding component 774. The bid request data 790 may be augmented with demographic data 772 and provided to the optimization model 652 for use in model evaluation or implementation to generate a classification or prediction output that predicts the likelihood of conversion, such as the likelihood of a consumer receiving a prescription for a treatment or product and / or the likelihood of a service provider providing a prescription for a treatment or product. Thus, after importing the optimization model 762, the bidding component 774 uses the optimization model 652 to score the incoming bid request data 790 using open real-time bidding (OpenRTB fields) along with the demographic data 772. Based on the predictions of the optimization model 652, the bidding component 774 adjusts bid prices accordingly and targets relevant advertisements served over path 792 to the consumer and / or service provider user computers 786 to optimize for improved conversion or other goals.

[0114] Output predictions from the optimization model 652 are automatically loaded into the bidding section 774 or an administrative user with access to the DSP environment 610 performs reviews through a user interface and manually applies configuration changes to the bidding section 774.

[0115] In some embodiments, the execution of the optimization model 652, which relates to the bid data, the output of predictions, and the configuration of the DSP based on the predictions, may be fully automated using programmatic control. Using this approach, or the approach described above, in embodiments, continuous optimization of the performance of the bidding component 774 may be performed via a feedback loop in which execution of the optimization model 652 on newly received impression data 612 causes rapid updates of bids or other parameters of the bidding component 774 to serve ads to user computers 786 via path 792 to improve goal fit. Thus, the optimization process involves bids being optimized by the optimization model, which reads bid request data and demographic data and has the bidding component determine whether and how much to bid, and then uses the output to serve ads based on these optimization parameters.

[0116] In one particular implementation, the machine learning model 640 trained in operation 778 to ultimately result in the optimized model 652 may implement a gradient-boosted decision tree. Other embodiments may use deep learning neural networks or other forms of machine learning models. For example, in some embodiments, the XGBoost library may be used to implement the model. For this implementation, data preparation in operation 776 may include converting the resulting data into a format suitable for the XGBoost library. In one approach, each record is assigned a label that specifies whether the record is converted based on the presence or absence of a medical record or a business transaction. Binary values ​​may be used. Furthermore, all other data points are represented as integers.

[0117] In one particular implementation, operation 776 may include converting the training dataset into LibSVM format, which may be expressed as label feature1:1 feature2:1. The value label:1|0 is a Boolean flag indicating whether a match was found between the medical transaction record and the given impression record, and may have a value of "1" for a find or "0" for a non-find. LibSVM is just one format supported by the XGBoost library; other embodiments may use other formats to present the test dataset.

[0118] Furthermore, in one implementation, all features are hashed using the murmur3 hashing algorithm, after which a modulus operation is performed to further reduce the number. The hashing operation is murmur3('f01: <value>'), where f01 is the index of the feature, <value>is the value of the feature. As an example, for inventory type="site", the feature is "166685:1", where 166685 is the murmur3 hash of "f01:site". The "murmur3" algorithm is just one example of a hashing algorithm that may be used; other embodiments may use different hashing algorithms.

[0119] One example of a suitable machine learning model is XGBoost, a decision tree-based ensemble algorithm that uses a gradient boosting framework. For prediction problems involving structured or tabular data, decision tree-based algorithms are preferred. In other embodiments, other decision tree models or other machine learning models may be substituted. In one embodiment, the XGBoost library may be configured with the following parameters: eta - Step size reduction used in updates to prevent overfitting max_depth - maximum tree depth num_round - number of rounds of boosting min_child_weight - minimum sum of instance weights (Hessian matrix) required for child nodes Gamma - the minimum loss reduction required to make a further split at a leaf node of the decision tree · objective - specifies the learning task and the corresponding learning objective tree_method - The tree construction algorithm used by XGBoost. XGBoost supports approx, hist, and gpu_hist for distributed training. Experimental validation for external memory is available for approx and gpu_hist. eval_metric - evaluation metric for validation data. A default metric is assigned depending on the objective (rmse for enrollment, logloss for classification, mean precision for ranking). A detailed description of these parameters is provided in the file "parameter.html" at the path " / en / latest / " of the domain xgboost.readthedocs.io. The parameter values ​​are adjusted after observing the training or evaluation results, resulting in a continuous optimization process of DSP660.

[0120] FIG. 7B illustrates an example measurement management process for a given campaign.

[0121] For purposes of a clear example, assume that a user represented by computer, account, or element 732 works with planner 734 of DSP system 110 to define a campaign that targets HCPs and defines the HCP audience. Asynchronously, the DSP's operations or Ops function 736 is used to create a consumer campaign that targets patients. Defining the consumer campaign involves working with patient modeling environment 740 to define the consumer audience. Data representing both the service provider audience and the consumer audience is loaded into DSP 110. The campaign setup process can be performed using, for example, audience modeling and other techniques described above with respect to Figures 1, 2, 3, 4, and 5.

[0122] In block 742, in one embodiment, the process performs a test to determine whether the advertising campaign has enabled the integrated monitoring reporting feature through the DSP system 110. If the feature has not been enabled, the DSP system 110 performs as described above and integrated reporting is not performed. If block 742 is "true" or "yes," control passes to block 744.

[0123] At block 744, the process is programmed to check whether the campaign is new. “New,” in this context, means that data about the campaign has not been posted or registered in the MDR 620. If block 744 evaluates to “true” or “yes,” control passes to block 746, where the new campaign may be registered in the MDR environment 620. Registration, in one embodiment, involves performing an operation to transfer a data set 748 containing data describing the campaign to the MDR environment 620. For example, a POST operation may be used to transfer the data in the form of a JSON blob, parameterized HTTP, or other structured data transfer that includes the data set 748. In one embodiment, the data set 748 specifies one or more identifiers for the organization, the advertiser, one or more HCP campaign groups, one or more target HCP counts, one or more patient campaign groups, one or more patient target attributes, and measurement metrics. Custom code executing in the analysis code space 630 may be programmed to respond to the registration operation and perform the sending of a report or other output definition to the configuration database 751.

[0124] If the test at block 744 is false or no, in one embodiment, control passes to block 750, where the process is programmed to determine whether an analytical report on the performance of the advertising campaign should be updated. Block 750 represents examining a stored plan to check whether the system clock value returns to a time value equal to the report execution date according to the plan. As an example, the DSP measurement server 610 may automatically execute a first iteration of running, measuring, and updating an advertising campaign based on a digitally stored execution plan. Such an execution plan may result in reports being generated on request or at regular intervals, such as weekly, daily, or real-time. Alternatively, the test at block 750 may represent a CRON job or other schedule job that runs according to the plan and triggers a signal to the MDR environment 620 at a specified time.

[0125] If the test at block 750 is false or no, monitoring of the advertising campaign may continue. If the test at block 750 is true or yes, in one embodiment, operations 632 and 636 (FIG. 6) may be performed in which the tokenized impression data and the HCP identifier are sent to the MDR environment 620 and combined with the tokenized billing data record containing the HCP identifier from database 624. In some embodiments, metadata about the impression is also sent for use in the combining operation. In one embodiment, as noted at blocks 752 and 753, a report of campaign impression data resulting from actual impressions to service providers and / or consumers may be generated and sent or posted to the MDR environment 620. For example, for one or more measured output campaigns, the impression data set of the first impression associated with a particular campaign of the multiple campaigns may be updated to generate an updated first impression. A set of anonymized patient tokens representing consumers associated with the first set according to one or more healthcare attributes may be appended to the updated first impression, and the particular campaign may be updated based on the updated first impression and the appended set of anonymized consumer tokens. One or more second iterations of running, measuring, and updating the advertising campaign based on the digitally stored execution plan may then be performed using the newly received campaign impression data from the impression database 612. In particular embodiments, the impression data set may be updated, and updated first impression data may be generated by a machine learning algorithm, such as one implemented by an artificial neural network, as described below with respect to FIG. 10. As detailed in block 752, the update process may be programmed to obtain campaign impression data, for example, from a scratchpad database 756, tokenize, anonymize, combine the campaign impression data with clinical / patient data, run reporting / analysis routines, and send or populate updated reports to the scratchpad database 756 as further represented in block 754.

[0126] In one embodiment, at block 758, the report data is processed for presentation and loaded for storage into a user interface (UI) database 760. At block 762, the stored analysis reports can be retrieved and presented via a user interface.

[0127] In certain embodiments, the measurement server computer obtains impression data from a demand-side platform (DSP) specifying a plurality of first anonymized consumer tokens representing consumers who received digital impressions of a first campaign set associated with a first set of one or more healthcare attributes, the first campaign set being included in a plurality of diverse campaigns executed by the DSP. Each anonymized consumer token may be linked to a first healthcare attribute of the first healthcare attribute set and may further be linked to a first anonymized tokenized claim data record from a set of anonymized tokenized claim data records. Furthermore, each campaign of the first campaign set may be defined by the DSP using at least one clinical attribute of the first healthcare attribute set. By way of example, each clinical attribute may be an ICD-10 code, a CPT code, or an NDC code.

[0128] In certain embodiments, the measurement server computer may obtain impression data from the DSP specifying a plurality of second healthcare provider (HCP) identifiers representing healthcare providers (HCPs) that received digital impressions of a first campaign set associated with a first set of healthcare attributes. As an example, the HCP identifiers may be National Healthcare Professional Identifier (NPI) values.

[0129] In certain embodiments, the metering server computer receives, based on analysis instructions executed on the database server, a collection of de-identified tokenized claim data records, each of the data records relating to at least one claim for a prescription of a specified product.

[0130] Using the above approach, large data sets are successively scrutinized into smaller data sets representing service providers and / or consumers who qualify for, are exposed to, and result in conversions for the campaign. Figure 8 illustrates an example of the report setup process for a given campaign where successive filtering and flow-down of audiences occurs, resulting in actionable performance measurements.

[0131] In one embodiment, the campaign definition and audience definition process described above with respect to FIG. 7B may initially result in creating a measurement objective 802 for the campaign, a first data set of eligible patients 804, and a second data set of eligible HCPs 810. In one embodiment, the measurement may specify a particular drug in terms of one or more NDC values ​​that identify the drug for national regulatory or claims purposes. In the example of FIG. 8, measurement objective 802 specifies SYNJARDY, identified by NDC values ​​of 0597-0295-60 and 0597-0295-61. Further, in the example of FIG. 8, eligible patients 804 include patients identified in diagnostic visits where the HCP codes claims using the ICD10 value "E11," resulting in a patient population of 1.2 million in one test run example. Eligible HCPs 810 are typically derived from a target list from informal or proprietary sources and may include 10,000 or more HCP identifiers.

[0132] The eligible patient data set 804 is first filtered to the number of linked patients 806, which means patients who are eligible and are associated with a particular token that is further linked to data from database server 620. The linked patients 806 are again filtered to identify exposed patients 808, which means patients who are known to have been exposed to the ad as indicated by exposure data from the DSP. In this example, there are approximately 500,000 identifiers in the exposed patient data set 808.

[0133] The eligible HCP database 810 is filtered to obtain approximately 6,000 exposed HCPs 812 .

[0134] Additionally or alternatively, data from the first campaign set may be obtained. By way of example, this data may include the number of eligible HCPs (HCPs with a targeted NPI as determined by claims data, prescription data, demographic data, work experience / professional data, or other relevant data described with respect to FIG. 1) and the number of exposed HCPs (HCPs exposed to the ad).

[0135] Next, via the data join operations previously described, a qualified encounter dataset 815 is formed to measure the total number of interactions between qualified patients 804 and qualified HCPs 810, resulting in the creation and storage of qualified encounter dataset 814. In the example of Figure 8, dataset 814 would have some number of HCPs less than 10,000, representing all HCPs and patients who interacted in the same clinical encounter.

[0136] The resulting data set can be transformed again to capture a qualified exposure count value 816 indicating the total number of times either qualified patients or qualified HCPs were exposed to the campaign associated with the measurement objective 802 near the time the interaction occurred.

[0137] Finally, an exposure conversion count value 818 may be formed that indicates the total number of times a patient was prescribed and treated, or an HCP or service provider filled a prescription or administered a treatment, for the product associated with the measurement objective. In the example of Figure 8, value 818 would include a collection of eligibility and exposure records also coded with one of the NDC codes for SYNJARDY, or any other item identified in measurement objective 802.

[0138] In certain embodiments, the measurement server computer receives from the database server a result set of one or more aggregate measurement records identifying one or more measured campaigns from a diverse plurality of campaigns, the one or more measured campaigns being associated with prescriptions for specified products in at least one claim data record associated with at least one consumer token and / or at least one HCP identifier.

[0139] In certain embodiments, the measurement server computer generates and presents one or more analytical reports based on the integrated measurement records. In certain embodiments, a result set may be generated based on analytical instructions that execute one or more database join operations on the claims data records, the consumer token, the impression metadata, and the HCP identifier in the database server / MDR environment 620. As one example, the generation and presentation of the one or more analytical reports may include measuring the total number of interactions between the consumer token and the HCP identifier in qualified visits. As another example, the generation and presentation of the one or more analytical reports may include measuring the total number of interactions between the consumer token and the HCP identifier in qualified visits when the specific consumer token or the specific HCP identifier was exposed to a particular campaign among a plurality of diverse campaigns. As yet another example, the generation and presentation of the one or more analytical reports may include measuring the total number of interactions between the consumer token and the HCP identifier when the specific consumer token or the specific HCP identifier was exposed to a particular campaign among a plurality of diverse campaigns, where a particular product of the particular campaign is specified in the claims data records associated with the particular consumer token.

[0140] As yet another example, generating and presenting one or more analytics reports may include generating a data value indicating the elapsed time between a timestamp of a first digital impression associated with a particular campaign of the diverse plurality of campaigns and a first measured campaign of the particular campaign. This value allows an advertising campaign sponsor to determine the effectiveness and importance of various frequency levels of ads delivered to HCPs and patients prior to treatment with a target treatment. Presentation of this elapsed time from the first impression to the first measured target allows the sponsor to understand what type of frequency is required to affect results.

[0141] FIG. 9 illustrates an example of a GUI display for presenting an analytical report of a campaign being measured.

[0142] In one embodiment, GUI 900 includes a status panel 902, a filter panel 904, a summary panel 906, a graph panel 908 containing a graph 910, an analytics table 912, a funnel data panel 914, and a toolbar 916. In one embodiment, status panel 902 is programmed to display one or more metadata values ​​related to other aspects of GUI 900, such as approval status, audience type, start date, end date, creation or update date, and / or other metadata. Example values ​​can be seen in status panel 902 in FIG. 9 .

[0143] In one embodiment, the filter panel 904 comprises a GUI widget programmed to accept a selection of filter criteria for the data represented by elements 908, 910, and 912 and apply the specified filter criteria to the data. In the example of Figure 9, a campaign filter specifying "TODAY" is selected, and in response, the system is programmed to filter the data for only the current day and present the filtered data in the other elements of the GUI 900. In one embodiment, the summary panel 906 includes a count value specifying total impressions to consumers, total impressions to HCPs, a payment value, and a cost value. In one embodiment, the execution analysis element 636 of Figures 6A and 6B is programmed to perform calculations within the analysis code space 630 to generate the underlying data for the elements of Figure 9.

[0144] In one embodiment, the graph panel 908 contains one or more GUI widgets that, when selected, are programmed to generate or update the graph 910. In the example of Figure 9, the GUI widgets specify TRx and NRx as metrics, "Daily view," and grouping by exposed HCP and patient. The graph 910 reflects these parameters. In one embodiment, selection of various GUI widget options causes the graph 910 to be recalculated and updated.

[0145] In one embodiment, the analytics table 912 includes multiple selectable tabs labeled "Exposures," "Campaign Groups," and other labels in this example. The labels are programmed as executable links that, when selected, cause various data to be displayed in the table 912. In the example of FIG. 9, the "Exposures" label is selected, and the table 912 includes rows specifying exposure groups and analytical data related to those groups. The funnel data panel 914 can be programmed to display successively narrower or smaller counts for individuals, accounts, or records based on various analytical criteria and calculations such as exposure counts, treatment opportunity counts, and conversion counts. The toolbar 916 can be programmed to display various multiple selectable display tools that, when selected, cause the system to perform functions such as changing the graph type (line graph, bar graph, scatter plot, etc.), applying a display zoom, etc.

[0146] Based on programmatic analytical calculations performed using the aforementioned functional elements, the data in GUI 900 aggregates patient and provider prescription fill behavior data across various dimensions (e.g., creative, publisher, format, audience), thus enabling buyers running advertising campaigns to visualize the merits and performance of various ad serving dimensions for further optimization. The data reported in GUI 900 can also be used as a training data set to train one or more machine learning models to predict bids or other values ​​to update one or more elements of DSP 660, such as bidding section 774 (FIG. 7C), and automate campaign optimization. One example of a benefit of campaign optimization is making campaigns more effective by reducing ad costs in publishers or regions that do not have a large number of active patients.

[0147] The panels of the GUI 900 can be programmed to display the report dimensions and analyses as a scorecard. As an example, any of the variables above and / or the following analyses are shown: HCP Impressions: The number of impressions delivered to the HCP campaign linked to the measurement report. HCPs exposed (and % achieved): (from NPI level report) number of HCPs reaching the ad and % of HCPs reached from the initial target list (% reached). HCP Ad Cost: The total cost of ads conveyed to the HCP campaign linked to the measurement report. CPM-HCP: The unit cost of an ad conveyed to the HCP campaign linked to the measurement report. Patient Impressions: Total impressions delivered to the patient campaign linked to the measurement report. Exposed Patients (and Reach % and Linked %): Number of patients exposed to the ad (linked to data from the database server). Patient Ad Cost: The total cost of impressions delivered to the patient campaign linked to the measurement report. CPM-Patient: Unit cost of the ad communicated to the patient.

[0148] In these analyses, "total patients" may be defined as the number of unique patients represented by anonymized consumer tokens who are "eligible" or "selected" as having the target ICD-10, CPT, or NDC code. In certain embodiments, this value may be calculated via Boolean logic provided during campaign setup, such as "TARGET_ATTRIBUTES." This value may be further used to calculate the "Population %" metric. Similarly, "linked patients" may be defined as the number of patients who are eligible and further associated with a particular token who are linked to data from a database server. This value may be used to calculate the "Linked %" metric. In certain embodiments, if an advertising company only wishes to measure patient campaigns, the HCPs for eligible HCPs may be used in further calculations, but the "Exposed HCPs" would be set to 0.

[0149] The panel can be programmed to display a report means for each analysis report, including any of the following: Total Cost: The total ad cost from the DSP for the impressions purchased. Impressions: The total impressions from the DSP. Clinical Opportunity: An instance where an eligible patient and eligible HCP were observed interacting within the claims data (e.g., a patient with type 2 diabetes filed a claim with a targeted HCP). Total Prescriptions (TRx): The total number of prescriptions filled by an HCP for a particular drug over a specified time period. This value may include prescriptions for refills and renewals (prescriptions that patients obtain when their refills run out). In contrast, the NRx values ​​below do not include refills but may include renewals. New Prescriptions (NRx): A count of new prescriptions issued. NRx does not include refills, but may include renewals. NRx differs from the NBRx value below in that the NRx metric does not consider whether the patient was previously using the product. First-Time Prescriptions (NBRx): A count of patients who start taking a prescription drug without previous use of that product. Cost Per TRx: The total advertising cost paid (by the DSP) for an impression divided by TRx. Cost Per NRx: The total advertising cost paid (by the DSP) for an impression divided by NRx. Cost Per NBRx: The total advertising cost paid (by the DSP) for an impression divided by NBRx. Total Care Occasions: The number of times a patient (out of total patients) and a target HCP appear together in claims data records from database server 620 (indicating an encounter or some type of opportunity where a medication is prescribed). In some embodiments, "total care occasions" can be programmed as the number of times a clinically relevant consumer and a targeted service provider appear together in claims data records and / or the number of times a targeted clinically relevant consumer and a clinically relevant service provider appear together in claims data records. Exposed Care Opportunities: The number of times a care opportunity occurred and either the HCP or patient was exposed to the ad prior to the timestamp of the care opportunity. Total Conversions: The total number of conversion events (from measurements) linked to users targeted by your ad, along with the total cost of those conversions (the sum of ad spend across all users divided by the TRx amount). New Conversions: The total number of new conversion events (from measurement) linked to users targeted by your ad, along with the total cost of these conversions (the sum of ad spend across all users divided by the NRx amount). First-time conversions: The total number of first-time conversions (from measurement) linked to users targeted by your ad, along with the total cost of those conversions (total ad spend across all users divided by NRx amount).

[0150] The report table may display any of the following analyses: Exposure: A report showing the relative effectiveness of the ad by exposed HCP and patient group. This report may include any of the following analyses: Exposed HCP and Patient: A clinical opportunity where both an eligible / target HCP and an eligible patient exposed to the ad. Exposed Patients Only: Clinical encounters where only eligible patients were exposed. Exposed HCPs Only: Clinical encounters where only eligible HCPs were exposed. No Exposure: A clinical encounter in which neither the Eligible HCP nor the Eligible Patient was exposed. Campaigns: Reports showing the performance of various campaigns and the users exposed to ads from these campaigns. Format: A report showing the relative performance of various inventory formats sourced by inventory type (banner, video, CTV, audio, native, etc.). Device: A report showing the relative performance of various device impressions sourced by bidRequest.device.type. Inventory: Reports showing the relative performance of various types of publisher inventory (region-specific and non-region-specific), individual publishers, etc. Audience: A report showing the relative performance of users within various audiences (e.g., one report for patients and another for HCPs). In certain embodiments, this report may be sourced from the dataCharge of impressions. Frequency: A report that provides guidance on the importance of various frequency levels of ads given to HCPs and patients prior to treatment with the target procedure, where the elapsed time from first impression to first measured result helps buyers understand what type of frequency is needed to impact results in various audiences (such as patient audiences vs. HCP audiences). Creative: A report that provides guidance on the relative performance of diverse creative outcomes drivers. Physician: Reports showing individual physician level outcomes / behaviors. This is similar to the NPI report but may more clearly show outcomes at the individual physician level. Expertise: Reports showing the performance of diverse experts when HCP campaigns are effective. Patients: Reports showing performance for various demographic segments along with various model thresholds. Regional: A report showing the performance of each DMA along with each state.

[0151] These measurements may be used to determine various outcomes of advertising campaigns. In certain embodiments, the measurement server computer may determine changes in prescription filling behavior associated with the consumer token and HCP identifier based on the consolidated measurement record and present the changes in an analytics report. Subsequently, through the described optimization process, the measurement server computer may automatically adjust one or more parameters of the DSP for a particular campaign of a diverse plurality of campaigns based on the changes. As an example, if it is determined that purchase impressions on a first website lead to a greater change in prescription filling behavior than impressions on a second website, the DSP may automatically shift funding for the advertising campaign from the first website to purchase more impressions on the second website. In certain embodiments, the measurement server computer may determine the cost of each of the one or more measured campaigns based on the consolidated measurement record and, in response, automatically send a signal to the DSP to modify one or more configuration parameters to increase the cost of at least one of the measured campaigns having the lowest prescription cost for a particular product. As an example, if an impression on a first website results in a conversion and therefore lower revenue than the cost of purchasing the impression on the first website (purchasing the impression on the first website effectively loses money on the advertising campaign), the DSP may automatically shift payments to focus more heavily on the second website that shows impressions that result in greater revenue than would have been paid for purchasing the impression on the second website. In some embodiments, performing optimizations can include updating parameters to pay more for particular campaigns, particular devices, particular times of day, particular publishers, or other attributes.

[0152] 4. Implementation Example 4.1 Computer System Overview According to various embodiments, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and / or other computing devices coupled using a network, such as a packet data network. The computing device may be hardwired to perform the techniques, may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) that are permanently programmed to perform the techniques, or may include at least one general-purpose hardware processor that is programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such a computing device may combine custom hardwired logic, ASICs, or FPGAs with custom programming to achieve the described techniques. The computing device may be a server computer, a workstation, a personal computer, a portable computing system, a handheld device, a mobile computing device, a wearable device, a body-worn or embedded device, a smartphone, a smart appliance, an internetworking device, an autonomous or semi-autonomous device such as a robot or an unmanned ground or air vehicle, other electronic device incorporating hardwiring and / or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0153] Figure 5 is a block diagram illustrating an example computer system in which one embodiment may be implemented. In the example of Figure 5, computer system 500 and instructions for implementing the disclosed technology in hardware, software, or a combination of hardware and software are shown schematically, for example as boxes or circles, at a level of detail commonly used by those skilled in the art to which this disclosure pertains to communicate computer architecture and computer system implementations.

[0154] Computer system 500 includes an input / output (I / O) subsystem 502, which may include buses and / or other communication mechanisms for communicating information and / or instructions between components of computer system 500 via electronic signal paths. I / O subsystem 502 may include an I / O controller, a memory controller, and at least one I / O port. Electronic signal paths are represented schematically in the drawings, for example, as straight lines, single-headed arrows, or double-headed arrows.

[0155] At least one hardware processor 504 is coupled to the input / output subsystem 502 for processing information and instructions. The hardware processor 504 may include, for example, a general-purpose microprocessor or microcontroller, and / or an embedded system or a graphics processing unit (GPU), or a special-purpose microprocessor such as a digital signal processor or ARM processor. The processor 504 may include an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0156] Computer system 500 includes one or more units of memory 506, such as main memory, coupled to input / output subsystem 502 for electronically and digitally storing data and instructions executed by processor 504. Memory 506 may include volatile memory, such as various forms of random access memory (RAM) or other dynamic storage devices. Memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions executed by processor 504. When such instructions are stored on a non-transitory computer-readable storage medium accessible to processor 504, computer system 500 may be embodied as a special-purpose machine customized to perform the operations specified in the instructions.

[0157] The computer system 500 further includes non-volatile memory, such as read-only memory (ROM) 508 or other static storage device coupled to the input / output subsystem 502 for storing information and instructions for the processor 504. The ROM 508 may include various forms of programmable ROM (PROM), such as erasable programmable read-only memory (EPROM) or electronically erasable programmable read-only memory (EEPROM). A unit of persistent storage 510, including various forms of non-volatile RAM (NVRAM), such as FLASH® memory, solid-state storage, magnetic or optical disks, such as CD-ROMs or DVD-ROMs, may be coupled to the input / output subsystem 502 for storing information and instructions. The storage device 510 is one example of a non-transitory computer-readable medium that may be used to store instructions and data that, when executed by the processor 504, cause a computer-implemented method to perform the techniques herein.

[0158] The instructions in the memory 506, ROM 508, or storage device 510 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may include operating system and / or system software, one or more libraries supporting multimedia, programming, or other functions, data protocol instructions or stacks implementing TCP / IP, HTTP, or other communication protocols, file processing instructions for interpreting and translating files coded using HTML, XML, JPEG, MPEG, or PNG, user interface instructions for translating or interpreting commands for a graphical user interface (GUI), command line interface, or text user interface, application software such as an office suite, Internet access application, design and manufacturing application, graphics application, audio application, software engineering application, educational application, game, or miscellaneous application. The instructions may implement a web server, a web application server, or a web client. The instructions may be organized as a presentation layer, an application layer, a data storage layer such as a relational database system, using Structured Query Language (SQL) or non-SQL, an object store, a graph database, a flat file system, or other data storage device.

[0159] The computer system 500 may be coupled to at least one output device 512 via the input / output subsystem 502. In one embodiment, the output device 512 is a digital computer display. Examples of displays that may be used in various embodiments include a touchscreen display, a light-emitting diode (LED) display, a liquid crystal display (LCD), or an electronic paper display. The computer system 500 may include other types of output device 512 alternatively, or in addition to a display device. Examples of other output devices 512 include a printer, a ticket printer, a plotter, a projector, a sound or video card, a speaker, a buzzer or piezoelectric device or other audible device, a lamp or LED or LCD indicator, a tactile device, an actuator, or a servo.

[0160] At least one input device 514 is coupled to the input / output subsystem 502 for communicating signals, data, command selections, or gestures to the processor 504. Examples of input device 514 include touchscreens, microphones, still and video digital cameras, alphanumeric or other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and / or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors, and / or various types of transceivers such as wireless, radio frequency (RF) or infrared (IR) transceivers, such as cellular, WiFi, and global positioning system (GPS) transceivers.

[0161] Another type of input device is the control device 516, which may alternatively or additionally perform cursor control or other automated control functions, such as navigating a graphical interface on a display screen. The control device 516 may be a touchpad, mouse, trackball, or cursor direction keys for communicating directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. The input device may have at least two degrees of freedom along two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify a position on a plane. Another type of input device is a wired, wireless, or optical control device, such as a joystick, wand, console, steering wheel, pedals, gear shift mechanism, or other type of control device. The input device 514 may include a combination of various input devices, such as a video camera and a depth sensor.

[0162] In another embodiment, computer system 500 may include an Internet of Things (IoT) system that omits one or more of output device(s) 512, input device(s) 514, and control device(s) 516. Alternatively, in such an embodiment, input device(s) 514 may include one or more cameras, motion detectors, thermometers, microphones, earthquake detectors, other sensors or detectors, measurement devices, or encoders, and output device(s) 512 may include a dedicated display, such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator, or a servo.

[0163] Computer system 500 is a mobile computing device, and input device 514 may include a Global Positioning System (GPS) receiver coupled to a GPS module capable of determining and generating geolocation or position data, such as triangulation of multiple GPS satellites, latitude-longitude values ​​of the geophysical location of computer system 500. Output device 512 may include hardware, software, firmware, and interfaces for generating position report packets, notifications, pulse or heartbeat signals, or transmitting other repetitive data specifying the location of computer system 500, alone or in combination with other application-specific data, directed to host 524 or server 530.

[0164] Computer system 500 may implement the techniques described herein using customized hardwired logic, at least one ASIC or FPGA, firmware and / or program instructions, or other logic that, when loaded and used or executed in combination with a computer system, causes or programs the computer system to operate as a special-purpose machine. According to one embodiment, the techniques described herein are performed by computer system 500 when processor 504 executes at least one sequence of at least one instruction stored in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions stored in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0165] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular fashion. Such storage media may include non-volatile and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as memory 506. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical or physical data storage media, memory chips, and the like.

[0166] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media involves transferring information to or from storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a bus of I / O subsystem 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0167] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a communications link, such as a telephone line using fiber optic or coaxial cable or a modem. A modem or router locally connected to computer system 500 may accept the data over the communications link and convert the data for reading by computer system 500. For example, a receiver such as a radio antenna or infrared detector may accept the data carried in a radio or optical signal and appropriate circuitry may provide the data to input / output subsystem 502, such as by placing the data on a bus. I / O subsystem 502 carries the data to memory 506, from which processor 504 retrieves and executes the instructions. The instructions received by memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0168] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides a two-way data communication coupling to a network link 520 that is directly or indirectly connected to at least one communications network, such as a network 522 or a public or private cloud over the Internet. For example, communication interface 518 may be an Ethernet networking interface, an Integrated Services Digital Network (ISDN), a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of communications line, such as an Ethernet cable or any type of metallic or fiber optic cable or telephone line. Network 522 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, an interconnected network, or a combination thereof. Communication interface 518 may include a LAN card that provides a data communication connection to a compatible LAN, or a cellular radiotelephone interface wired to transmit and receive cellular data in accordance with cellular radiotelephone wireless networking standards, or a satellite radio interface wired to transmit and receive digital data in accordance with satellite wireless networking standards. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

[0169] Network link 520 typically provides electrical, electromagnetic, or optical data communication to other data devices, directly or through one or more other networks, for example using satellite, cellular, Wi-Fi, or Bluetooth technology. For example, network link 520 may provide a connection to a host computer 524 through network 522.

[0170] Further, network link 520 may provide a connection to network 522 or to other computing devices through internetworking devices and / or computers operated by an Internet Service Provider (ISP) 526. ISP 526 provides data communication services through a worldwide packet data communication network represented as Internet 528. Server computer 530 may be coupled to Internet 528. Server 530 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer running a containerized programming system such as DOCKER or KUBERNETES. Server 530 may be implemented using more than one computer or instance and may represent an electronic digital server that is accessed and used by sending web service requests, uniform resource locator (URL) strings with HTTP payload parameters, API calls, application service calls, or other service calls. The computer system 500 and the server 530 may form elements of a distributed computing system that includes other computers, processing clusters, server farms, or other arrangements of computers that cooperate to perform tasks or run applications or services. The server 530 may contain one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps.The instructions may include operating system and / or system software, one or more libraries supporting multimedia, programming, or other functionality, data protocol instructions or stacks implementing TCP / IP, HTTP, or other communication protocols, file format processing instructions for interpreting or translating files coded using HTML, XML, JPEG, MPEG, or PNG, user interface instructions for translating or interpreting commands for a graphical user interface (GUI), a command line interface, or a text user interface, application software such as an office suite, Internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or miscellaneous applications. Server 530 may include a web application server with a presentation layer, an application layer, and a data storage layer, such as a relational database system, using Structured Query Language (SQL) or non-SQL, an object store, a graph database, a flat file system, or other data storage.

[0171] Computer system 500 can send messages and receive data and instructions, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518. The received code may be executed by processor 504 as it is received and / or stored in storage device 510, or other non-volatile storage for later execution.

[0172] Execution of the instructions described in this section may implement a process in the form of an instance of a computer program that is executed and consists of program code and its current activity. Depending on the operating system (OS), a process may consist of multiple threads of execution that execute instructions simultaneously. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of these instructions. Several processes may be associated with the same program. For example, launching several instances of the same program often means that more than one process is running. Multitasking may be implemented, allowing multiple processes to share the processor 504. Although each processor 504 or processor core executes a single task at a time, the computer system 500 may be programmed to implement multitasking operations that switch each processor between executing tasks without having to wait for each task to finish. In one embodiment, a switch may be performed when a task performs an input / output operation, when a task indicates that it has been switched, or upon a hardware interrupt. Time sharing may be implemented to enable fast response for interactive user applications by rapidly performing context switches to provide the appearance of simultaneous execution of multiple processes simultaneously. In one embodiment, for security and reliability, the operating system may prevent direct communication between independent processes and provide strictly mediated and controlled inter-process communication facilities.

[0173] 4.2 Artificial Neural Networks FIG. 10 illustrates an example artificial neural network ("ANN") 1100. In particular embodiments, an ANN may refer to a computational model comprising one or more nodes. The example ANN 1100 may comprise an input layer 1110, hidden layers 1120, 1130, and 1140, and an output layer 1150. Each layer of the ANN 1100 may comprise one or more nodes, such as node 1105 or node 1115. In particular embodiments, each node of the ANN may be connected to another node of the ANN. By way of example and not limitation, each node of the input layer 1110 may be connected to one or more nodes of the hidden layer 1120. In particular embodiments, one or more nodes may be bias nodes (e.g., nodes of a layer that are not connected to any nodes of a previous layer and do not receive input from them). In particular embodiments, each node of each layer may be connected to one or more nodes of a previous or next layer. 10 depicts a particular ANN including a particular number of layers, a particular number of nodes, and particular inter-node connections, any suitable number of layers, any suitable number of nodes, and any suitable inter-node connections are contemplated within this disclosure. By way of example and not limitation, while FIG. 10 depicts connections between each node in the input layer 1110 and each node in the hidden layer 1120, one or more nodes in the input layer 1110 may be unconnected to one or more nodes in the hidden layer 1120.

[0174] In particular embodiments, the ANN may be a feedforward ANN (e.g., an ANN that does not contain cycles or loops in which communication between nodes proceeds in one direction, starting from the input layer and proceeding to successive layers). By way of example and not limitation, the inputs to each node in the hidden layer 1120 may include the outputs of one or more nodes in the input layer 1110. As another example and not by way of limitation, the inputs to each node in the output layer 1150 may include the outputs of one or more nodes in the hidden layer 1140. In particular embodiments, the ANN may be a deep neural network (e.g., a neural network with at least two hidden layers). In particular embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN with hidden layers organized into residual blocks. The inputs to each residual block after the first residual block may be correlated to the output of the previous residual block and the input of the previous residual block. By way of example and not limitation, the input to residual block N is F(x)+x, where F(x) is the output of residual block N-1 and x is the input to residual block N-1. Although particular ANNs are described in this disclosure, any suitable ANN is contemplated in this disclosure.

[0175] In particular embodiments, an activation function may correspond to each node of the ANN. The activation function of a node may define the output of the node for a given input. In particular embodiments, the input to a node may comprise a set of inputs. By way of example and not by way of limitation, the activation function may be an identity function, a binary step function, a logistic function, or any other suitable function. As another example and not by way of limitation, the activation function for node k may be a sigmoid function Fk(sk)=1 / (1+e -sk ), the hyperbolic tangent function Fk(sk)=(e sk -e -sk ) / (e sk +e -sk ), rectifier F k (s k )=max(0,s k ), or some other suitable function F k (s k ) and s k may be a valid input to node k. In particular embodiments, the inputs of activation functions corresponding to nodes may be weighted. Each node may generate an output using a corresponding activation function based on the weighted inputs. In particular embodiments, each connection between nodes may be associated with a weight. By way of example and not by way of limitation, connection 1125 between node 1105 and node 1115 may have a weighting factor of 0.4, which may indicate that 0.4 times the output of node 1105 is used as an input to node 1115. As another example and not by way of limitation, the output y of node k may be k Yes k =F k (s k ) and F k is the activation function corresponding to node k, and sk=Σ j (w jk x j ) is a valid input to node k, and x j is the output of node j connected to node k, and w jk may be a weighting coefficient between node j and node k. In particular embodiments, inputs to nodes in the input layer may be based on vectors representing objects. Although this disclosure describes particular inputs and outputs of nodes, this disclosure contemplates any suitable inputs and outputs of nodes. Also, although this disclosure describes particular connections and weights between nodes, this disclosure contemplates any suitable connections and weights between nodes.

[0176] In particular embodiments, the ANN may be trained using training data. By way of example and not limitation, the training data may include inputs to the ANN 1100 and expected outputs. As another example and not limitation, the training data may include vectors, each representing a training object, and a predicted label for each training object. In particular embodiments, training the ANN may include modifying weights associated with connections between nodes of the ANN by optimizing an objective function. By way of example and not limitation, a training method (e.g., conjugate gradient, gradient descent, stochastic gradient descent) may be used to backpropagate a sum-of-squares error, measured as the distance between vectors representing the training objectives (e.g., using a cost function that minimizes the sum-of-squares error). In particular embodiments, the ANN may be trained using a dropout technique. By way of example and not limitation, one or more nodes may be temporarily omitted during training (e.g., by not accepting inputs and not producing outputs). For each training objective, one or more nodes of the ANN have some probability of being omitted. The nodes that are omitted for a particular training objective may differ from the nodes that are omitted for other training objectives (e.g., nodes may be temporarily omitted on an objective basis). Although this disclosure describes training an ANN in a particular manner, this disclosure contemplates training an ANN in any suitable manner.

[0177] 5. Extensions and Substitutions The foregoing specification describes embodiments of the disclosure with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the disclosure, and what Applicant intends as the scope of the disclosure, is the literal scope of the following claims and the equivalent scope of claims issuing from this application in particular by such claims, including any subsequent amendments. [Appendix 1] 1. A computer-implemented method comprising: acquiring, using a measurement server computer, impression data specifying a first campaign collection, the first campaign collection being included in a plurality of diverse campaigns executed by the DSP; using the measurement server computer to obtain a first plurality of records including anonymized consumer tokens representing consumers who received digital impressions of the first campaign set; using analytical instructions executed on a database server to access a database containing de-identified tokenized claim data records, each of said data records relating to at least one claim for a prescription for a specified product and containing at least one of said consumer tokens; using the analysis instructions to perform one or more database join operations on the billing data records and consumer tokens to output a result set of one or more integrated measurement records specifying one or more measurement target campaigns among the diverse plurality of campaigns, the one or more measurement target campaigns being associated with the prescription for the specified product in at least one billing data record associated with at least one of the consumer tokens; receiving, at the measurement server computer, the integrated measurement record from the database server; generating and presenting one or more analysis reports using the measurement server computer; using the measurement server computer to perform a post call to the database server to register new campaign impression data with the database server; A method that encompasses [Appendix 2] 2. The method of claim 1, further comprising: determining, based on the integrated measurement record, a change in prescription filling behavior associated with one of the advertisements delivered for the measurement target campaign, the measurement target campaign being associated with the consumer token; and presenting the change in the analysis report. [Appendix 3] 2. The method of claim 1, further comprising training a machine learning model using a training dataset including features selected from the plurality of records and the anonymized tokenized billing data records to create an optimization model, wherein the machine learning model is trained to accept other aggregate measurement records for other campaigns and to output a predicted bid value for a target campaign from the diverse plurality of campaigns for use in automatically adjusting one or more parameters of the DSP. [Appendix 4] 4. The method of claim 3, including using the optimization model to determine the cost of each of the one or more measured campaigns based on the consolidated measurement records, and in response, automatically signaling the DSP to modify one or more configuration parameters to increase payments for at least one of the measured campaigns having the lowest cost for prescriptions of the particular product. [Appendix 5] 4. The method of claim 3, further comprising: using the optimization model to receive bid request data and a set of attribute data including personal and demographic data and output a predicted bid value; updating a bidding section of the DSP using the predicted bid value output from the optimization model; and serving one or more advertisements from the DSP based on the updated bidding section. [Appendix 6] when executed by one or more measurement calculation server devices, using the measurement server computer to obtain impression data specifying a first campaign set, the first campaign set being included in a plurality of diverse campaigns executed by a DSP; using the measurement server computer to obtain a plurality of records including healthcare provider (HCP) identifiers representing HCPs that received digital impressions of the first campaign set; using analytical instructions executed on a database server to access a database containing de-identified tokenized claims data records, each of said data records including at least one of said HCP identifiers in association with at least one claim for a prescription of a designated product; using the analysis instructions to perform one or more database join operations on the claims data records and HCP identifiers to output a result set of one or more integrated measurement records specifying one or more measurement target campaigns among the diverse plurality of campaigns, the one or more measurement target campaigns being associated with the prescription for the specified product in at least one claims data record associated with at least one of the HCP identifiers; receiving, at the measurement server computer, the integrated measurement record from the database server; generating and presenting one or more analysis reports using the measurement server computer; using the measurement server computer to perform a post call to the database server to register new campaign impression data with the database server; One or more non-transitory storage media storing instructions for performing a method including: [Appendix 7] 7. The storage medium of claim 6, further comprising a sequence of instructions that, when executed, causes the one or more measurement calculation server devices to make a determination based on the integrated measurement records regarding a change in prescription filling behavior associated with one of the advertisements delivered for the measurement target campaign, the measurement target campaign being associated with the HCP identifier, and presenting the change in the analysis report. [Appendix 8] 7. The storage medium of claim 6, further comprising a sequence of instructions that, when executed, causes the one or more measurement calculation server devices to train a machine learning model using a training dataset including features selected from the plurality of records and the anonymized tokenized billing data records to create an optimization model, the machine learning model being trained to accept other aggregate measurement records for other campaigns for a target campaign in the diverse plurality of campaigns and to output a predicted bid value for use in automatically adjusting one or more parameters of the DSP. [Appendix 9] 9. The storage medium of claim 8, further comprising a sequence of instructions that, when executed, causes the one or more measurement calculation server devices to use the optimization model to determine a cost for each of the one or more measured campaigns based on the consolidated measurement records, and, in response, automatically signal the DSP to modify one or more configuration parameters to increase payments for at least one of the measured campaigns having the lowest cost for prescriptions of the particular product. [Appendix 10] 9. The storage medium of claim 8, further comprising a sequence of instructions that, when executed, causes the one or more measurement calculation server devices to use the optimization model to accept bid request data and an attribute data set including personal and demographic data and output a prediction of a bid value; update a bid portion of the DSP using the bid value prediction output from the optimization model; and serve one or more advertisements from the DSP based on the updated bid portion. [Appendix 11] 1. A computer system comprising: one or more processors; coupled to one or more of the processors, when executed by the one or more processors, Automatically on the first iteration, and obtaining impression data from a demand-side platform (DSP) specifying a first campaign set associated with a first set of one or more healthcare attributes, the first campaign set being included in a plurality of diverse campaigns executed by the DSP; obtaining a plurality of first records including anonymized consumer tokens representing consumers who received digital impressions of the first campaign set associated with the first set of healthcare attributes; obtaining a plurality of second records including healthcare provider (HCP) identifiers representing HCPs that received digital impressions of the first campaign set associated with the first set of healthcare attributes; using analytical instructions executed on a database server to access a database containing de-identified tokenized claim data records, each of said data records containing at least one of said consumer tokens and / or at least one of said HCP identifiers associated with at least one claim for a prescription for a designated product; using the analysis instructions to perform one or more database join operations on the billing data records and at least one of the consumption tokens and / or at least one of the HCP identifiers to output a result set of one or more consolidated measurement records specifying one or more measurement target campaigns among the diverse plurality of campaigns, wherein the one or more measurement target campaigns are associated with prescriptions for the specified products in at least one billing data record associated with at least one of the consumer tokens and / or at least one of the HCP identifiers; receiving the integrated measurement record from the database server; generating and presenting one or more analytical reports based on the integrated measurement records; performing a post call to the database server to register new campaign impression data with the database server; automatically executing the instructions in one or more second iterations using the new campaign impression data; and one or more computer-readable non-transitory storage media storing instructions operable to cause the one or more processors to perform the method of the present invention; A computer system comprising: [Appendix 12] 12. The system of claim 11, wherein the storage medium further comprises instructions that, when executed by one or more of the processors, cause the system to train a machine learning model using a training dataset including features selected from the anonymized tokenized billing data records and at least one of the first plurality of records or the second plurality of records to create an optimization model, wherein the machine learning model is trained to accept other aggregate measurement records for other campaigns and output a predicted bid value for a target campaign from the diverse plurality of campaigns, for use in automatically adjusting one or more parameters of the DSP. [Appendix 13] 13. The system of claim 12, wherein the storage medium further includes instructions that, when executed by one or more of the processors, cause the system to use the optimization model to determine a cost for each of the one or more measured campaigns based on the consolidated measurement records, and in response, automatically signal the DSP to modify one or more configuration parameters to increase payments for at least one of the measured campaigns having the lowest cost for prescriptions of the particular product. [Appendix 14] 12. The system of claim 11, wherein the result set is generated based on the analysis instructions being executed on the database server to perform one or more database join operations on the anonymized tokenized billing data records and at least one set of impression data records having anonymized consumer tokens and / or at least one set of impression data records having an HCP identifier. [Appendix 15] 12. The system of claim 11, wherein the storage medium further comprises instructions that cause the system to create and present one or more analytical reports based on the integrated measurement record by measuring the total number of qualified consultation interactions associated with at least one specific consumer represented by one of the anonymized consumer tokens or a specific HCP represented by one of the HCP identifiers exposed to a specific campaign among the diverse plurality of campaigns, and a specific product of the specific campaign is specified in a specific claims data record for the specific consumer or the specific HCP. [Explanation of symbols]

[0178] 110 Server Computer 112 Protected environment 114 Anonymized Training Data 115 Training Data Verification Command 116 Machine Learning Generation Training Instructions 117 Trained Machine Learning Systems 118 Machine Learning Verification Command 122 Anonymized Attribute Data 124 Anonymized Claims Data 130 Billing Processors 132 Billing Data 134 Identification Information 136 Cryptographic Tokens 140 attribute database 142 attribute data 144 Identification Information 146 Cryptographic Tokens 150 Media Servers 152 media items 154 Client Computing Device Attribute Data 156 Trained Machine Learning Systems 160 client computing devices 500 Computer Systems 502 I / O Subsystem 504 processor 506 memory 508 ROM (Read Only Memory) 510 Storage device 512 output devices 514 input devices 516 Control Device 518 Communication Interface 520 Network Link 522 Network 524 host 526 ISP (Internet Service Provider) 528 Internet 530 Server 600 Distributed Computer Systems 610 DSP (Demand Side Platform) Environment 612 Impression Data 614 Data Mapping Process 615 HCP (health care provider) identifier 616 Measurement Results / UI (User Interface) 620 Medical Data Repository Environment 622 Rx-Dx Claims Database 623 Token Process 624 Anonymized Tokenized Billing Data 630 Dimension Space 640 machine learning models 652 Optimization Model 772 Demographic Data 774 Bidding Department 784 routes 786 User Computers 788 website 790 Bid Request Data 792 routes 1100 Artificial Neural Networks 1105 nodes 1110 Input layer 1115 nodes 1120 Hidden Layer 1125 connections 1130 Hidden Layer 1140 Hidden Layer 1150 output layer< / value> < / value> < / segmentid>

Claims

1. 1. A computer-implemented method executed by a measurement server computer and a database server of a demand-side platform (DSP), comprising: The measurement server computer acquires advertising impression data associated with a first campaign set associated with a first set of one or more health care attributes, the measurement server computer automatically executing repeated operations of running, measuring, and updating advertising campaigns of the first campaign set based on a digitally stored execution plan, the various campaigns executed by the DSP comprising a health care campaign that conveys timely information to consumers and their service providers to help them make more informed decisions about their health, the health care campaign being run, measured, and optimized by the DSP, the first campaign set, and the advertising impression data being data about impressions related to the first campaign set; The measurement server computer acquires a plurality of first records including anonymized consumer tokens representing consumers who received impressions of the first campaign set, the anonymized consumer tokens having certain data that has been anonymized to protect privacy; executing, by the database server, analysis instructions to access a database containing de-identified tokenized claims data records related to medical claims records identifying diagnostic codes, each of the de-identified tokenized claims data records relating to at least one claim for a prescription for a specified product and including at least one of the de-identified consumer tokens; the database server executes analysis instructions to combine the anonymized tokenized claim data records and the anonymized consumer tokens to output a result set of one or more combined measurement records specifying one or more measured campaigns among the diverse plurality of campaigns, the one or more measured campaigns being associated with the prescription for the specified product in at least one of the anonymized tokenized claim data records associated with at least one of the anonymized consumer tokens; the measurement server computer receiving the integrated measurement record from the database server; causing the database server to generate and present one or more analytical reports on the performance of the advertising campaign based on the consolidated measurement records; the measurement server computer registering new campaign impression data resulting from actual impressions to service providers and / or consumers in the database server; A method that encompasses

2. 10. The method of claim 1, further comprising: the measurement server computer determining, based on the consolidated measurement record, a change in prescription filling behavior associated with one of the advertisements delivered for the measured campaign, the measured campaign being associated with the anonymized consumer token; and presenting the change in the analysis report.

3. The method of claim 1, comprising: the measurement server computer using an optimization model to determine the cost of each of the one or more measured campaigns based on the integrated measurement record; and, in response, automatically signaling the DSP to modify one or more configuration parameters to increase payments for at least one of the measured campaigns having the lowest cost for prescriptions of a particular product.

4. The method of claim 3, further comprising the measurement server computer using the optimization model to accept bid request data and an attribute data set including personal and demographic data and output a predicted bid value; updating a bidding section of the DSP using the predicted bid value output from the optimization model; and serving one or more advertisements from the DSP based on the updated bidding section.

5. When executed by a measurement server computer and a database server of a demand-side platform (DSP), The measurement server computer acquires impression data associated with a first campaign set associated with a first set of one or more healthcare attributes, the measurement server computer automatically executing repeated operations of running, measuring, and updating advertising campaigns of the first campaign set based on a digitally stored execution plan, the various campaigns executed by the DSP being healthcare campaigns that convey timely information to consumers and their service providers to help them make more informed decisions about their health, the healthcare campaigns being run, measured, and optimized by the DSP, including the first campaign set, and the impression data being data about impressions related to the first campaign set; The measurement server computer obtains a plurality of records including health care provider (HCP) identifiers representing HCPs that received impressions of the first campaign set; executing, by the database server, analysis instructions to access a database containing de-identified tokenized claim data records for medical claim records identifying diagnostic codes, each of the de-identified tokenized claim data records including at least one of the HCP identifiers associated with at least one claim for a prescription for a specified product, the de-identified tokenized claim data record having some data that has been de-identified to protect privacy; the database server executes analysis instructions to combine the de-identified tokenized claim data records and the HCP identifiers to output a result set of one or more combined measurement records specifying one or more measured campaigns among the diverse plurality of campaigns, the one or more measured campaigns being associated with the prescriptions for the specified products in at least one of the de-identified tokenized claim data records associated with at least one of the HCP identifiers; the measurement server computer receiving the integrated measurement record from the database server; causing the database server to generate and present one or more analytical reports on the performance of the advertising campaign based on the consolidated measurement records; the measurement server computer registering new campaign impression data resulting from actual impressions to service providers and / or consumers in the database server; One or more non-transitory storage media storing instructions for performing a method including:

6. A storage medium as described in claim 5, further comprising an instruction sequence that, when executed, causes the measurement server computer to make a determination based on the integrated measurement record regarding a change in prescription filling behavior associated with one of a plurality of advertisements communicated for the measured campaign, the measured campaign being associated with the HCP identifier, and presenting the change in the analysis report.

7. The storage medium of claim 5, further comprising an instruction sequence that, when executed, causes the measurement server computer to use an optimization model to determine the cost of each of the one or more measured campaigns based on the integrated measurement record, and in response, automatically signal the DSP to modify one or more configuration parameters to increase payments for at least one of the measured campaigns having the lowest cost for prescriptions of a particular product.

8. The storage medium of claim 7, further comprising an instruction sequence that, when executed, causes the measurement server computer to use an optimization model to accept bid request data and an attribute data set including personal and demographic data and output a bid value prediction, update a bidding section of the DSP using the bid value prediction output from the optimization model, and serve one or more advertisements from the DSP based on the updated bidding section.

9. A computer system comprising a measurement server computer and a database server of a demand-side platform (DSP), one or more processors; coupled to one or more of the processors, when executed by the one or more processors, obtaining impression data associated with a first campaign set associated with a first set of one or more healthcare attributes, the first campaign set being included in a diverse plurality of campaigns executed by the DSP, the diverse plurality of campaigns being healthcare campaigns that communicate timely information to consumers and their service providers to help them make informed decisions about their health, and that are managed, measured, and optimized by the DSP, the impression data being data about impressions related to the first campaign set; obtaining a plurality of first records including anonymized consumer tokens representing consumers who received impressions of the first campaign set associated with the first set of one or more healthcare attributes, the anonymized consumer tokens having certain data that has been anonymized to protect privacy; obtaining a plurality of second records including healthcare provider (HCP) identifiers representing HCPs that received impressions of the first campaign set associated with the first set of one or more healthcare attributes; accessing a database containing de-identified tokenized claim data records relating to medical claim records identifying diagnostic codes, each of the de-identified tokenized claim data records including at least one of the de-identified consumer tokens and / or at least one of the HCP identifiers associated with at least one claim for a prescription for a specified product; combining the anonymized tokenized claim data records and the anonymized consumer tokens to output a result set of one or more combined measurement records specifying one or more measured campaigns among the diverse plurality of campaigns, the one or more measured campaigns being associated with prescriptions for the specified products in at least one anonymized tokenized claim data record associated with at least one of the anonymized consumer tokens and / or at least one of the HCP identifiers; receiving the integrated measurement record from the database server; generating and presenting one or more analytical reports about the performance of the advertising campaigns of the first campaign set based on the consolidated measurement records; registering new campaign impression data resulting from actual impressions to service providers and / or consumers in said database server; one or more computer-readable non-transitory storage media storing instructions operable to cause the one or more processors to perform the method of the present invention; A computer system comprising:

10. The system of claim 9, wherein the storage medium further includes instructions that, when executed by one or more of the processors, cause the system to use an optimization model to determine the cost of each of the one or more measured campaigns based on the integrated measurement record, and in response, automatically signal the DSP to modify one or more configuration parameters to increase payments to at least one of the measured campaigns having the lowest cost for prescriptions of a particular product.

11. A system as described in claim 9 or 10, wherein the result set is generated based on instructions executed on the database server to perform one or more operations on the anonymized tokenized billing data records and at least one set of impression data records having the anonymized consumer token, and / or at least one set of impression data records having the HCP identifier.

Images