Multi-market calibration of convenience panel data to reduce behavioral bias

Abstract

Example methods, apparatus, systems, and articles of manufacture for implementing calibration of convenience panel data to reduce behavioral bias are disclosed. The disclosed example apparatus includes a distribution estimator for: determining a first behavioral distribution of first convenience panel data associated with a first market and a measurement period; determining a second behavioral distribution of second convenience panel data associated with a second market and the measurement period; and determining a third behavioral distribution of probability panel data associated with the second market and the measurement period. The disclosed example apparatus also includes a distribution calibrator for: calibrating a first behavioral distribution determined for first convenience panel data associated with a first market based on (i) a second behavioral distribution determined for second convenience panel data associated with the second market and (ii) a third behavioral distribution determined for probability panel data associated with the second market.

Description

[0001] Related Applications

[0002] This patent claims the benefit of and priority to U.S. Provisional Application Serial No. 62 / 753,657, filed on October 31, 2018, and entitled “MULTI-MARKET PROBABILISTIC SUBSTITUTION USING ITERATIVE LEARNING TO ESTIMATE BEHAVIORAL BIASES.” U.S. Provisional Application Serial No. 62 / 753,657 is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to audience measurement and, more particularly, to multi-market calibration of convenience panel data to reduce behavioral bias. Background Art

[0004] Some audience measurement systems for measuring the audience of online media (e.g., website visits, online advertising, online programs, etc.) utilize exposure data (impression data) obtained from probability panels and / or convenience panels. Probability panels can include individuals, families, etc. who are recruited (e.g., via telephone and / or face-to-face interviews) to meet specified demographic goals (e.g., corresponding to the demographic distribution of the target population). Probability panels can provide accurate granularity real data sets that represent the behavior of individuals (and therefore demographic groups) in the panel. However, the sample size of probability panels may be small and the maintenance cost may be high. In contrast, convenience panels may include individuals, families, etc. who are included in the panel by chance (such as in response to an online prompt to join the panel). The behavioral information of panel members in the convenience panel may be known or unknown, may be biased towards certain behaviors, and / or may not represent the target population. However, convenience panels can have large sample sizes and can be relatively cheap to maintain. Summary of the invention

[0005] One aspect of the present disclosure relates to an apparatus for performing multi-market calibration on convenience panel data, the apparatus comprising: a distribution estimator, the distribution estimator being used to: determine a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; determine a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; and determine a third behavioral distribution of probability panel data associated with the second market and the first measurement period; and a distribution calibrator, the distribution calibrator being used to: calibrate the first behavioral distribution determined for the first convenience panel data associated with the first market based on (i) the second behavioral distribution determined for the second convenience panel data associated with the second market and (ii) the third behavioral distribution determined for the probability panel data associated with the second market; and output the calibrated first behavioral distribution.

[0006] Another aspect of the present disclosure relates to a computer-readable medium comprising computer-readable instructions that, when executed, cause a processor to perform at least the following operations: determine a first behavioral distribution of first convenience group data associated with a first market and a first measurement period; determine a second behavioral distribution of second convenience group data associated with a second market and the first measurement period, the second market being different from the first market; determine a third behavioral distribution of probability group data associated with the second market and the first measurement period; calibrate the first behavioral distribution determined for the first convenience group data associated with the first market based on (i) the second behavioral distribution determined for the second convenience group data associated with the second market and (ii) the third behavioral distribution determined for the probability group data associated with the second market; and output the calibrated first behavioral distribution.

[0007] Another aspect of the present disclosure relates to a method for multi-market calibration of convenience panel data, the method comprising the following steps: using a processor to determine a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; and using the processor to determine a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; using the processor to determine a third behavioral distribution of probability panel data associated with the second market and the first measurement period; using the processor to calibrate the first behavioral distribution determined for the first convenience panel data associated with the first market based on (i) the second behavioral distribution determined for the second convenience panel data associated with the second market and (ii) the third behavioral distribution determined for the probability panel data associated with the second market; and outputting the calibrated first behavioral distribution.

[0008] Another aspect of the present disclosure relates to a system for performing multi-market calibration of convenience panel data, the system comprising: a device for estimating distribution, the device for estimating distribution being used to: determine a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; determine a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; and determine a third behavioral distribution of probability panel data associated with the second market and the first measurement period; and a device for calibrating distribution, the device for calibrating distribution being used to: calibrate the first behavioral distribution determined for the first convenience panel data associated with the first market based on (i) the second behavioral distribution determined for the second convenience panel data associated with the second market and (ii) the third behavioral distribution determined for the probability panel data associated with the second market; and output the calibrated first behavioral distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an example audience measurement system that uses probabilistic surrogacy and iterative learning to implement multi-market calibration of convenience panel data to reduce behavioral bias in accordance with the teachings of the present disclosure.

[0010] Figure 2 is a block diagram of a second example audience measurement system that implements multi-market calibration of convenience panel data to reduce behavioral bias in accordance with the teachings of the present disclosure.

[0011] Figure 3 is included Figure 2 Block diagram of an example behavioral distribution estimator in an example audience measurement system.

[0012] Figure 4 is included Figure 2 Block diagram of an example behavioral distribution calibrator in an example audience measurement system.

[0013] FIG. 5A to FIG. 5D A first example technique for calibrating a behavior distribution determined for convenience panel data associated with a first market based on a behavior distribution determined for probability panel data associated with the first market is illustrated.

[0014] FIG. 6A to FIG. 6D A second example technique for determining a behavioral distribution of convenience panel data associated with a first market is illustrated.

[0015] 7A to 7D A third example technique is illustrated for iteratively calibrating a behavior distribution determined for convenience panel data associated with a first market based on an initial behavior distribution determined for probability panel data associated with a second market different from the first market.

[0016] FIG. 8A to FIG. 8D A fourth example technique is illustrated for calibrating a behavior distribution determined for convenience panel data associated with a first market based on a behavior distribution determined for probability panel data and convenience panel data associated with a second market different from the first market.

[0017] 9A to 9C Depicted FIG. 5A to FIG. 5D The first example technique shown, FIG. 6A to FIG. 6D The second example technique shown and FIG. 8A to FIG. 8D A comparison of the fourth example technique is shown.

[0018] Fig.10 It means that it can be executed to achieve Figure 1 A flow chart of example machine readable instructions for an example audience measurement system.

[0019] Fig.11 It means that it can be executed to achieve Figure 2 A flow chart of example machine readable instructions for an example audience measurement system.

[0020] Fig.12 is constructed to execute Fig.10 and / or Fig.11 Example machine readable instructions to implement Figure 1 and / or Figure 2 Block diagram of an example processor platform for an example audience measurement system.

[0021] The drawings are not drawn to scale. Generally, the same reference numerals will be used throughout the drawings and accompanying written description to refer to the same or like parts, elements, etc.

[0022] In this article, when identifying multiple elements or components that can be mentioned separately, the descriptive terms "first", "second", "third", etc. are used. Unless otherwise specified or understood based on the context of use of these descriptive terms, these descriptive terms are not intended to give any meaning of priority or order in terms of time, but are only used as marks for referring to multiple elements or components respectively, so as to facilitate understanding of the disclosed examples. In some examples, the descriptive term "first" can be used to refer to the elements in the detailed description, while different descriptive terms such as "second" or "third" can be used in the claims to refer to the same element. In this case, it should be understood that such descriptive terms are only used to easily refer to multiple elements or components. DETAILED DESCRIPTION

[0023] This article discloses example methods, devices, systems and articles (e.g., physical storage media) for implementing multi-market calibration of convenience panel data to reduce behavioral bias. As described above, some audience measurement systems for measuring audiences of online media (e.g., website visits, online advertising, online programs, etc.) utilize exposure data obtained from probability panels and / or convenience panels. Probability panels can include individuals, families, etc. who are recruited (e.g., via telephone and / or face-to-face interviews) to meet specified demographic goals (e.g., corresponding to the demographic distribution of the target population). Panel members in the probability panel can agree to provide detailed demographic information to the audience measurement entity (AME) that manages the panel. Panel members in the probability panel can also agree to use metering equipment and / or installed metering applications to monitor their access to online media and / or interaction with online media. Therefore, the probability panel can provide an accurate granular real data set that represents the behavior of individuals in the panel. However, the sample size of the probability panel may be small and the maintenance cost may be high. In contrast, the convenience panel may include individuals, families, etc. who are included in the panel by chance (such as in response to online prompts, slogans, etc. to join the panel). For example, a website serving online media may prompt a user to consent to providing their visit information to a convenience panel, and may prompt the user to enter demographic information. In some examples, the demographic information of panel members in a convenience panel may be known, unknown, and / or unverified, may be biased toward certain demographics, and / or may not represent the target population. However, convenience panels may have large sample sizes and may be relatively inexpensive to maintain.

[0024] In order to take advantage of the relative strengths of probability panel data and convenience panel data while not emphasizing their respective weaknesses, some audience measurement systems combine probability panel data and convenience panel data to obtain an overall measurement data set for a given market, or in other words, a target panel data set. For example, some such audience measurement systems combine probability panel data and convenience panel data based on demographic and / or geographic information (such as by demographic weighting or geo-demographic weighting). In order to perform such a combination, some such audience measurement systems determine the demographic distribution of the population in a given market based on both the available probability panel data and the available convenience panel data for a given market. Since probability panel data is intended to accurately represent the population in a given market, probability panel data can be combined with convenience panel data to reduce or otherwise calibrate possible deviations associated with convenience panel data while taking advantage of the increased sample size provided by the convenience data panel. However, as described above, the maintenance cost of probability panels may be high. In addition, in some markets, it may be difficult to recruit a sufficient number of panel members to represent the target population, and therefore, some demographic categories may be underrepresented, making it impossible for only demographic weighting (or in combination with location-based weighting) to fully reduce the deviations associated with convenience panel data. Thus, in some markets, probability panel data may not be of sufficient quality (eg, in terms of representing the demographics of the target population with a corresponding target sample size) to correct for biases associated with convenience panel data in those markets.

[0025] The example audience measurement system disclosed herein provides a technical solution for the technical problems associated with the non-representative probability panel data of a given market. Specifically, the disclosed example audience measurement system realizes multi-market calibration of convenience panel data to reduce behavioral bias. In order to solve the problem of non-representative probability panel data of a given market, some disclosed example audience measurement systems use behavioral features such as behavioral distribution forms to calibrate the convenience panel data. For example, some disclosed example audience measurement systems use the behavioral distribution determined for the probability panel data from another market (e.g., referred to as the calibration market in this article) as the initial origin (seed) to calibrate the behavioral distribution determined for the convenience panel data associated with a given target market. Then, this disclosed example audience measurement system will iteratively combine the calibrated behavioral distribution determined according to the previous iteration (which is obtained by using the probability panel behavior distribution determined for the calibration market as the initial origin) with the new behavioral distribution determined for the convenience panel data from the given target market to determine the new calibrated behavioral distribution of the convenience panel data in the given target market. Then, the obtained calibrated behavioral distribution can be output to be used as an input to the downstream audience measurement processing stage, which uses the audience distribution determined for the given target market. The calibrated behavior distribution determined for the current processing iteration is also used as the initial behavior distribution for the next processing iteration. In this way, a higher quality probability panel behavior distribution for a different market (e.g., the calibration market) can be used as the origin of a calibration process for calibrating the convenience panel data in a given target market, with any possible differences between the characteristics of the probability panels for other calibration markets and the characteristics of the given target market being reduced over time with each processing iteration.

[0026] Some disclosed example audience measurement systems use probability panel data and convenience panel data obtained from one or more other calibration markets for a measurement interval (e.g., monthly or some other time period) to calibrate the convenience panel data for a given market and the measurement interval. For example, and as disclosed in further detail below, some such audience measurement systems can determine the behavioral distribution of convenience panel data from a given target market, the behavioral distribution of probability panel data from a calibration market, and the behavioral distribution of convenience panel data from a calibration market for a given measurement interval. As described in further detail below, the behavioral distribution can represent the count of individuals associated with different behavioral categories (such as the count of panel members in a probability panel, the count of panel members in a convenience panel, the count of people in a focus group, etc.). Then some disclosed example audience measurement systems use the behavioral distribution of probability panel data from a calibration market and the behavioral distribution of convenience panel data from a calibration market for a given measurement interval to calibrate the behavioral distribution of convenience panel data from a given target market. Then, some such audience measurement systems can output the resulting calibrated behavioral distribution of convenience panel data from a given target market, and / or apply the calibrated behavioral distribution to the convenience panel data to generate audience measurement data associated with online media access / exposure in a target market.

[0027] Go to the attached picture, Figure 1 A block diagram of an example audience measurement system 100 for implementing multi-market calibration of convenience panel data to reduce behavioral bias according to the teachings of the present disclosure is illustrated. The example audience measurement system 100 includes an example probability panel interface 105, an example convenience panel interface 110, an example behavior distribution estimator 115, an example calibration distribution selector 120, an example behavior distribution calibrator 125, and an example target convenience panel data calibrator 130. In the illustrated example, the probability panel interface 105 is configured to obtain probability panel data by interfacing with a first exposure monitoring system (e.g., via a network) that generates online media exposure data for panel members in a probability panel for a target market and / or generates online media exposure data for panel members in probability panels for one or more calibration markets. The exposure data may correspond to, for example, counts of one or more different websites or web pages visited by corresponding panel members in a given probability panel, counts of one or more different online advertisements watched by corresponding panel members in a given probability panel, counts of one or more different online media programs visited by corresponding panel members in a given probability panel, and the like.

[0028] In the example shown, the convenience panel interface 110 is configured to obtain convenience panel data by (e.g., via a network) interfacing with a second exposure monitoring system that generates online media exposure data for panel members in a convenience panel for a target market and / or generates online media exposure data for panel members in a convenience panel for one or more calibration markets. The exposure data may correspond to, for example, counts of one or more different websites visited by corresponding panel members in a given convenience panel, counts of one or more different online advertisements viewed by corresponding panel members in a given convenience panel, counts of one or more different online media programs visited by corresponding panel members in a given convenience panel, etc. In some examples, the convenience panel data obtained through the convenience panel interface 110 is limited to exposure data aggregated across panel members in a convenience panel. In some examples, the market for which the convenience panel data is obtained through the convenience panel interface 110 may be the same or different from the market for which the probability panel data is obtained through the probability panel interface 105. For example, the market for convenience panel data may be a first target market of interest for determining audience measurement data, while the market for probability panel data may be a second calibrated market different from the first market, the second calibrated market being characterized by having panel data of higher quality than the panel data available in the first market.

[0029] In the illustrated example, the behavior distribution estimator 115 is configured to determine the probability group data obtained through the probability group interface 105 and the corresponding behavior overall estimate (universe estimate) (UE) distribution of the convenience group data obtained through the convenience group interface 110. In some examples, the behavior UE represents the total number of behavior-related events in the target population. In some such examples, the marginal behavior UE represents the score of the total number of behavior-related events associated with the behavior-related features (or categories) associated with a given margin, and a set of one or more group members in the target population. In some such examples, the behavior UE distribution represents the set of marginal behavior UEs of a given target population. In some such examples, the behavior UE distribution corresponds to the following UE distribution, in which the panel member features / categories of the UE margin are based on the panel member behavior about accessing online media (e.g., the number / frequency of website visits in the measurement period, the number / frequency of online advertisements accessed in the measurement period, the number / frequency of online media (e.g., programs, movies, clips, etc.) accessed in the measurement period, etc.). This is in contrast to a demographic UE distribution, which corresponds to a UE distribution in which UE and marginal UE represent the number of individuals (rather than events) in a population, and the characteristics / categories of the UE marginals are based on the demographics of the group members (e.g., age, gender, income, etc.).

[0030] Table 1 illustrates an example of a behavior UE distribution determined by an example behavior distribution estimator 115 of probability group data obtained by the probability group interface 105. In Table 1, the behavior UE distribution represents the total count of group members who visited search engines / portals and communities in a given market (e.g., the U.S. market) in a given measurement period (e.g., 1 month) and in accordance with different behavior categories corresponding to different frequencies of visiting websites in a given measurement period. Therefore, in the example of Table 1, the behavior distribution estimator 115 is configured to determine a probability group behavior UE distribution, wherein the marginal behavior UE of the distribution represents the size of different group member groups with different frequencies, and the group members visit the search engines / portals and communities in a given market with these different frequencies. For example, the behavior distribution estimator 115 can sort the website visits of different group members according to the order of the frequency of different group members visiting the monitored websites during the measurement period, and then divide the sorted website visits into quartiles (quartiles) (e.g., four equal parts). In the example of Table 1, the first quartile corresponds to the number of group members falling into the lowest quartile of website visit frequency, and the fourth quartile corresponds to the number of group members falling into the highest quartile of website visit frequency. In the example of Table 1, the marginal behavior count is also weighted based on one or more demographic features to represent the demographics of the crowd. In the example of Table 1, the behavior distribution estimator 115 also determines the marginal behavior count, which represents the group members who visited the website during the measurement period but the website was not in the category of concern (corresponding to NO VISIT in Table 1) and the group members who were online but had no browser activity during the measurement period (corresponding to INACTIVE in Table 1). As shown in Table 1, the behavior distribution estimator 115 can be configured to determine the behavior UE distribution as a relative percentage (as shown in the last column of the table).

[0031] Table 1

[0032]

[0033] In order to implement multi-market calibration of convenience panel data to reduce behavioral bias, the behavioral distribution estimator 115 of the illustrated example determines a behavioral distribution of the convenience panel data obtained for a first target market of interest during a first measurement interval, such as a distribution similar to the example shown in Table 1. For example, the target market may correspond to a European Union (EU) country, and the first measurement interval may correspond to a given month, such as the current month. The behavioral distribution estimator 115 of the illustrated example also determines a behavioral distribution of the probability panel data obtained for a second calibration market during the first measurement interval, such as a distribution similar to the example shown in Table 1. For example, the second market may correspond to the United States (US). The calibration distribution selector 120 of the illustrated example then selects the probability panel behavioral distribution of the second market as an initial distribution (e.g., becomes the origin) for iteratively determining subsequent target behavioral distributions for audience measurement in the first market.

[0034] The behavior distribution calibrator 125 of the illustrated example then combines the initial calibration behavior distribution (which is the probability panel behavior distribution for the second market) with the convenience panel behavior distribution determined for the target market by the behavior distribution estimator 115 to determine a new (later) target behavior distribution for the first market for use in determining audience measurement data associated with the first measurement period and the first market. For example, the behavior distribution calibrator 125 may combine the initial calibration behavior distribution for the calibration market with the convenience panel behavior distribution for the target market by weighting the initial calibration behavior distribution (e.g., by weighting the individual marginal counts of the initial calibration behavior distribution) and weighting the convenience panel behavior distribution (e.g., by weighting the individual marginal counts of the convenience panel behavior distribution) and then combining the weighted distributions (e.g., by combining the corresponding weighted marginal counts from the two distributions) to determine the new (later) target behavior distribution for the target market. In some examples, the weights may be set equal (e.g., such as a value equal to 0.5) so that the initial calibration behavior distribution and the convenience panel behavior distribution contribute equally to the combination. In some examples, the weights of the initial calibration behavior distribution can be different from the weights of the convenience panel behavior distribution so that one distribution contributes more to the combination than the other.

[0035] exist Figure 1In the illustrated example, the target convenience panel data calibrator 130 reports the new (post) target behavior UE distribution determined by the behavior distribution calibrator 125 for the target market and the first measurement period to the downstream audience measurement processing. In some examples, the target convenience panel data calibrator 130 may also use the new (post) target behavior UE distribution determined for the target market and the first measurement period to calibrate (e.g., weight) the convenience panel data obtained by the convenience panel interface 110 for the target market during the first measurement period to reduce the behavioral bias present in the convenience panel data obtained for the first market. In some examples, the downstream audience measurement processing may use the calibrated convenience panel data and / or the target behavior UE distribution reported by the target convenience panel data calibrator 130 for the target market and the first measurement period to determine audience estimates (e.g., ratings, reach, exposure, etc.) for online media in the target market.

[0036] In the example shown, the target convenience panel data calibrator 130 also provides the new (post) target behavior distribution determined for the target market and the first measurement period to the calibration distribution selector 120, which selects the distribution as the initial calibration behavior distribution for the next processing iteration corresponding to the next measurement period. Then, during the next processing iteration corresponding to the next measurement period, the behavior distribution estimator 115 of the example shown determines the convenience panel behavior distribution for the target market of interest and for the next measurement period (e.g., the next month). The behavior distribution calibrator 125 then combines the target behavior distribution from the previous measurement period with the convenience panel behavior distribution for the next measurement period to determine the new (post) target behavior distribution for the next measurement period. This iterative processing continues, with the new (post) target behavior distribution determined during one iteration becoming the initial calibration behavior distribution to be used during the next processing iteration. Thus, during the next processing iteration corresponding to the next measurement period, the behavior distribution estimator 115 of the example shown determines the convenience panel behavior distribution for the target market of interest and for the next measurement period (e.g., the next month). The behavior distribution calibrator 125 then combines the target behavior distribution from the previous measurement period with the convenience panel behavior distribution for the next measurement period to determine the new (post) target behavior distribution for the next measurement period. This iterative processing continues, with the new (post) target behavior distribution determined during one iteration becoming the initial calibration behavior distribution to be used during the next processing iteration. Thus, during Figure 1 In the illustrated example of , the probabilistic panel behavior distribution for the calibration market is used as a probabilistic starting point for calibrating future target behavior distributions, which will be weighted and combined with the convenience panel behavior distributions in the target market associated with future measurement periods to determine new target behavior distributions that will be used in audience measurement calculations associated with those future measurement periods. However, the impact of the calibration market's data decreases with each processing iteration.

[0037] As described above, the behavior distribution determined by the behavior distribution estimator 115 may be weighted based on one or more demographic characteristics of the population, and the resulting new (post) target behavior distribution determined by the audience measurement system 100 may be used in downstream audience measurement processing. An example of such an overall audience measurement process is as follows:

[0038] 1. Get the demographic UEs of the target market of interest (eg, first market).

[0039] 2. Weight the probability panel data for the calibration market (eg, the second market) based on demographic UEs.

[0040] 3. The weighted probability panel data is used as input to the first processing iteration in the audience measurement system 100 as described above.

[0041] 4. Combine the convenience panel data for the target market with the probability panel data for the target market (if probability panel data for the target market is available).

[0042] 5. Weight the target market's convenience panel data (or the combined panel data determined according to step 4) based on the demographic UEs and the new (post) target behavior distribution determined by the audience measurement system 100 for the current measurement period.

[0043] 6. Based on the weighted convenience panel data from step 5 (or weighted combined panel data if available), determine and report audience measurement metrics.

[0044] Figure 2 A block diagram of a second example audience measurement system 200 for implementing multi-market calibration of convenience panel data to reduce behavioral bias according to the teachings of the present disclosure is illustrated. The example audience measurement system 200 is configured to calibrate convenience panel data (also referred to herein as convenience samples) obtained for a target market by creating a behavioral target for market A using (i) geo-demographically weighted convenience panel data from a target market of interest (also referred to herein as market A in the following description) and (ii) a relationship of a re-weighted probability panel behavior distribution (also referred to herein as probability sample behavior distribution) to a re-weighted convenience panel behavior distribution (also referred to herein as convenience sample behavior distribution) in one or more calibration markets (also referred to herein as markets B, C, ... in the following description). For example, market A may correspond to a target country of interest for which the convenience panel data will be calibrated, market B may correspond to a first calibration market (such as a first calibration country) to be used to calibrate the convenience panel data for market A, and markets C, D, E, ... may correspond to other calibration markets (such as other calibration countries) to be used to calibrate the convenience panel data for market A. The example audience measurement system 200 then weights the convenience panel data for market A based on existing geo-demographic controls and behavioral targets determined via the calibration process described above to obtain calibrated convenience panel data for market A, which can be used to determine online media audience estimates (e.g., ratings, reach, exposure, etc.) with reduced behavioral bias relative to uncalibrated convenience panel data.

[0045] Go to Figure 2 , similar to Figure 1 The example audience measurement system 100, the example audience measurement system 200 includes an example probability panel interface 105, an example convenience panel interface 110, an example behavior distribution estimator 215, an example behavior distribution calibrator 225, and an example target convenience panel data calibrator 230. Figure 2 In the illustrated example, as described above, the probability panel interface 105 is configured to interface with one or more example calibration market probability panel data sources 205 (e.g., via one or more networks). In the illustrated example, the calibration market probability panel data source 205 provides probability panel data for one or more calibration markets that are different from the target market for which the convenience panel data will be calibrated. For example, the probability panel interface 105 can be interfaced with one or more exposure monitoring systems, the one or more exposure monitoring systems including metering devices and / or monitoring applications that interface with computing devices of probability panel members in one or more calibration markets and / or execute on computing devices of probability panel members in one or more calibration markets to generate online media exposure data. Additionally or alternatively, in some examples, the probability panel interface 105 can be interfaced with metering devices and / or monitoring applications (e.g., via one or more networks) that interface with computing devices of probability panel members in one or more calibration markets and / or execute on computing devices of probability panel members in one or more calibration markets to obtain online media exposure data. Examples of exposure data are described above.

[0046] exist Figure 2In the illustrated example of, as described above, the convenience panel interface 110 is configured to interface (e.g., via one or more networks) with one or more example target market convenience panel data sources 210 and one or more example calibration market convenience panel data sources 212. In the illustrated example, the target market convenience panel data source 210 provides convenience panel data for a given target market of interest for which the convenience panel data will be calibrated. For example, the convenience panel interface 110 can be interfaced with one or more exposure monitoring systems, which are interfaced with websites serving online media in the target market to obtain online media exposure data provided by those websites (e.g., when a user accesses online media from a website, the online media exposure data is obtained from a user in the target market in response to a prompt from the website). Additionally or alternatively, in some examples, the convenience panel interface 110 can be interfaced with a website serving online media in the target market (e.g., via one or more networks) to obtain online media exposure data from the website. Similarly, the calibration market convenience panel data source 212 provides convenience panel data for one or more calibration markets, which are different from the target market for which the convenience panel data will be calibrated. For example, the convenience panel interface 110 may interface with one or more exposure monitoring systems that interface with websites serving online media in the convenience market to obtain online media exposure data provided by those websites (e.g., when users access online media from the websites, the online media exposure data is obtained from users in the convenience market in response to prompts from the websites). Additionally or alternatively, in some examples, the convenience panel interface 110 may interface with a website serving online media in the convenience market (e.g., via one or more networks) to obtain online media exposure data from the website. Examples of exposure data are described above.

[0047] The example audience measurement system 200 includes an example behavior distribution estimator 215, which may be similar to Figure 1 Example behavior distribution estimator 115. Figure 2 In the illustrated example of , the example behavior distribution estimator 215 is configured to determine a behavior distribution of probability panel data obtained for the calibration market through the probability panel interface 105 (e.g., obtained from the calibration market probability panel data source 205). The behavior distribution estimator 215 is also configured to determine corresponding behavior distributions of convenience panel data obtained for the calibration market through the convenience panel interface 110 (e.g., obtained from the calibration market convenience panel data source 212) and for the target market (e.g., obtained from the target market convenience panel data source 210). Figure 3 Illustrated Figure 2 An example implementation of the behavior distribution estimator 215 is shown.

[0048] Go to Figure 3 , the example behavior distribution estimator 215 includes an example behavior distribution initializer 305, an example target market panel data weighter 310, an example calibration market panel data weighter 315, and an example behavior distribution calculator 320. The behavior distribution initializer 305 is configured to define behavior control variables (also referred to as behavior categories) for which calibration targets (e.g., corresponding to calibrated convenience panel distributions) will be determined for a target market (referred to as market A in the following description). For example, the behavior categories defined by the behavior distribution initializer 305 can correspond to the behavior categories specified in Table 1, which correspond to different panelist quartiles associated with different frequencies of panelists in market A visiting monitored websites. In some examples, the behavior distribution initializer 305 defines the behavior categories for which calibration targets will be determined based on user input, one or more configuration scripts, and the like.

[0049] In the illustrated example, the behavior distribution initializer 305 is also configured to obtain geodemographic weighting controls that will be used to initially weight the convenience panel data for target market A prior to performing behavior-based calibration. For example, the behavior distribution initializer 305 can obtain demographic weighting controls that define one or more demographic targets to which the convenience panel data for target market A will be weighted to meet the one or more demographic targets. For example, the demographic targets can correspond to target age distributions, gender distributions, income distributions, education level distributions, etc., or any combination thereof, representing the population in market A. Additionally or alternatively, the behavior distribution initializer 305 can obtain location weighting controls that define one or more location targets to which the convenience panel data for target market A will be weighted to meet the one or more location targets. For example, the location targets can correspond to target population sizes in different geographic regions of market A. In some examples, the behavior distribution initializer 305 obtains demographically weighted controls and / or location-weighted controls from one or more data sources, such as government census sources, market research sources, etc.

[0050] In the illustrated example, the target market panel data weighter 310 weights a convenience sample of target market A (or in other words, the convenience panel data) to correspond to the geo-demographic weighted controls obtained for target market A by the behavioral distribution initializer 305. For example, the convenience sample of target market A may include demographic data and / or location data associated with exposure data included in the panelist entries represented in the convenience sample. In some such examples, the target market panel data weighter 310 may scale, duplicate, remove, etc. some of the panelist entries included in the convenience sample of target market A in a manner such that the resulting exposure contributions of different demographic and / or location categories correspond to the geo-demographic weighted controls obtained for target market A by the behavioral distribution initializer 305. For example, the target market panel data weighter 310 may weight (e.g., scale, duplicate, and / or remove) panelist entries corresponding to a first demographic category in the convenience sample of target market A to match the demographic weighted controls obtained for the first demographic category of market A by the behavioral distribution initializer 305. Similarly, the market panel data weighter 310 may weight (e.g., scale, duplicate, and / or remove) panelist entries corresponding to a second demographic category in the convenience sample for target market A to match the demographic weighting control obtained for that second demographic category for market A by the behavioral distribution initializer 305, and so on for different demographic weighting controls obtained for market A by the behavioral distribution initializer 305. Additionally or alternatively, the market panel data weighter 310 may perform similar weighting for different location categories represented in the convenience sample based on corresponding location weighting controls obtained by the behavioral distribution initializer 305.

[0051] In the following description, the resulting weighted convenience sample determined for target market A by target market panel data weighter 310 is referred to as target market convenience sample A2.

[0052] In the illustrated example, the calibration market panel data weighter 315 weights the probability sample (or in other words, the probability panel data) for calibration market B to correspond to the demographic weighted controls obtained for target market A by the behavioral distribution initializer 305. For example, the probability sample for calibration market B may include demographic data associated with the exposure data included in the panelist entries represented in the probability sample. In some such examples, the calibration market panel data weighter 315 may scale, duplicate, remove, etc. some of the panelist entries included in the probability sample for calibration market B in a manner such that the resulting exposure contributions of different demographic categories correspond to the demographic weighted controls obtained for target market A by the behavioral distribution initializer 305. For example, the calibration market panel data weighter 315 may weight (e.g., scale, duplicate, and / or remove) the panelist entries corresponding to a first demographic category in the probability sample for calibration market B to match the weighted controls obtained for that demographic category for target market A by the behavioral distribution initializer 305. Similarly, the calibration market panel data weighter 315 can weight (e.g., scale, copy and / or remove) panel member entries corresponding to a second demographic category in the probability sample for calibration market B to match the weighting control obtained for that demographic category for market A via the behavioral distribution initializer 305, and so on for different demographic categories obtained for market A via the behavioral distribution initializer 305.

[0053] In some examples, the probability sample of calibration market B may be geographically disproportionate with respect to the convenience sample of calibration market B and / or with respect to calibration market B as a whole. For example, different geographic regions in calibration market B may contribute differently to the probability sample and / or convenience sample based on different panel sampling rates used in the respective geographic regions. For example, a first geographic region in calibration market B may contain 50% of the probability sample in calibration market B, while two other geographic regions contain 20% and 30% of the probability sample, respectively. The convenience sample in calibration market B may contribute 40%, 30%, and 30% from the respective regions. Thus, in such an example, the probability sample from calibration market B is geographically differently distributed from the convenience samples in the first geographic region and the second geographic region.

[0054] In some examples, to account for geographically disproportionate probability and convenience samples from calibration market B, calibration market panel data weighter 315 applies geographic weighting to probability samples from market B. For example, calibration market panel data weighter 315 may determine a geographic weighting target to apply to probability samples from calibration market B by determining a geographic UE distribution for probability samples from market B and then applying that geographic UE distribution for market B to an aggregate population (e.g., sample size) of convenience samples from target market A. In some examples, market panel data weighter 315 determines the geographic UE distribution for probability samples from market B in a manner similar to how the behavioral UE distribution is determined, but the distribution categories correspond to different geographic locations / regions rather than different behavioral categories.

[0055] In the following description, the resulting weighted probability sample determined for calibration market B by calibration market panel data weighter 315 is referred to as calibration market probability sample B1.

[0056] In the illustrated example, the calibration market panel data weighter 315 similarly weights the convenience sample of calibration market B (or in other words, the convenience panel data) to correspond to the demographic weighted controls obtained for target market A by the behavioral distribution initializer 305. For example, the convenience sample of calibration market B may include demographics associated with the exposure data included in the panelist entries represented in the convenience sample. In some such examples, the calibration market panel data weighter 315 may scale, duplicate, remove, etc. some of the panelist entries included in the convenience sample of calibration market B in a manner such that the resulting exposure contributions of different demographic categories correspond to the demographic weighted controls obtained by the behavioral distribution initializer 305 for target market A. For example, the calibration market panel data weighter 315 may weight (e.g., scale, duplicate, and / or remove) panelist entries corresponding to a first demographic category in the convenience sample of calibration market B to match the weighted controls obtained for that demographic category for market A by the behavioral distribution initializer 305. Similarly, the calibration market panel data weighter 315 can weight (e.g., scale, copy and / or remove) panelist entries corresponding to a second demographic category in the convenience sample of calibration market B to match the weighted control obtained for that demographic category for market A via the behavioral distribution initializer 305, and so on for different demographic categories obtained for market A via the behavioral distribution initializer 305.

[0057] In some examples, the convenience sample of calibration market B may be geographically disproportionate with respect to the probability sample of calibration market B and / or with respect to calibration market B overall. For example, different geographic regions in calibration market B may contribute differently to the convenience sample and the probability sample based on different panel sampling rates used in the corresponding geographic regions. In some examples, to account for the geographically disproportionate convenience sample and probability sample from calibration market B, the calibration market panel data weighter 315 applies geographic weighting to the convenience sample from market B. For example, the calibration market panel data weighter 315 may weight the probability sample from market B using the geographic weighting target determined as described above, such that the convenience sample from market B is also weighted.

[0058] In the following description, the resulting weighted convenience sample determined for calibration market B by calibration market panel data weighter 315 is referred to as calibration market convenience sample B2.

[0059] Next, the behavior distribution calculator 320 uses the behavior control variables determined by the behavior distribution initializer 305 to initialize the behavior categories of the behavior distribution to be determined for the target market convenience sample A2, the calibration market probability sample B1, and the calibration market convenience sample B2. In some examples, the behavior distribution calculator 320 omits the behavior categories not included in the calibration market probability sample B1. For each behavior category, the behavior distribution calculator 320 calculates the corresponding percentage of the weighted calibration market probability sample B1 in each defined behavior category to determine the behavior distribution of the calibration market probability sample B1. In the following description, the behavior distribution percentage value of the cth behavior category in the behavior distribution of the calibration market probability sample B1 is represented by B1c%.

[0060] Similarly, for each behavior category, the behavior distribution calculator 320 calculates the corresponding percentage of the weighted calibration market convenience sample B2 in each defined behavior category to determine the behavior distribution of the calibration market convenience sample B2. In the following description, the behavior distribution percentage value of the c-th behavior category in the behavior distribution of the calibration market convenience sample B2 is represented by B2c%. Similarly, for each behavior category, the behavior distribution calculator 320 calculates the corresponding percentage of the weighted target market convenience sample A2 in each defined behavior category to determine the behavior distribution of the target market convenience sample A2. In the following description, the behavior distribution percentage value of the c-th behavior category in the behavior distribution of the target market convenience sample A2 is represented by A2c%.

[0061] return Figure 2, the behavior distribution estimator 215 outputs the behavior distribution of the target market convenience sample A2 (e.g., the values ​​of A2c% for different behavior categories c), the behavior distribution of the calibration market probability sample B1 (e.g., the values ​​of B1c% for different behavior categories c), and the behavior distribution of the calibration market convenience sample B2 (e.g., the values ​​of B2c% for different behavior categories c) to the behavior distribution calibrator 225. In the example shown, the behavior distribution calibrator 225 calibrates the behavior distribution of the target market convenience sample A2 (e.g., the values ​​of A2c% for different behavior categories c) based on the behavior distribution of the calibration market probability sample B1 (e.g., the values ​​of B1c% for different behavior categories c) and the behavior distribution of the calibration market convenience sample B2 (e.g., the values ​​of B2c% for different behavior categories c). Figure 4 An example implementation of the behavior distribution calibrator 225 is illustrated.

[0062] Go to Figure 4 , the example behavior distribution calibrator 225 includes an example preliminary distribution calibrator 405 and an example final distribution calibrator 410. The preliminary distribution calibrator 405 performs an initial calibration on the behavior distribution of the target market convenience sample A2 based on the behavior distribution of the calibration market probability sample B1 and the behavior distribution of the calibration market convenience sample. For example, for each behavior category c, the preliminary distribution calibrator 405 uses the value B1c% of the behavior distribution of the calibration market probability sample B1 and the value B2c% of the behavior distribution of the calibration market convenience sample B2 according to Formula 1 to calculate the preliminary calibration value of the value A2c% of the behavior distribution of the target market convenience sample A2, Formula 1 is:

[0063] PTBc=(B1c% / B2c%)×A2c%

[0064] Formula 1

[0065] Therefore, according to Formula 1, for each behavior category c, the preliminary distribution calibrator 405 scales the value A2c% of the behavior distribution of the target market convenience sample A2 according to the ratio of (i) the value B1c% of the behavior distribution of the calibration market probability sample B1 to (ii) the value B2c% of the behavior distribution of the calibration market convenience sample B2 to determine the preliminary calibration value of the behavior distribution of the target market convenience sample A2. The preliminary calibration value is represented as PTBc in Formula 1.

[0066] In some examples, if the value B1c% of the cth category of the behavioral distribution of the calibration market probability sample B1 does meet the threshold (e.g., is less than the threshold), then the ratio of the value of the broader (or larger or higher level) hierarchical behavioral category in the behavioral distribution of (i) the calibration market probability sample B1 to the value of the broader hierarchical behavioral category in the behavioral distribution of (ii) the calibration market convenience sample B2 is used to scale the value A2c% of the behavioral distribution of the target market convenience sample A2. For example, the threshold can be configured based on user input, specified as a configuration parameter, hard-coded, etc. In some examples, the behavioral categories of the behavioral distribution can be arranged in a hierarchical manner. For example, the first higher level of the behavioral category hierarchy can divide the group according to the type of online media access, and the first level categories are, for example, news, entertainment, public service, and commercials. The second lower level of the behavioral category hierarchy can divide the behavioral categories of news media access into political news, sports news, and weather news. If the preliminary distribution calibrator 405 attempts to calibrate a category in the lower level category of the behavior distribution of the target market convenience sample A2 (e.g., political news, sports news, or weather news), and that category in the behavior distribution of the calibration market probability sample B1 does not meet a threshold, the preliminary distribution calibrator 405 uses a higher level category corresponding to the lower level category in the behavior distribution of the calibration market probability sample B1 and the behavior distribution of the calibration market target sample B2 (e.g., news in this example) to calibrate the lower level behavior category in the behavior distribution of the target market convenience sample A2.

[0067] For example, assume that the value A2c% of the behavior distribution of the target market convenience sample A2 corresponds to the "political news" behavior category in the previous example, and the corresponding value B1c% of the "political news" behavior category in the behavior distribution of the calibration market probability sample B1 does not meet the threshold. Also assume that the corresponding values ​​of the higher-level category "news" in the behavior distribution of the calibration market probability sample B1 and the behavior distribution of the calibration market target sample B2 are B1c'% and B2c'%, respectively. Then, the preliminary distribution calibrator 405 uses the value B1c'% of the behavior distribution of the calibration market probability sample B1 and the value B2c'% of the behavior distribution of the calibration market convenience sample B2 to calculate the preliminary calibration value of the value A2c% of the behavior distribution of the target market convenience sample A2 according to Formula 2, which is:

[0068] PTBc=(B1c'% / B2c'%)×A2c%

[0069] Formula 2

[0070] The final distribution calibrator 410 performs a final calibration of the behavior distribution of the target market convenience sample A2 to normalize the values ​​of the behavior distribution to a sum of 100% (or a probability value of 1). In some examples, the sum of the preliminary calibration values ​​PTBc determined by the preliminary distribution calibrator 405 for the behavior distribution of the target market convenience sample A2 is not 100% (or a probability value is not 1). Therefore, for each behavior category c, the final distribution calibrator 410 calculates the final calibration value (expressed as FTBc) for the initial calibration value (expressed as PTBc) of the behavior distribution of the target market convenience sample A2 according to Formula 3, which is:

[0071] FTBc=(100 / ∑cPTBc)×PTBc

[0072] Formula 3

[0073] In Equation 3, the summation is performed over the behavioral categories defined for the behavioral distribution of the convenience sample A2 in the target market.

[0074] In some examples, the final distribution calibrator 410 outputs a value of FTBc as a calibrated behavioral distribution of the target market convenience sample A2. For example, such a FTBc value output by the final distribution calibrator 410 may correspond to a calibrated version of the value in the "Weighted Distribution" column of Table 1 above.

[0075] In some examples, the behavior distribution estimator 215 and the behavior distribution calibrator 225 of the audience measurement system 200 repeat the aforementioned operations for additional calibration markets (e.g., such as calibration markets C, D, E, etc.). In such examples, the behavior distribution estimator 215 and the behavior distribution calibrator 225 determine different calibrated behavior distributions for the target market convenience sample A2, each of which is calibrated based on a different one of the calibration markets. However, in some such examples, the behavior categories initialized for some or all of the behavior distributions determined by the behavior distribution estimator 215 are limited to the behavior categories included in the calibration sample of the first calibration market (such as, market B in the previous example).

[0076] In some examples, if additional calibration markets are utilized, the behavior distribution calibrator 225 calculates the final calibrated behavior distribution of the target market convenience sample A2 as an average of the calibrated behavior distributions determined based on the different calibration markets. In some examples, the behavior distribution calibrator 225 calculates the final calibrated behavior distribution of the target market convenience sample A2 as a weighted average of the calibrated behavior distributions determined based on the different calibration markets, wherein, based on one or more criteria, some calibrated behavior distributions have a greater weight in the average than other calibrated behavior distributions.

[0077] return Figure 2 , the target convenience panel data calibrator 230 of the audience measurement reports the calibrated behavioral UE distribution determined by the behavioral distribution calibrator 225 for the convenience sample of the target market A and the given measurement period to the downstream audience measurement processing. In some examples, the target convenience panel data calibrator 230 can also use the calibrated behavioral UE distribution determined for the target market A and the given measurement period to calibrate (e.g., weight) the convenience panel data (e.g., convenience sample) obtained by the convenience panel interface 110 for the target market A during the given measurement period to reduce the behavioral bias in the convenience panel data obtained for the target market. For example, the target convenience panel data calibrator 230 can weight the convenience panel data (e.g., convenience sample) obtained by the convenience panel interface 110 for the target market A during the given measurement period according to the above-mentioned geo-demographic controls for meeting demographic and geographic-based goals and according to the calibrated behavioral goals represented by the calibrated behavioral UE distribution. As a result, calibrated convenience panel data (e.g., calibrated convenience sample) for the target market A during the given measurement period is obtained. Additionally or alternatively, the targeted convenience panel data calibrator 230 may weight the population of people intended to represent target market A and any other data samples for a given measurement period according to the geo-demographic controls for target market A and the calibrated behavioral targets represented by the calibrated behavioral UE distribution to determine a corresponding calibrated data sample for target market A. In some examples, downstream audience measurement processing may use the calibrated convenience panel data, any other calibrated data samples determined for the target market, and / or the calibrated behavioral UE distribution reported by the targeted convenience panel data calibrator 230 for the target market and given measurement period to determine audience estimates (e.g., ratings, reach, exposure, etc.) for online media in the target market. For example, such output audience estimates may correspond to calibrated versions of the values ​​in the "Weighted Estimates" column of Table 1 above.

[0078] exist Figure 2 In the illustrated example, the probability panel interface 105, the convenience panel interface 110, the behavior distribution estimator 215, the behavior distribution calibrator 225, and the example target convenience panel data calibrator 230 repeat the above process for each subsequent measurement interval to determine a new calibrated behavior UE distribution for the convenience sample of the target market for each subsequent measurement interval. In some examples, the target convenience panel data calibrator 230 also uses the calibrated behavior UE distribution determined for each subsequent measurement interval to determine a calibrated convenience sample (and / or other calibrated data sample) for the target market and for the subsequent measurement interval.

[0079] FIG. 5A to FIG. 5DA first example technique 500 is illustrated for calibrating a convenience panel behavior UE distribution for a first market based on a behavior UE distribution determined for probability panel data associated with the first market and a behavior UE distribution determined for convenience panel data also associated with the first market. Figures 1 to 4 The first example technique 500 described in different examples begins with a first measurement period (e.g., month 1), wherein an example convenience panel behavior UE distribution 505 for a first market (e.g., target market) and the first measurement period is weighted to align with an example probability panel behavior UE distribution 510 for the same first market and the first measurement period to determine an example calibrated convenience panel behavior UE distribution 515 for the first market and the first measurement period. Then, during a second measurement period (e.g., month 2), an example convenience panel behavior UE distribution 520 for the first market and the second measurement period is weighted to align with an example probability panel behavior UE distribution 525 for the same first market and the second measurement period to determine an example calibrated convenience panel behavior UE distribution 530 for the first market and the second measurement period. Then, during a third measurement period (e.g., month 3), an example convenience panel behavior UE distribution 535 for the first market and the third measurement period is weighted to align with an example probability panel behavior UE distribution 540 for the first market and the third measurement period to determine an example calibrated convenience panel behavior UE distribution 545 for the first market and the third measurement period. Then, during a fourth measurement period (e.g., month 4), example convenience panel behavior UE distribution 550 for the first market and fourth measurement period is weighted to align with example probability panel behavior UE distribution 555 for the first market and fourth measurement period to determine example calibrated convenience panel behavior UE distribution 560 for the first market and fourth measurement period. FIG. 5A to FIG. 5D In the illustrated example of , for each measurement period, the convenience group behavior UE distribution obtained for a first market and for the measurement period is weighted to align with the probability group behavior UE distribution obtained for the same first market and measurement period.

[0080] FIG. 6A to FIG. 6D A second example technique 600 is illustrated for determining a behavioral UE distribution for a first market based on a behavioral UE distribution determined only for convenience panel data associated with the first market. Figures 1 to 4A second example technique 600 of a different example is described starting with a first measurement period (e.g., month 1) where an example target behavior UE distribution 610 for a first market (e.g., target market) and the first measurement period is determined using an example convenience group behavior UE distribution 605 for a first market (e.g., target market) and the first measurement period in the absence of probability group data. Then, during a second measurement period (e.g., month 2), an example target behavior UE distribution 620 for the first market and the second measurement period is determined using an example convenience group behavior UE distribution 615 for the first market and the second measurement period in the absence of probability group data. Then, during a third measurement period (e.g., month 3), an example target behavior UE distribution 630 for the first market and the third measurement period is determined using an example convenience group behavior UE distribution 625 for the first market and the third measurement period in the absence of probability group data. Then, during a fourth measurement period (e.g., month 4), an example target behavior UE distribution 640 for the first market and the fourth measurement period is determined using an example convenience group behavior UE distribution 635 for the first market and the fourth measurement period in the absence of probability group data.

[0081] 7A to 7D A third example technique 700 is illustrated for iteratively calibrating a convenience panel behavioral UE distribution for a first market based on an initial behavioral UE distribution determined from probability panel data associated with a second market (e.g., a calibration market) different from the first market (e.g., a target market). Figure 1The third example technique 700 of the described example begins with a first measurement period (e.g., month 1), wherein an example convenience panel behavioral UE distribution 705 for a first market and the first measurement period is weighted and combined with an example weighted probability panel behavioral UE distribution 710 for a second market and the first measurement period to determine an example calibrated behavioral UE distribution 715 for the first market and the first measurement period. Then, during a second measurement period (e.g., month 2), an example convenience panel behavioral UE distribution 720 for the first market and the second measurement period is weighted and combined with the weighted calibrated behavioral UE distribution 715 determined for the first measurement period to determine a new example calibrated behavioral UE distribution 730 for the first market and the second measurement period. Then, during a third measurement period (e.g., month 3), an example convenience panel behavioral UE distribution 735 for the first market and the third measurement period is weighted and combined with the weighted calibrated behavioral UE distribution 730 determined for the second measurement period to determine a new example calibrated behavioral UE distribution 745 for the first market and the third measurement period. Then, during a fourth measurement period (e.g., month 4), the example convenience panel behavioral UE distribution 750 for the first market and the fourth measurement period is weighted and combined with the weighted calibrated behavioral UE distribution 745 determined for the third measurement period to determine a new example calibrated behavioral UE distribution 460 for the first market and the fourth measurement period. 7A to 7D In the example shown in , for an initial measurement period, the convenience panel behavioral UE distribution obtained for a first market and for the measurement period is weighted and combined with the weighted probability panel behavioral UE distribution obtained for a different second market to determine a calibrated behavioral UE distribution for the first market and the initial measurement period. Thereafter, for each subsequent measurement period, the convenience panel behavioral UE distribution obtained for the first market and for a given subsequent measurement period is weighted and combined with the weighted calibrated behavioral UE distribution determined for the previous measurement period to determine a new calibrated behavioral UE distribution for the first market and the given measurement period.

[0082] FIG. 8A to FIG. 8D A fourth example technique 800 is illustrated for calibrating a behavioral UE distribution determined from convenience panel data for a first market based on a behavioral UE distribution determined from probability panel data and convenience panel data associated with a second market (e.g., a calibration market) different from the first market (e.g., a target market). Figures 2 to 4The fourth example technique 800 of the described examples begins with a first measurement period (e.g., month 1), wherein an example convenience group behavior UE distribution 805 for a first market and the first measurement period is calibrated based on an example combination 810 of a probability group behavior UE distribution and a convenience group behavior UE distribution for a second market and the first measurement period to determine an example calibrated convenience group behavior UE distribution for the first market and the first measurement period 815. Then, during a second measurement period (e.g., month 2), an example convenience group behavior UE distribution 820 for the first market and the second measurement period is calibrated based on an example combination 825 of a probability group behavior UE distribution and a convenience group behavior UE distribution for the second market and the second measurement period to determine a new example calibrated convenience group behavior UE distribution 830 for the first market and the second measurement period. Then, during a third measurement period (e.g., month 3), the example convenience group behavior UE distribution 835 for the first market and the third measurement period is calibrated based on the example combination 840 of the probability group behavior UE distribution and the convenience group behavior UE distribution for the second market and the third measurement period to determine a new example calibrated convenience group behavior UE distribution 845 for the first market and the third measurement period. Then, during a fourth measurement period (e.g., month 4), the example convenience group behavior UE distribution 850 for the first market and the fourth measurement period is calibrated based on the example combination 855 of the probability group behavior UE distribution and the convenience group behavior UE distribution for the second market and the fourth measurement period to determine a new example calibrated convenience group behavior UE distribution 860 for the first market and the fourth measurement period. Thus, during 7A to 7D In the example shown, for each measurement period, the convenience group behavior UE distribution obtained for a first market (e.g., a target market) and for the measurement period is calibrated based on a combination of the probability group behavior UE distribution and the convenience group behavior UE distribution obtained for different second markets to determine a calibrated convenience group behavior UE distribution for the first market and the measurement period.

[0083] 9A to 9C Depicted FIG. 5A to FIG. 5D The first example technique 500 shown, FIG. 6A to FIG. 6D The second example technique 600 is shown and FIG. 8A to FIG. 8D 800. Specifically, Fig.9A The resulting calibrated behavior UE distributions 515, 530, 545, and 560 determined by the first example technique 500 during four example measurement periods are depicted, as described above in conjunction with FIG. 5A to FIG. 5D as described. Fig. 9BDepicted are the resulting behavioral UE distributions 610, 620, 630, and 640 determined by the second example technique 600 during four example measurement periods, as described above in conjunction with FIG. 6A to FIG. 6D as described. Fig. 9C The resulting calibrated behavior UE distributions 815, 830, 845, and 860 determined by the fourth example technique 800 during four example measurement periods are depicted, as described above in conjunction with FIG. 8A to FIG. 8D As described. 9A to 9C It can be seen that through Figures 2 to 4 The fourth example technique 800 implemented by the example audience measurement system 200 of generates a calibrated behavior UE distribution having less volatility than the behavior UE distributions obtained according to the other example techniques 500 and 600 .

[0084] Although in Figures 1 to 4 An example manner of implementing the audience measurement system 100 and the audience measurement system 100 are illustrated in Figures 1 to 4 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figures 1 to 4The example probability panel interface 105, the example convenience panel interface 110, the example behavior distribution estimator 115, the example calibration distribution selector 120, the example behavior distribution calibrator 125, the example convenience panel data calibrator 130, the example behavior distribution estimator 215, the example behavior distribution calibrator 225, the example target convenience panel data calibrator 230, the example behavior distribution initializer 305, the example target market panel data weighter 310, the example calibration market panel data weighter 315, the example behavior distribution calculator 320, the example preliminary distribution calibrator 405, the example final distribution calibrator 410, and more generally the example audience measurement system 100 and / or the audience measurement system 200 may be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware. Thus, for example, the example probability panel interface 105, the example convenience panel interface 110, the example behavior distribution estimator 115, the example calibration distribution selector 120, the example behavior distribution calibrator 125, the example convenience panel data calibrator 130, the example behavior distribution estimator 215, the example behavior distribution calibrator 225, the example target convenience panel data calibrator 230, the example behavior distribution initializer 305, the example target market panel data weighter 310, the example calibration market panel data weighter 315, the example behavior distribution calculator 320, the example preliminary distribution calibrator 405, the example final distribution calibrator 410, and more generally any of the example audience measurement system 100 and / or the example audience measurement system 200 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and / or field programmable logic devices (FPLDs). When any apparatus or system claim of this patent is read to cover pure software and / or firmware implementations, at least one of the example audience measurement system 100, the example audience measurement system 200, the example probability panel interface 105, the example convenience panel interface 110, the example behavior distribution estimator 115, the example calibration distribution selector 120, the example behavior distribution calibrator 125, the example convenience panel data calibrator 130, the example behavior distribution estimator 215, the example behavior distribution calibrator 225, the example target convenience panel data calibrator 230, the example behavior distribution initializer 305, the example target market panel data weighter 310, the example calibration market panel data weighter 315, the example behavior distribution calculator 320, the example preliminary distribution calibrator 405 and / or the example final distribution calibrator 410 is hereby expressly defined to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., which includes software and / or firmware.Furthermore, example audience measurement system 100 and / or example audience measurement system 200 may be additional. Figures 1 to 4 In addition to or in place of those elements, processes and / or devices illustrated in Figures 1 to 4 The elements, processes and / or devices illustrated in the present invention may include one or more elements, processes and / or devices, and / or may include more than one of any or all of the elements, processes and devices illustrated. As used herein, the phrase "communicating" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also additionally includes selective communication at regular intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0085] Figure 10 to Figure 11 A flowchart is shown that represents example hardware logic, machine readable instructions, hardware-implemented state machines, and / or any combination thereof for implementing audience measurement system 100 and audience measurement system 200. In this example, the machine readable instructions may be one or more executable programs or portions thereof for execution by a computer processor, such as the following in conjunction with Fig.12 The processor 1212 shown in the example processor platform 1200 discussed above. One or more programs, or portions thereof, may be implemented in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or other storage medium associated with the processor 1212. TM or memory), but the entire program or multiple programs and / or portions thereof may alternatively be executed by a device other than the processor 1212, and / or implemented in firmware or dedicated hardware. In addition, although reference is made to Figure 10 to Figure 11 The flowcharts shown describe example procedures, but many other methods of implementing the example audience measurement system 100 and / or the example audience measurement system 200 may alternatively be used. For example, referring to Figure 10 to Figure 11 The flowchart shown may change the execution order of the blocks, and / or some of the blocks described may be changed, eliminated, combined and / or subdivided into multiple blocks. Additionally or alternatively, any or all of the blocks in the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are configured to perform corresponding operations without executing software or firmware.

[0086] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a packaged format, etc. The machine-readable instructions described herein may be stored as data (e.g., portions of instructions, code, code representations, etc.) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, etc., so that they can be directly read and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts, each of which is compressed, encrypted, and stored on a separate computing device, wherein the parts, after decryption, decompression, and merging, form an executable instruction set that implements a program (such as the program described herein). In another example, the machine-readable instructions may be stored in a state in which they are computer-readable, but require additional libraries (e.g., dynamic link libraries), software development kits (SDKs), application programming interfaces (APIs), etc., in order to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions and / or corresponding programs may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before they can be executed in whole or in part. Therefore, the disclosed machine-readable instructions and / or corresponding programs are intended to cover such machine-readable instructions and / or programs regardless of the specific format or state of the machine-readable instructions and / or programs when stored or otherwise at rest or transmitted.

[0087] As described above, executable instructions (eg, computer and / or machine readable instructions) may be used to implement Figure 10 to Figure 11 The example processing of the executable instructions is stored on a non-transitory computer and / or machine readable medium, such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk where information is stored for any duration (e.g., for an extended period of time, permanently, for a transient situation, for temporary buffering, and / or for caching of information). As used herein, the term "non-transitory computer readable medium" is expressly defined to include any type of computer readable storage device and / or storage disk, and excludes propagating signals and excludes transmission media. Likewise, as used herein, the terms "computer readable" and "machine readable" are considered equivalent unless otherwise noted.

[0088] "Include" and "comprising" (and all their forms and tenses) are used herein as open-ended terms. Thus, whenever a claim employs any form of "include" and "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or in any kind of claim recitation, it should be understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "include" and "comprising." The term "and / or," when used (e.g., in a form such as A, B, and / or C), refers to any combination or subset of A, B, and C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, and (7) A, B, and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the execution or implementation of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or implementation of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" means an implementation that includes any of (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0089] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude the plural. As used herein, the term "a" or "an" entity refers to one or more of the entity. The terms "one" (or "an"), "one or more," and "at least one" are used interchangeably herein. In addition, although listed separately, multiple devices, elements, or method actions may be implemented by, for example, a single unit or processor. In addition, although individual features may be included in different examples or claims, these individual features may be combined, and being included in different examples or claims does not mean that the combination of features is not feasible and / or unfavorable.

[0090] Fig.10 illustrates that can be executed to implement Figure 1 Example procedure 1000 of example audience measurement system 100. Referring to the preceding figures and associated written description, Fig.10 The example program 1000 begins execution at box 1005, where, if the measurement interval is the first measurement interval in which calibration is to be performed, execution proceeds to box 1010. At box 1010, the example probability panel interface 105 accesses probability panel data for the calibration market and for the first measurement interval, as described above. At box 1015, the example behavior distribution estimator 115 determines the behavior UE distribution of the probability panel data obtained for the calibration market and for the first measurement interval, as described above. At box 1020, the example calibration distribution selector 120 selects the probability panel behavior UE distribution determined in box 1015 as the initial calibration behavior distribution, as described above. At box 1025, the example convenience panel interface 110 accesses convenience panel data for the target market and for the first measurement interval, as described above. At box 1030, the behavior distribution estimator 115 determines the behavior UE distribution of the convenience panel data obtained for the target market and for the first measurement interval, as described above. At block 1035, the example behavior distribution calibrator 125 weights and combines the behavior UE distribution of the convenience panel data determined at block 1030 with the initial calibrated behavior distribution determined at block 1020 to determine a target behavior UE distribution for the target market and the first measurement interval, as described above. At block 1040, the example target convenience panel data calibrator 130 reports the calibrated convenience panel UE distribution for the target market and the first measurement interval, as described above.

[0091] If processing is to continue for a subsequent measurement interval, execution returns to block 1005. Because the subsequent measurement interval is not the first measurement interval, execution further continues to block 1050. At block 1050, the example calibration distribution selector 120 selects the target behavior UE distribution determined for the target market and the previous measurement interval at block 1040 as the initial calibration behavior distribution for the current measurement interval, as described above. Processing then continues as described above through blocks 1025 to 1040, which produce a target behavior UE distribution for the target market and the subsequent measurement interval, as described above. Execution then continues to iterate from block 1045 to block 1050, followed by blocks 1025 to 1040, to produce a target behavior UE distribution for the target market and the subsequent measurement interval.

[0092] Fig.11 illustrates that can be executed to implement Figures 2 to 4 Example procedure 1100 of the example audience measurement system 200. Referring to the preceding figures and associated written description, Fig.11The example program 1100 begins execution at box 1105, where the example behavior distribution initializer 305 of the example behavior distribution estimator 215 included in the audience measurement system 200 initializes the behavior distribution parameters used by the behavior distribution estimator 215, as described above. At box 1110, the example target market panel data weighter 310 of the behavior distribution estimator 215 weights the convenience panel data from the target market based on the geo-demographic weighting parameters of the target market as described above to determine the weighted target market convenience panel data (e.g., the above-mentioned target market convenience sample A2). At box 1115, the example calibration market panel data weighter 315 of the behavior distribution estimator 215 weights the probability panel data from the calibration market based on the demographic weighting parameters of the target market as described above to determine the weighted calibration market probability panel data (e.g., the above-mentioned calibration market probability sample B1). At block 1120 , the calibration market panel data weighter 315 weights the convenience panel data from the calibration market based on the demographic weighting parameters of the target market as described above to determine weighted calibration market convenience panel data (eg, calibration market convenience sample B2 described above).

[0093] At block 1125, the example behavior distribution calculator 320 of the behavior distribution estimator 215 calculates the behavior UE distribution of the weighted target market convenience panel data (e.g., with the above-mentioned category value A2c%) as described above. At block 1125, the behavior distribution calculator 320 calculates the behavior UE distribution of the weighted calibration market probability panel data (e.g., with the above-mentioned category value B1c%) as described above. At block 1125, the behavior distribution calculator 320 further calculates the behavior UE distribution of the weighted calibration market convenience panel data (e.g., with the above-mentioned category value B2c%) as described above.

[0094] In box 1130, the example preliminary distribution calibrator 405 of the example behavior distribution calibrator 225 included in the audience measurement system 200 performs preliminary calibration of the behavior UE distribution determined for the weighted target market convenience panel data (e.g., with the above-mentioned category value B1c%) based on the behavior UE distribution determined for the weighted calibration market probability panel data (e.g., with the above-mentioned category value B2c%) and the behavior UE distribution determined for the weighted calibration market convenience panel data (e.g., with the above-mentioned category value B2c%) as described above. In box 1135, the example final distribution calibrator 410 of the behavior distribution calibrator 225 performs final calibration of the preliminary calibration value of the convenience panel behavior UE distribution for the target market (e.g., corresponding to the above-mentioned PTBc) as described above to produce a final calibration value of the convenience panel behavior UE distribution for the target market (e.g., corresponding to the above-mentioned FTBc).

[0095] At block 1140, the behavior distribution estimator 215 and the behavior distribution calibrator 225 repeat the processing of blocks 1105 to 1135 to calibrate the convenience group behavior UE distribution for the target market based on the probability group behavior UE distribution and the convenience group behavior UE distribution for the other calibration markets, as described above. At block 1140, the behavior distribution calibrator 225 combines (e.g., averages) the calibration results based on the different calibration markets to determine a final calibrated convenience group behavior UE distribution for the target market and the current measurement interval.

[0096] At box 1145, the example targeted convenience panel data calibrator 230 of the audience measurement system 200 weights the convenience panel data (e.g., convenience sample) for the target market and the current measurement interval according to the geo-demographic weighting parameters for the target market and the calibrated behavioral targets represented by the calibrated behavioral UE distribution as described above to determine the calibrated convenience panel data (e.g., calibrated convenience sample) for the target market and the current measurement interval. Additionally or alternatively, at box 1145, the targeted convenience panel data calibrator 230 weights any other data samples intended to represent the target market and the current measurement interval according to the geo-demographic weighting parameters for the target market and the calibrated behavioral targets represented by the calibrated behavioral UE distribution to determine the corresponding calibrated data sample for the target market.

[0097] At box 1150, the example targeted convenience panel data calibrator 230 of the audience measurement system 200 outputs the final calibrated convenience panel behavior UE distribution for the target market and the current measurement interval for use in downstream audience measurement processing. Additionally or alternatively, at box 1150, the targeted convenience panel data calibrator 230 outputs calibrated convenience panel data and / or other calibrated data samples for the target market for use in downstream audience measurement processing. Additionally or alternatively, at box 1150, the targeted convenience panel data calibrator 230 determines and outputs online media audience estimates (e.g., ratings, reach, exposure, etc.) based on the calibrated convenience panel data and / or other calibrated data samples determined for the target market and the current measurement interval.

[0098] Fig.12 is a block diagram of an example processor platform 1200 configured to perform Fig.10 and / or Fig.11 Instructions to achieve Figures 1 to 4 The example audience measurement system 100 and / or the example audience measurement system 200 of FIG. 1200 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, an iPad, etc.). TMtablet computer, etc.), a personal digital assistant (PDA), an Internet appliance, or any other type of computing device.

[0099] The processor platform 1200 of the illustrated example includes a processor 1212. The processor 1212 of the illustrated example is hardware. For example, the processor 1212 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor 1212 can be a semiconductor-based (e.g., silicon-based) device. In the illustrated example, when the example audience measurement system 100 is implemented, the processor 1212 implements the example probability panel interface 105, the example convenience panel interface 110, the example behavior distribution estimator 115, the example calibration distribution selector 120, the example behavior distribution calibrator 125, and the example convenience panel data calibrator 130. In the illustrated example, when the example audience measurement system 200 is implemented, the processor 1212 implements the example probability panel interface 105, the example convenience panel interface 110, the example behavior distribution estimator 215, the example behavior distribution calibrator 225, and the example target convenience panel data calibrator 230.

[0100] The processor 1212 of the illustrated example includes a local memory 1213 (e.g., a cache). The processor 1212 of the illustrated example communicates with a main memory including a volatile memory 1214 and a non-volatile memory 1216 via a link 1218. The link 1218 may be implemented by a bus, one or more point-to-point connections, etc., or a combination thereof. The volatile memory 1214 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1216 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1214, 1216 is controlled by a memory controller.

[0101] The processor platform 1200 of the illustrated example also includes an interface circuit 1220. The interface circuit 1220 may be implemented by any type of interface standard (such as an Ethernet interface, a universal serial bus (USB), interface, near field communication (NFC) interface and / or PCI Express interface).

[0102] In the example shown, one or more input devices 1222 are connected to the interface circuit 1220. The input devices 1222 allow a user to input data and / or commands into the processor 1212. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, a track bar (such as isopoint), a voice recognition system, and / or any other human-computer interface. Moreover, many systems (such as the processor platform 1200) may allow a user to control a computer system and provide data to a computer using physical gestures (such as, but not limited to, hand or body movements, facial expressions, and facial recognition).

[0103] One or more output devices 1224 are also connected to the interface circuit 1220 of the illustrated example. The output device 1224 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switch (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 1220 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0104] The interface circuitry 1220 of the illustrated example also includes communication devices (such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces) to facilitate exchanging data with external machines (e.g., any type of computing device) via network 1226. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0105] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 for storing software and / or data. Examples of such mass storage devices 1228 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0106] and Fig.10 and / or Fig.11 The machine executable instructions 1232 corresponding to the instructions may be stored in a mass storage device 1228, a volatile memory 1214, a non-volatile memory 1216, a local memory 1213, and / or on a removable non-transitory computer-readable storage medium (such as a CD or DVD) 1236.

[0107] From the foregoing, it will be appreciated that example methods, apparatus, and articles of manufacture have been disclosed for performing multi-market calibration of convenience panel data to reduce behavioral bias. The disclosed methods, apparatus, and articles of manufacture improve the efficiency of using a computing device by enabling large convenience panel data samples to be used in an audience measurement system without introducing behavioral bias into the system, or at least with reduced behavioral bias compared to behavioral bias associated with an uncalibrated convenience panel data sample. The disclosed methods, apparatus, and articles of manufacture therefore relate to one or more improvements in the functionality of a computer.

[0108] Although certain example methods, apparatus, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture that fully fall within the scope of the claims of this patent.

[0109] The following claims are hereby incorporated into the Detailed Description by reference, with each claim standing on its own as a separate embodiment of the disclosure.

Claims

1. A device for performing multi-market calibration on convenience panel data, the device comprising: A distribution estimator, the distribution estimator being used to: determining a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; determining a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; as well as determining a third behavioral distribution of probability panel data associated with the second market and the first measurement period; as well as A distribution calibrator, the distribution calibrator being used for: calibrating the first behavior distribution determined for the first convenience panel data associated with the first market based on (i) the second behavior distribution determined for the second convenience panel data associated with the second market and (ii) the third behavior distribution determined for the probability panel data associated with the second market; as well as Output the calibrated first row as the distribution.

2. The device according to claim 1, wherein: The distribution estimator comprises: a first data weighter that weights the first convenience subgroup data associated with the first market to determine weighted first convenience subgroup data associated with the first market, the first data weighter weighting the first convenience subgroup data based on at least one of a demographic weighted control and a location weighted control obtained for the first market; and A distribution calculator determines the first behavioral distribution based on the weighted first convenience panel data associated with the first market.

3. The device according to claim 2, wherein: The distribution estimator also includes: a second data weighter that weights the second convenience panel data associated with the second market to determine weighted second convenience panel data associated with the second market, the second data weighter weighting the second convenience panel data based on the demographic weighting control obtained for the first market; and The distribution calculator determines the second behavioral distribution based on the weighted second convenience panel data associated with the second market.

4. The device according to claim 3, wherein: the second data weighter weights the probability panel data associated with the second market to determine weighted probability panel data associated with the second market, the second data weighter weighting the probability panel data based on the demographic weighting control obtained for the first market; and The distribution calculator determines the third behavior distribution based on the weighted probability panel data associated with the second market.

5. The device according to any one of claims 1 to 4, wherein: The distribution calibrator includes a first distribution calibrator and a second distribution calibrator, and the first distribution calibrator calibrates the value of the first category of the first behavior distribution based on the corresponding value of the first category of the second behavior distribution and the corresponding value of the first category of the third behavior distribution to determine the first calibration value of the first category of the first behavior distribution.

6. The device according to claim 5, wherein: The first distribution calibrator calibrates the values ​​of the first category of the first behavior distribution by scaling the values ​​of the first category of the first behavior distribution according to a ratio of (i) the corresponding values ​​of the first category of the third behavior distribution and (ii) the corresponding values ​​of the first category of the second behavior distribution.

7. The device according to claim 5, wherein: The second distribution calibrator calibrates the first calibration value of the first category of the first behavior distribution to determine a second calibration value of the first category of the first behavior distribution.

8. The apparatus according to any one of claims 1 to 4, further comprising a data calibrator that calibrates the first convenience panel data associated with the first market based on the calibrated first behavior distribution.

9. A computer-readable medium comprising computer-readable instructions that, when executed, cause a processor to at least: determining a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; determining a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; determining a third behavioral distribution of probability panel data associated with the second market and the first measurement period; calibrating the first behavior distribution determined for the first convenience panel data associated with the first market based on (i) the second behavior distribution determined for the second convenience panel data associated with the second market and (ii) the third behavior distribution determined for the probability panel data associated with the second market; as well as Output the calibrated first row as the distribution.

10. The computer-readable medium of claim 9, wherein: When executed, the instructions cause the processor to perform the following operations: weighting the first convenience panel data associated with the first market to determine weighted first convenience panel data associated with the first market, the first convenience panel data being weighted based on at least one of a demographic weighted control and a location weighted control obtained for the first market; as well as The first behavioral distribution is determined based on the weighted first convenience panel data associated with the first market.

11. The computer-readable medium of claim 10, wherein: When executed, the instructions cause the processor to perform the following operations: weighting the second convenience panel data associated with the second market to determine weighted second convenience panel data associated with the second market, the second convenience panel data being weighted based on the demographic weighted control obtained for the first market; as well as The second behavior distribution is determined based on the weighted second convenience panel data associated with the second market.

12. The computer-readable medium of claim 11, wherein: When executed, the instructions cause the processor to perform the following operations: weighting the probability panel data associated with the second market to determine weighted probability panel data associated with the second market, the probability panel data to be weighted based on the demographic weighted control obtained for the first market; and The third behavior distribution is determined based on the weighted probability panel data associated with the second market.

13. The computer-readable medium according to any one of claims 9 to 12, wherein: When executed, the instructions cause the processor to perform the following operations: Calibrate the value of the first category of the first behavior distribution based on the corresponding value of the first category of the second behavior distribution and the corresponding value of the first category of the third behavior distribution to determine a first calibrated value of the first category of the first behavior distribution; as well as The first calibrated value for the first category of the first behavior distribution is further calibrated to determine a second calibrated value for the first category of the first behavior distribution.

14. The computer-readable medium of claim 13, wherein: When executed, the instructions cause the processor to perform the following operations: calibrate the values ​​of the first category of the first behavior distribution by scaling the values ​​of the first category of the first behavior distribution according to the ratio of (i) the corresponding values ​​of the first category of the third behavior distribution and (ii) the corresponding values ​​of the first category of the second behavior distribution.

15. A method for multi-market calibration of convenience panel data, the method comprising the following steps: determining, with a processor, a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; as well as determining, with the processor, a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; determining, with the processor, a third behavioral distribution of probability panel data associated with the second market and the first measurement period; calibrating, with the processor, the first behavior distribution determined for the first convenience panel data associated with the first market based on (i) the second behavior distribution determined for the second convenience panel data associated with the second market and (ii) the third behavior distribution determined for the probability panel data associated with the second market; as well as Output the calibrated first row as the distribution.

16. The method according to claim 15, wherein: The step of determining the first behavior distribution comprises: weighting the first convenience panel data associated with the first market to determine weighted first convenience panel data associated with the first market, the first convenience panel data being weighted based on at least one of a demographic weighted control and a location weighted control obtained for the first market; and The first behavioral distribution is determined based on the weighted first convenience panel data associated with the first market.

17. The method according to claim 16, wherein: The step of determining the second behavior distribution comprises: weighting the second convenience panel data associated with the second market to determine weighted second convenience panel data associated with the second market, the second convenience panel data being based on the demographically weighted control weights obtained for the first market; and The second behavior distribution is determined based on the weighted second convenience panel data associated with the second market.

18. The method according to claim 17, wherein: The step of determining the third behavior distribution comprises: weighting the probability panel data associated with the second market to determine weighted probability panel data associated with the second market, the probability panel data being based on the demographically weighted control weights obtained for the first market; and The third behavior distribution is determined based on the weighted probability panel data associated with the second market.

19. The method according to any one of claims 15 to 18, wherein: The step of calibrating the first behavior distribution comprises: Calibrate the value of the first category of the first behavior distribution based on the corresponding value of the first category of the second behavior distribution and the corresponding value of the first category of the third behavior distribution to determine a first calibrated value of the first category of the first behavior distribution; and The first calibrated value for the first category of the first behavior distribution is further calibrated to determine a second calibrated value for the first category of the first behavior distribution.

20. The method according to claim 19, wherein: The step of calibrating the value of the first category of the first behavior distribution includes: scaling the value of the first category of the first behavior distribution according to the ratio of (i) the corresponding value of the first category of the third behavior distribution and (ii) the corresponding value of the first category of the second behavior distribution.

21. A system for multi-market calibration of convenience panel data, the system comprising: Means for estimating a distribution, the means for estimating a distribution being configured to: determining a first behavioral distribution of first convenience panel data associated with a first market and a first measurement period; determining a second behavioral distribution of second convenience panel data associated with a second market and the first measurement period, the second market being different from the first market; as well as determining a third behavioral distribution of probability panel data associated with the second market and the first measurement period; as well as Means for calibrating a distribution, the means for calibrating a distribution being used to: calibrating the first behavior distribution determined for the first convenience panel data associated with the first market based on (i) the second behavior distribution determined for the second convenience panel data associated with the second market and (ii) the third behavior distribution determined for the probability panel data associated with the second market; as well as Output the calibrated first row as the distribution.

22. The system of claim 21, wherein: The means for estimating the distribution is used to: weighting the first convenience group data associated with the first market to determine weighted first convenience group data associated with the first market, the first convenience group data being weighted based on at least one of a demographic weighted control and a location weighted control obtained for the first market; as well as The first behavioral distribution is determined based on the weighted first convenience panel data associated with the first market.

23. The system of claim 22, wherein: The means for estimating the distribution is used to: weighting the second convenience panel data associated with the second market to determine weighted second convenience panel data associated with the second market, the second convenience panel data being weighted based on the demographically weighted control weights obtained for the first market; as well as The second behavior distribution is determined based on the weighted second convenience panel data associated with the second market.

24. The system of claim 23, wherein: The means for estimating the distribution is used to: weighting the probability panel data associated with the second market to determine weighted probability panel data associated with the second market, the probability panel data being based on the demographically weighted control weights obtained for the first market; and The third behavior distribution is determined based on the weighted probability panel data associated with the second market.

25. A system according to any one of claims 21 to 24, wherein: The device for calibrating the distribution is used to: Calibrate the value of the first category of the first behavior distribution based on the corresponding value of the first category of the second behavior distribution and the corresponding value of the first category of the third behavior distribution to determine a first calibrated value of the first category of the first behavior distribution; as well as The first calibrated value for the first category of the first behavior distribution is further calibrated to determine a second calibrated value for the first category of the first behavior distribution.

26. The system of claim 25, wherein: The device for calibrating the distribution is used to calibrate the value of the first category of the first behavior distribution by scaling the value of the first category of the first behavior distribution according to the ratio of (i) the corresponding value of the first category of the third behavior distribution and (ii) the corresponding value of the first category of the second behavior distribution.

Images