Demographic Estimation System, Method, and Non-Transitory Computer-Readable Storage Medium

Abstract

The present disclosure relates to a demographic estimation system, method, and non-transitory computer-readable storage medium. Example methods, devices, systems, and articles of manufacture (e.g., physical storage media) are disclosed that implement neural network processing of set-top box return path data to estimate household demographics. The example demographic estimation system disclosed herein includes a feature generator that generates features based on return path data reported from a set-top box associated with a return path data household. The disclosed example demographic estimation system also includes a neural network that processes the features generated based on the return path data to predict demographic classification probabilities for the return path data household, the neural network being trained based on panel data reported from a meter that monitors media devices associated with panelist households. The disclosed example demographic estimation system also includes a demographic assignment engine that assigns one or more demographic categories to the respective return path data household in the return path data household based on the predicted demographic classification probabilities.

Description

[0001] Related Applications

[0002] This application claims the benefit and priority of U.S. Provisional Application Serial No. 62 / 743,925, entitled “NEURAL NETWORK PROCESSING OF SET-TOP BOX RETURN PATH DATA TO ESTIMATE HOUSEHOLD DEMOGRAPHICS,” filed on Oct. 10, 2018. The entire content of U.S. Provisional Application Serial No. 62 / 743,925 is incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to neural networks, and more particularly, to neural network processing of return path data to estimate household demographics. Background Art

[0004] Audience measurement entities (AMEs) such as The Nielsen Company (US), LLC may extrapolate rating metrics and / or other audience measurement data for the overall television audience based on a relatively small sample of panel homes. Panel homes may be studied in depth and are typically selected to represent the entire audience universe. Also, to help supplement panel data, an AME such as The Nielsen Company (US), LLC may enter into an agreement with a pay television provider company to obtain television tuning information from set-top boxes and / or other devices / software, which is referred to herein and in the industry as return path data. Summary of the Invention

[0005] In a first aspect, the present disclosure relates to a demographic estimation system, the demographic estimation system comprising: a processor; and a memory storing machine-readable instructions thereon, the machine-readable instructions, when executed by the processor, causing operations to be performed including the following: generating features based on return path data reported from a set-top box associated with a return path data household, wherein the generating includes: for the respective return path data reported by each return path data household: compiling the respective return path data into a set of video blocks generated by the return path data household during a specific observation period, wherein each video block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective return path data into a plurality of video block-level features represented by a video block-level feature vector, the video block-level features including: a channel change rate determined based on a ratio of (i) the number of channel changes occurring during the time interval to (ii) the duration of the time interval; for each of a plurality of radio stations to which the set-top box of the return path data household is configured to tune, the respective total number of minutes the radio station is watched; and a list of radio stations of the plurality of radio stations accessed during the time interval; and generating a two-dimensional N×F video block-level feature vector representing a record of all video blocks of the return path data household during the specific observation period, wherein N is the total number of video blocks in the set of video blocks, and F is the total number of video block-level features; for each return path data household, aggregating household-level features into a one-dimensional H×1 household-level feature vector, wherein the household-level features are different from the video block-level features, wherein H is the number of household-level features, and wherein the household-level features include the total number of video blocks reported for the return path data household during the specific observation period and the total number of tuners known to be included in the return path data household; implementing a neural network to process the video block-level feature vector of each return path data household generated based on the return path data to predict the demographic classification probability of the return path data household, wherein the neural network is trained based on panel data reported from meters that monitor media devices associated with panelist households, the neural network including: a time-distributed dense layer configured to reduce, for each return path data household, the video block-level features to a compressed set of features having a quantity less than the video block-level features, wherein the time-distributed dense layer includes a set of weights to map the video blocks of the video block-level features into the compressed set of features; a recurrent neural network layer configured to generate, for each return path data household, a one-dimensional feature vector by processing the compressed set of features;and a merging layer configured to generate a merged feature vector for each return path data family by merging the features of the generated one-dimensional feature vectors with the family-level feature vectors of the return path data family; and assign one or more demographic categories to the corresponding return path data family in the return path data families based on the predicted demographic classification probabilities.

[0006] Second aspect, the present disclosure relates to a non-transitory computer-readable storage medium including computer-readable instructions that, when executed, cause a processor to perform operations including: generating features based on return path data reported from a set-top box associated with a return path data household, wherein the generating includes: for the respective return path data reported by each return path data household: compiling the respective return path data into a set of video blocks generated by the return path data household during a specific observation period, wherein each video block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective return path data into a plurality of video block-level features represented by a video block-level feature vector, the video block-level features including: a channel change rate determined based on a ratio of (i) the number of channel changes that occurred during the time interval to (ii) the duration of the time interval; for each of a plurality of radio stations to which the set-top box of the return path data household is configured to tune, the respective total number of minutes the radio station was watched; and a list of radio stations of the plurality of radio stations accessed during the time interval; and generating a two-dimensional N×F video block-level feature vector representing a record of all video blocks of the return path data household during the specific observation period, wherein N is the total number of video blocks in the set of video blocks, and F is the total number of video block-level features; for each return path data household, aggregating household-level features into a one-dimensional H×1 household-level feature vector, wherein the household-level features are different from the video block-level features, wherein H is the number of household-level features, and wherein the household-level features include the total number of video blocks reported for the return path data household during the specific observation period and the total number of tuners known to be included in the return path data household; implementing a neural network to process the video block-level feature vector of each return path data household generated based on the return path data to predict a demographic classification probability of the return path data household, wherein the neural network is trained based on panel data reported by a meter that monitors media devices associated with panelist households, the neural network including: a time-distributed dense layer configured to, for each return path data household, reduce the video block-level features to a compressed set of features having a quantity less than the video block-level features, wherein the time-distributed dense layer includes a set of weights to map the video blocks of the video block-level features into the compressed set of features; a recurrent neural network layer configured to, for each return path data household, generate a one-dimensional feature vector by processing the compressed set of features;and a merging layer configured to generate a merged feature vector for each return path data family by merging the features of the generated one-dimensional feature vectors with the family-level feature vectors of the return path data family; and assign one or more demographic categories to the corresponding return path data family in the return path data family based on the predicted demographic classification probabilities.;

[0007] In a third aspect, the present disclosure relates to a demographic estimation method, which is executed by a computing system including a processor and a memory. The demographic estimation method includes: generating features based on return path data reported from a set-top box associated with a return path data household, wherein the generating includes: for the respective return path data reported by each return path data household: compiling the respective return path data into a set of video blocks generated by the return path data household within a specific observation period, wherein each video block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective return path data into a plurality of video block-level features represented by a video block-level feature vector, the video block-level features including: a channel change rate determined based on a ratio of (i) the number of channel changes that occurred during the time interval to (ii) the duration of the time interval; for each of a plurality of radio stations to which the set-top box of the return path data household is configured to tune, the respective total number of minutes the radio station was watched; and a list of radio stations of the plurality of radio stations accessed during the time interval; and generating a two-dimensional N×F video block-level feature vector representing a record of all video blocks of the return path data household within the specific observation period, wherein N is the total number of video blocks in the set of video blocks, and F is the total number of the video block-level features; for each return path data household, aggregating household-level features into an H-dimensional (1×H) household-level feature vector, wherein the household-level features are different from the video block-level features, wherein H is the number of household-level features, and wherein the household-level features include the total number of video blocks reported for the return path data household within the specific observation period and the total number of tuners known to be included in the return path data household; implementing a neural network to process the video block-level feature vector of each return path data household generated based on the return path data to predict the demographic classification probability of the return path data household, wherein the neural network is trained based on panel data reported from meters that monitor media devices associated with panelist households, the neural network including: a time-distributed dense layer configured to reduce, for each return path data household, the video block-level features to a compressed set of features having a quantity less than the video block-level features, wherein the time-distributed dense layer includes a set of weights to map the video blocks of the video block-level features into the compressed set of features; a recurrent neural network layer configured to generate, for each return path data household, a one-dimensional feature vector by processing the compressed set of features; and a merging layer configured to generate, for each return path data household, a merged feature vector by merging the features of the generated one-dimensional feature vector with the household-level feature vector of the return path data household;and assign one or more demographic categories to the corresponding return path data households in the return path data households based on the predicted demographic classification probabilities; BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of an example processing flow for estimating demographic classification probabilities based on set-top box return path data using a neural network in accordance with the teachings of the present disclosure.

[0009] Figure 2 is in accordance with the teachings of the present disclosure using Figure 1 is a block diagram of an example processing flow for assigning demographics to households using the demographic classification probabilities estimated by the example processing flow.

[0010] Figure 3 is configured to implement in accordance with the teachings of the present disclosure Figure 1 and Figure 2 is a block diagram of an example neural network-based demographic estimation system for estimating household demographics based on set-top box return path data by implementing the processing flows of

[0011] Figures 4A to 4B illustrates example features generated by an example feature generator included in an example neural network-based demographic estimation system of Figure 3

[0012] Figure 5 is included in Figure 3 is a block diagram of an example implementation of an example demographic prediction neural network included in the example neural network-based demographic estimation system of

[0013] Figures 6A to 6C illustrates an example operation of a demographic prediction neural network in accordance with the teachings of the present disclosure of Figure 3 to estimate demographic classification probabilities based on set-top box return path data.

[0014] Figure 7 illustrates example pseudocode for implementing an example household demographic assignment engine included in an example neural network-based demographic estimation system of Figure 3

[0015] Figures 8A to 8E illustrates an example operation of a household demographic assignment engine in accordance with the teachings of the present disclosure of Figure 3 to assign demographics to households.

[0016] Figures 9A to 9C illustrates example simulated annealing operations that can be performed by a household demographic assignment engine of Figure 3

[0017] Figure 10 is a flowchart of example machine - readable instructions that can be executed to implement Figure 3 a neural - network - based demographic estimation system.

[0018] Figure 11 is configured to execute Figure 10 example machine - readable instructions to implement Figure 3 a block diagram of an example processor platform of an example neural - network - based demographic estimation system.

[0019] These figures are not drawn to scale. In general, the same reference numerals are used throughout the drawings and the accompanying written description to refer to the same or similar parts, elements, etc. DETAILED DESCRIPTION

[0020] Disclosed herein are example methods, devices, systems, and articles of manufacture (e.g., physical storage media) for implementing neural - network processing of return - path data to estimate household demographics. Example demographic estimation systems disclosed herein include a feature generator that generates features based on return - path data reported from a set - top box associated with a return - path data household. Example demographic estimation systems disclosed herein also include a neural network that processes the features generated based on the return - path data to predict demographic classification probabilities for the return - path data household. Example demographic estimation systems disclosed herein also include a demographic assignment engine that assigns one or more demographic categories to the corresponding return - path data household in the return - path data household based on the predicted demographic classification probabilities.

[0021] Below, these and other example methods, devices, systems, and articles of manufacture (e.g., physical storage media) for implementing neural - network processing of return - path data to estimate household demographics are disclosed in further detail.

[0022] As mentioned above, AME extrapolates ratings metrics and / or other audience data for the overall television audience from a relatively small sample of panelist households (also referred to herein as panel households). Panel households may have been studied in depth and are generally selected to represent the entire audience universe. However, it remains a challenge to accurately represent the geographical distribution and demographic diversity present in the total audience population using a small number of panel households. Incorporating additional information streams regarding media exposure into the total audience population can fill the gaps or biases inherent in any statistical sample.

[0023] To help supplement panel data, an AME such as The Nielsen Company (US), LLC may enter into an agreement with a pay-TV provider company to obtain television tuning information from set-top boxes, which is referred to in this document and in the industry as return path data (RPD). Set-top box (STB) data includes all data collected by the set-top box. STB data can, for example, include tuning events and / or commands received by the STB (e.g., power on, power off, change channel, change input source, start presenting media, pause media presentation, record media presentation, increase / decrease volume, etc.). Additionally or alternatively, STB data can include commands sent by the STB to content providers (e.g., switch input source, record media presentation, delete recorded media presentation, time / date of start of media presentation, completion time of media presentation, etc.), heartbeat signals, etc. Additionally or alternatively, set-top box data can include a home identifier (e.g., home ID) and / or an STB identifier (e.g., STB ID).

[0024] Return path data includes any data that can be received at a media service provider (such as a cable TV service provider, satellite TV service provider, streaming media service provider, content provider, etc.), for example, via a return path from a media consumer location to a service provider. Thus, return path data includes at least a portion of the set-top box data. Additionally or alternatively, return path data can include data from any other consumer device having network access capabilities (e.g., via a cellular network, the Internet, other public or private networks, etc.). For example, return path data can include any one or all of linear real-time data from an STB, boot user data from a boot server, clickstream data, key data (e.g., any click on a remote control - volume, mute, etc.), interactive activities such as video-on-demand, and any other data (e.g., data from middleware). Additionally or alternatively, RPD data can come from a network (e.g., via switched digital software) and / or any cloud-based data from the cloud (such as a remote server DVR).

[0025] RPD can provide insights into media exposure associated with a larger segment of the audience. This is because, compared to the households included in the AME panel households, RPD typically provides a richer stream of television viewing information to a much larger number of households. However, unlike the AME panel households that have been studied in depth, the demographic details of pay-TV subscribers are generally unknown. The lack of such demographic details in RPD leads to the following technical problem: preventing or at least restricting the ability to effectively use RPD to supplement the panel data of AME, because monitoring the behavioral profiles of various audience demographics requires knowledge of the demographic composition of the subscriber households that provide RPD.

[0026] The neural network processing of the set-top box RPD as disclosed herein to estimate household demographics provides a technical solution to the technical problem of combining RPD with panel data for audience measurement. As discussed further below in detail, an example neural network-based demographic estimation system implemented in accordance with the teachings of the present disclosure uses panel data collected from monitored AME panel households as a training set for training a neural network (e.g., a recurrent neural network) to be able to predict the probabilities of different household demographic characteristics associated with the corresponding RPD households in the RPD households reporting RPD data, based on the RPD tuning data that describes historical television tuning behavior. Then, the disclosed example neural network-based demographic estimation system predicts the demographic composition of households using the predicted probabilities of different household demographic characteristics. In this way, the example neural network-based demographic estimation system assigns a demographic composition to the subscriber households that provide RPD, thereby allowing RPD to be combined with or otherwise enhance the panel data driving the AME audience measurement system.

[0027] Turning to the drawings, Figure 1 A block diagram of an example processing flow 100 for estimating demographic classification probabilities based on a set-top box RPD using a neural network in accordance with the teachings of the present disclosure is illustrated. The example processing flow 100 includes: an example data collection phase 105, an example feature generation phase 110, and an example neural network demographic probability prediction phase 115. The example processing flow 100 is also divided into an example neural network training branch 120 and an example neural network application branch 125.

[0028] During the data collection phase 105 of the neural network training branch 120, example panelist engagement data 130 is collected from a meter that monitors media engagement in panel households recruited by AMEs. The panelist engagement data 130 may include any data that can be collected by a meter, such as, but not limited to, data identifying media presented by media devices in the panel households, demographic data identifying characteristics of panelists in the panel households, and the like. During the feature generation phase 110 of the neural network training branch 120, example features 135 are generated based on the collected panelist engagement data 130 and arranged to form a feature vector, as described in further detail below. During the neural network demographic probability prediction phase 115 of the neural network training branch 120, a neural network is trained to predict the probabilities of different household demographic characteristics associated with different panel households based on the features 135 generated from the collected panelist engagement data 130, as described in further detail below.

[0029] In the data collection phase 105 of the neural network application branch 125, example RPD tuning data 145 is collected from set-top boxes of one or more pay-TV providers (e.g., cable TV service providers, satellite TV service providers, streaming media service providers, content providers, etc.). A set-top box may also refer to any decoder, receiver, integrated receiver-decoder (IRD), media device, etc., from which RPD tuning data 145 may be collected. In the feature generation phase 110 of the neural network application branch 125, example features 150 are generated based on the collected RPD tuning data 145 and arranged to form a feature vector, as described in further detail below. In the neural network demographic probability prediction phase 115 of the neural network application branch 125, the trained neural network is applied to the features 150 generated based on the collected RPD tuning data 145 to predict example estimated probabilities 160 of different household demographic characteristics associated with different RPD subscriber households that reported the RPD tuning data 145, as described in further detail below.

[0030] Figure 2 The use of the teachings of the present disclosure is illustrated by Figure 1 14. A block diagram of an example process flow 200 for assigning demographics to households based on estimated demographic class probabilities 160 predicted by the example process flow 100 is shown. As discussed in further detail below, the process flow 200 utilizes an example mixed integer programming solution 205 that solves a constrained optimization problem based on the estimated demographic class probabilities 160 predicted by the example process flow 100, for example, providing an estimated demographic composition 210 of a subscriber's household for RPD tuning data 145.

[0031] Figure 3Illustrated are processing flows 100 and 200 configured to respectively implement Figure 1 and Figure 2 and a block diagram of an example neural network-based demographic estimation system 300 for estimating household demographics based on set-top box RPDs. The example neural network-based demographic estimation system 300 includes: an example network interface 305, an example panel tuning data collector 310, an example panelist database 315, an example RPD data collector 320, an example RPD database 325, an example feature generator 330, an example demographic prediction neural network 335, an example household demographic assignment engine 340, an example constraint database 345, and an example ratings calculator 350.

[0032] In the illustrated example, the panel tuning data collector 310 collects panelist tuning data 130 from example meters 355A to 355B via the network interface 305 that communicates with one or more example networks 355, and these example meters monitor media exposure associated with example media devices 360A to 360B (e.g., televisions, radios, computers, tablet devices, smart phones, etc.) in panel households recruited by AME. The panel tuning data collector 310 stores the collected panelist tuning data 130 in the panelist database 315. In the illustrated example, the RPD data collector 320 collects RPD tuning data 145 from one or more example service providers 370 via the network interface 305 that communicates with the one or more networks 355, and these service providers collect RPD tuning data 145 from example individual STBs 375 in subscriber households. Additionally or alternatively, in some examples, the RPD data collector 320 directly collects RPD tuning data 145 from one or more of the STBs in the individual STBs 375 in subscriber households via the network interface 305 that communicates with the one or more networks 355. The RPD data collector 320 stores the collected RPD tuning data 145 in the RPD database 325.

[0033] The exemplary example feature generator 330 generates features and feature vectors used by the exemplary demographic prediction neural network 335. In some examples, the RPD tuning data consists of an ordered log of when the respective set-top boxes are tuned to different stations. During a continuous television viewing session, an individual (e.g., an audience member) switches between multiple networks, and this pattern of activity can separately provide additional information about the household beyond the tuning records. To capture this behavior, the feature generator 330 compiles the STB records of television tuning into "view blocks" that aggregate the viewing behavior of one or more unknown viewers into a fixed number of features to summarize each successive viewing session. In some examples, the view block duration is capped at 1 hour or some other duration to account for the situation where multiple viewers can control the television without turning it off between sessions. In the exemplary example, each view block contains F features that record information about the start time of the view block, the channel click-through rate, the duration of the viewing session, and a list of the television stations accessed during the session.

[0034] Figures 4A to 4B Illustrates an example operation of the feature generator 330 to combine example RPD tuning data records 405 from the RPD adjustment data 145 into respective example view blocks 410 and 415. In Figure 4A the illustrative example, the respective data records in the data record 405 record the tuning events reported by the STB 375. A given data record 405 specifies an STB identifier (STB ID) 420 that identifies the STB corresponding to the event log, a start time 425 and an end time 430 that respectively correspond to the tuning event represented by the event log, a source identifier (SID) 435 that identifies the media source associated with the tuning event (e.g., channel number, station identifier, etc.), and a broadcast time 440 that identifies when the media associated with the tuning event was originally broadcast (e.g., to distinguish between live and time-shifted tuning events). In Figure 4B the illustrative example, the view block 410 aggregates the following tuning events recorded in the data record 405 of a given household: the tuning event occurred within the hour interval starting at 8:23 am on November 5, 2016. In Figure 4B the illustrative example, the view block 415 aggregates the following tuning events recorded in the data record 405 of a given household: the tuning event occurred within the hour interval starting at 6:04 pm on November 6, 2016.

[0035] The exemplary example feature generator 330 groups view blocks by household and assembles a set of N view blocks into a two-dimensional (N×F) matrix that contains a record of the view blocks generated by the household during a given observation period. In some examples, the feature generator 330 aggregates related household-level features (including the number of television tuners and the amount of television watched) with the view block data into an H-dimensional (1×H) additional feature vector for each household.

[0036] In some examples, each view block is a (1×173) feature vector that describes a corresponding television viewing session. Thus, for this example, the F dimension of the corresponding (N×F) matrix is 173. Table 1 illustrates the content of an example view block represented as a (1×173) feature vector.

[0037] Index Brief Description Valid Range 0 Day of the Week 0-6 1 Day of the Year 0-364 2 Quarter of the Day 0-95 3 Channel Change Rate 0 - Inf 4-173 Minutes of Viewing Each Network 0-60

[0038] Table 1

[0039] The first three features in Table 1 are self-explanatory. The "channel change rate" feature in Table 1 is the ratio of the number of times the channel changes during the view block to the duration of the view block in minutes. The "minutes watched on each network" feature is the total number of minutes watched on each television station. In the example of Table 1, the upper limit of the duration of the view block is 60 minutes, so the sum of these features across all networks is <= 60.0 minutes. In some such examples, therefore, a viewing session can be associated with one or more view blocks. In the example of Table 1, each station is randomly assigned an index value between 4 and 173.

[0040] In some examples, view blocks (from panel households) that contain less than 5 minutes of television viewing behavior are not used to train the demographic prediction neural network 335. Then, the view blocks of each household (e.g., panel households for neural network training and RPD households for neural network application) are stacked into a two-dimensional matrix, for example, with 400 rows (e.g., N = 400). In some examples, households that generate fewer than 400 unique view blocks will be zero-padded by the feature generator 330 until they have 400 rows, while those with more than 400 will be truncated by the feature generator 330 to the first 400 rows. Then, the two-dimensional arrays of each household are stacked by the feature generator 330 to form a three-dimensional matrix that can be fed into the demographic prediction neural network 335.

[0041] In some examples, the feature generator 330 utilizes three home-level features H to augment the viewing data, and these three home-level features H are incorporated into the demographic prediction neural network 335 after the recursive layer, as described below. Table 2 illustrates an example set of the three home-level features H, which correspond to (i) the total tuning volume reported for a given home over different durations (e.g., 24-hour periods) covered by the viewing blocks (corresponding to index 0 in the table), (ii) the number of viewing blocks reported for a given home over different durations (corresponding to index 1 in the table), and (iii) the total number of tuners included in the first return path data home in the return path data home (corresponding to index 2 in the table).

[0042] Index Brief Description Valid Range 0 Total TV Consumption (Minutes) 1 - Inf 1 Number of Recorded Video Blocks 1 - Inf 2 Number of TV Tuners in the Household 1 - Inf

[0043] Table 2

[0044] In the illustrated example, the demographic prediction neural network 335 is configured to predict 20 variables (e.g., a 1×20 vector), which represent the probabilities of different home-level demographics present in the home (although in other example implementations of the demographic prediction neural network 335, variables representing other quantities of other demographics may alternatively or additionally be predicted). In the illustrated example, fourteen home demographic target variables predicted by the demographic prediction neural network 335 indicate the corresponding probabilities (e.g., likelihoods) of the presence of 14 different age-sex combinations in the home, an example of which is shown in Table 3.

[0045]

[0046]

[0047] Table 3

[0048] In addition to the presence variables of Table 3, in some examples, the demographic prediction neural network 335 also predicts six additional target variables, which describe the demographic profile of the head of household (HOH), an example of which is shown in Table 4.

[0049] Index HOH Characteristics 14 HOH Age 15 HOH Gender 16 Hispanic 17 European American 18 African American 19 Asian American

[0050] Table 4

[0051] Figure 5 Illustrated Figure 3Example implementation of the demographic prediction neural network 335. In some examples, generally, the two-dimensional (N×F) feature vectors (e.g., 400×173 feature vectors) generated for the respective households (e.g., groups and / or RPD households) in the household being processed are sparse (e.g., during a given view block, many of the broadcast networks represented by the feature vectors are never accessed). To streamline this input into a smaller feature subset, the demographic prediction neural network 335 includes an example Temporal Distribution Dense Layer (TDDL) 505 that learns a single set of weights to map each view block to a streamlined representation of the input (N×F', where F' << F). This compressed data is then fed into an example Long Short Term Memory (LSTM) recurrent neural network layer 510. The LSTM 510 sequentially examines each row of the view block matrix and uses this information to selectively update a singular internal state vector that encodes information from each viewing session / view block. The output of the LSTM 510 is a one-dimensional (1×F') feature vector that summarizes the evidence history of each observed household. Figure 5 The example demographic prediction neural network 335 includes an example concatenation layer 515 that concatenates additional (1×H) household-level features with the one-dimensional representation of the viewing data output from the LSTM 510. The additional (1×H) household-level features include details about the total number of devices in the household, the total number of minutes watched over the observation window, and the total number of view blocks recorded for a particular household over the observation window, as described above.

[0052] In Figure 5 the example demographic prediction neural network 335, the enlarged feature vector output from the concatenation layer 515 is first passed through one or more additional example hidden layers 520 and then output from an example output layer 525 as a (1×C) probability vector representing the respective predicted probabilities of the C possible demographic classes present in the household. The C demographic classes modeled by the demographic prediction neural network 335 need not be mutually exclusive (e.g., a household may contain multiple people of different ages / genders), so the output vector encodes the relative probabilities of the presence of each modeled household-level demographic in an unknown household.

[0053] Table 5 lists Figure 5 the example dimensions of the data at each stage of the example demographic prediction neural network 335. In Table 5: N is the total number of view blocks per household, F is the number of features in each view block, F' is the number of dense features generated by the TDDL 505, and H is the number of additional household-specific features.

[0054] Dimension Number of Nodes M 3 N 400 F 173 F' 30 C 20

[0055] Table 5

[0056] In some examples, to prevent the demographic prediction neural network 335 from overfitting and to enable it to generalize better, during each training epoch, the feature generator 330 shuffles the order of the chunks fed into the demographic prediction neural network 335.

[0057] Figures 6A to 6C An example operation of the demographic prediction neural network 335 is illustrated to predict the demographic target variables 605 to 620 as the feature vectors 625 to 635 generated from the RPD tuning data 145 are applied to the demographic prediction neural network 335 after the demographic prediction neural network 335 has been trained with the feature vectors generated from the panelist data 130. In the illustrated example, the demographic prediction neural network 335 is trained by: (i) creating view chunks based on the panelist tuning data 130 of the reported panelist households; (ii) generating features for the corresponding panelist households in the panelist households based on the created view chunks of the corresponding panelist households, as described above; and (iii) applying the features of the corresponding panelist households in the panelist households to the neural network 335 according to any training process that adjusts the internal parameters of the neural network to reduce the error between the predicted demographic classification probability 160 output by the neural network 335 and the actual demographics of the known panelist households. As Figures 6A to 6C illustrated in the example, as more view chunks are applied to train the neural network 335, the output of the network 335 will converge to predict the demographic classification probability 160 that matches the actual demographics of the known panelist households.

[0058] Returning to Figure 3 , according to the teachings of the present disclosure, the example household demographic assignment engine 340 of the example neural network-based demographic estimation system 300 uses the estimated demographic classification probability (also referred to above as the predicted demographic target variable) output from the demographic prediction neural network 335 to assign demographics to the RPD households. Figure 7 An example pseudocode 700 for implementing the household demographic assignment engine 340 is illustrated. This example pseudocode 700 also corresponds to Figure 2 an example of the mixed integer programming solution 205 of Figure 7In the illustrative example, given a cost matrix C0 (which represents the cost of assigning different demographic categories to RPD households) that is subject to a set of constraints (the values of these constraints are stored in the example constraint database 345), the pseudocode 700 for implementing the household demographic assignment engine 340 determines the matrix X0 by solving an objective function, thereby assigning the demographics of the households. The matrix X0 is a Boolean matrix that represents the demographic categories assigned to different RPD households. In Figure 7 the example of Figure 7 the matrix X0 is a matrix with the number of rows equal to the number of RPD households and the number of columns equal to the number of different possible demographic categories that can be assigned to a household. Moreover, in the illustrative example, for a given row of X0 that represents a given RPD household, the elements of that row contain binary (Boolean) variables that represent the different possible demographic categories. If a given possible demographic category is assigned to that RPD household, the pseudocode 700 assigns the value 1 to the given binary variable that represents that demographic category; or if that demographic category is not assigned to that RPD household, the pseudocode 700 assigns the value 0 to that binary variable. In

[0059] the example of Figure 7 the pseudocode 700 uses any mixed-integer programming or similar technique to determine the demographic assignment matrix X0 by solving the following objective function that is subject to a set of constraints:

[0060]

[0061] Figure 7 the example constraints are based on: a matrix X1, which is a Boolean matrix that represents the different possible household sizes that can be assigned to different RPD households; and a size matrix S1 that represents the values of the different possible household sizes.

[0062] Figures 8A to 8E Illustrates an example operation of the household demographics assignment engine 340 implemented by the Figure 7 pseudocode 700 to assign a demographics category to an RPD household by solving the above expression subject to the Figure 7 example constraints. Figure 8A Illustrates an example C0 cost matrix 805 that has 5 rows representing 5 RPD households to which a demographics category is to be assigned, and 4 columns representing 4 possible demographics categories that can be assigned to the corresponding RPD household among the RPD households. The cost values for the different possible demographics categories are represented by dollar signs ($) in Figure 8A and the more dollar signs, the higher the cost. In the illustrated example, the cost included in the C0 cost matrix 805 is inversely proportional to the corresponding estimated demographics classification probability (also referred to above as the predicted demographics target variable) for the given household and demographics category output combination output from the output layer 525 of the demographics prediction neural network 335.

[0063] Referring to Figure 7 and Figures 8A to 8E , Figure 7 the example constraints include a first constraint 705 that specifies that the sum of the different demographics categories assigned to all RPD households is equal to the known overall estimate (UE) of the corresponding different demographics categories (e.g., within a tolerance level represented by the variable "slack"). Figure 8B Illustrates an example of the first constraint 705, where the sum of the corresponding demographics categories assigned to 5 households is equal to the corresponding example UE 810 for the different demographics categories (e.g., can be obtained from a service provider providing the RPD and stored in the constraint database 345). For example, in Figure 8B the first constraint 705 specifies that the number of households to be assigned the demographics category "man" is equal to the UE for this demographics category of 2; the number of households to be assigned the demographics category "woman" is equal to the UE for this demographics category of 4; the number of households to be assigned the demographics category "girl" is equal to the UE for this demographics category of 3; and the number of households to be assigned the demographics category "man" is equal to the UE for this demographics category of 2.

[0064] Figure 7 the example constraints include a second constraint 710 that specifies that there must be at least one adult demographics category assigned to each RPD household. Figure 8B Illustrates an example of the second constraint 710, where each RPD household is constrained to include the demographics category "man" and / or the demographics category "woman" (represented by reference numeral 815).

[0065] Figure 7 Example constraints include a third constraint 715 that specifies that the total number of different possible household sizes assigned to all RPD households is equal to a known ensemble estimate (UE) of the different possible household sizes (eg, within a tolerance level represented by the variable "slack"). Figure 8D An example of a third constraint 715 is illustrated, where the number of corresponding possible family sizes assigned for 5 families is equal to the corresponding example UEs 820 for the different possible family sizes (e.g., which can be obtained from a service provider providing RPD and stored in the constraint database 345). Figure 8D , the third constraint 715 specifies that: the number of households containing two people is equal to the UE of the household size is 3; the number of households containing three people is equal to the UE of the household size is 1; and the number of households containing four people is equal to the UE of the household size is 1.

[0066] Figure 7 Example constraints include a fourth constraint 720 specifying that each RPD household is to be assigned only one of the possible household sizes, and a fifth constraint 725 specifying that the number of different demographic categories assigned to a given RPD household is equal to the household size assigned to that household. Figure 8E Illustrated by using Figure 7 The pseudo code 700 implements the family demographic assignment engine 340 and the given Figures 8A to 8D The resulting example demographic category assignment 825 determined by the example constraints 705 to 725 is shown. Figure 8E In the example of FIG, the household demographic assignment engine 340 implemented using pseudo code 700 solves the above (and Figure 7 ) to assign: (1) the demographic categories of "women" and "boys" to the first RPD family, (2) the demographic categories of "women" and "boys" to the second RPD family, (3) the demographic categories of "men" and "girls" to the third RPD family, (4) the demographic categories of "women", "girls", and "boys" to the fourth RPD family, and (5) the demographic categories of "men", "women", "girls", and "boys" to the fifth RPD family. As can be seen in Figures 8A to 8E As seen in the example of , demographic category assignment 825 satisfies the specified constraints.

[0067] In some examples, household demographic assignment engine 340 implements simulated annealing to further adjust the demographic category assignments made for RPD households. Figures 9A to 9C An example operation of the household demographic assignment engine 340 performing simulated annealing is illustrated.Figure 9A , in the illustrated example, the household demographics assignment engine 340 has performed an initial household demographics assignment, where the demographics category assignment that includes both "boys" and "girls" is over-represented in five households relative to the UE constraints for that demographics category combination, and the demographics category assignment that includes both "men" and "girls" is under-represented in five households relative to that demographics category combination. As ​ shown, the household demographics assignment engine 340 can perform simulated annealing to identify the five households with the demographics category assignment that includes both over-represented "boys" and "girls" (see ​ ), and transfer the demographics category assignment of "girls" from those households to the five households that do not have the demographics category of both "men" and "girls" (see ​ ). The result is a corrected ​ of the over-representation and under-representation illustrated in the revised demographics category assignment.

[0068] In some examples, the household demographics assignment engine 340 breaks down ​ the illustrated demographics assignment problem into several smaller batches to reduce processing and memory requirements. For example, if the market contains 100,000 RPD households for which demographics categories are to be assigned, then the household demographics assignment engine 340 can break down the assignment problem into 100 groups of 1,000 households each or 1,000 groups of 100 households each, etc. In such an example, the ​ pseudocode 700 is adjusted such that the constraints related to the universe estimate (UE) are scaled by the ratio of the number of RPD households included in the batch group to the total number of RPD households, and the pseudocode 700 is applied to perform demographics category assignment for each batch group. However, since this simple scaling may not result in solvable constraints for all batch groups, a tolerance level (e.g., represented by "slack" in ​ ) is included to increase the likelihood that each batch group will have a solvable demographics assignment.

[0069] Returning to ​ , the neural-network-based demographics estimation system 300 includes a ratings calculator 350 that determines ratings data and / or other audience metrics by using the household demographics assignment determined by the household demographics assignment engine 340 for RPD households to augment / combine the panel tuning data from the panelist database 315, which already has associated demographics data, with the RPD tuning data from the RPD database 325.

[0070] Although ​ illustrates an example manner of implementing the neural network-based demographic estimation system 300, ​ one or more of the illustrated elements, processes, and / or apparatuses may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Additionally, the example network interface 305, the example panel tuning data collector 310, the panelist database 315, the example RPD data collector 320, the example RPD database 325, the example feature generator 330, the example demographic prediction neural network 335, the example household demographic assignment engine 340, the example constraint database 345, the example ratings calculator 350, and / or more generally ​The example neural network-based demographic estimation system 300 can be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the example network interface 305, the example panel tuning data collector 310, the panelist database 315, the example RPD data collector 320, the example RPD database 325, the example feature generator 330, the example demographic prediction neural network 335, the example household demographic assignment engine 340, the example constraint database 345, the example rating calculator 350, and / or more generally the example neural network-based demographic estimation system 300 can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and / or field programmable logic devices (FPLDs). When any of the apparatus or system claims of this patent are construed to cover a pure software and / or firmware implementation, at least one of the example neural network-based demographic estimation system 300, the example network interface 305, the example panel tuning data collector 310, the panelist database 315, the example RPD data collector 320, the example RPD database 325, the example feature generator 330, the example demographic prediction neural network 335, the example household demographic assignment engine 340, the example constraint database 345, and / or the example rating calculator 350 is hereby expressly defined to include: a non-transitory computer-readable storage device or storage disk including software and / or firmware, such as a memory, a digital versatile disc (DVD), a compact disc (CD), a Blu-ray disc, etc. Further, the example neural network-based demographic estimation system 300, in addition to ​ the elements, processes, and / or apparatus illustrated or instead of ​The illustrated elements, processes, and / or devices may also include one or more other elements, processes, and / or devices, and / or may include more than one of any or all of the illustrated elements, processes, and devices. As used herein, the phrase "in communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but rather includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events.

[0071] ​ A flowchart representing example hardware logic, machine readable instructions, hardware implemented state machines, and / or any combination thereof, for implementing an example neural network based demographic estimation system 300 is shown. In this example, the machine readable instructions may be instructions for executing a program executed by a computer processor (such as the one described below in conjunction with ​ The processor 1112 shown in the example processor platform 1100 discussed above is executed by one or more executable programs or a portion of the executable program. The executable program may be stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or the like. TM The one or more programs or a portion of the program may be implemented in software on a processor (or a memory associated with the processor 1112), but the entire program and / or its portion may alternatively be executed by other means other than the processor 1112, and / or implemented in firmware or dedicated hardware. In addition, although reference is made to ​ The example process is described by the flowchart illustrated in FIG, but many other methods of implementing the example neural network-based demographic estimation system 300 may alternatively be used. For example, referring to ​ The illustrated flowcharts may change the order of execution of the blocks, and / or may change, eliminate, combine some of the blocks, and / or subdivide the blocks into multiple blocks. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform corresponding operations without executing software or firmware.

[0072] As mentioned above, ​Example processing can be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transitory computer and / or machine-readable medium (such as a hard disk drive, flash memory, read-only memory, optical disk, digital versatile disk, cache memory, random access memory, and / or any other storage device or storage disk, where information is stored for any duration (e.g., for an extended period, permanently, for a simple instance, for temporary buffering, and / or for caching information)). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and / or storage disk, and to exclude propagated signals and to exclude transmission media. Moreover, as used herein, unless otherwise indicated, the terms "computer-readable" and "machine-readable" are considered synonymous.

[0073] "Including" and "comprising" (and all of their forms and tenses) are used herein as open-ended terms. Thus, whenever a claim employs any form of "including" or "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within any kind of claim statement, it is to be understood that additional elements, terms, etc. may exist without falling outside the scope of the corresponding claim or statement. As used herein, when the phrase "at least" is used as a transitional term in, for example, the preamble of a claim, it is open-ended in the same manner as the terms "comprising" and "including" are open-ended. The term "and / or" when used, for example, in the form such as A, B, and / or C, refers to any combination or subset of A, B, C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing a structure, component, item, object, and / or thing, the phrase "at least one of A and B" is intended to refer to an implementation that includes (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B. Similarly, as used herein in the context of describing a structure, component, item, object, and / or thing, the phrase "at least one of A or B" is intended to refer to an implementation that includes (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B. As used herein in the context of describing the implementation or execution of a process, instruction, action, activity, and / or step, the phrase "at least one of A and B" is intended to refer to an implementation that includes (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B. Similarly, as used herein in the context of describing the implementation or execution of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B.

[0074] ​ The illustrated flowcharts depict example programs 1000 of an example neural network-based demographic estimation system 300 that may be executed to implement ​ Referring to the previous figures and the associated written description, ​The example program 1000 begins execution at block 1005, where the example panel tuning data collector 310 of the neural network-based demographic estimation system 300 collects panelist tuning data, as described above. At block 1010, the example feature generator 330 of the neural network-based demographic estimation system 300 generates a feature vector for the panelist's family (e.g., such as the vectors described in Table 1 above) based on the collected panelist data, as described above. At block 1015, the feature generator 330 applies the panelist feature vector generated at block 1010 to the example demographic prediction neural network 335 of the neural network-based demographic estimation system 300 to train the demographic prediction neural network 335 to predict the demographic classification probability for the corresponding panelist's family, as described above.

[0075] At block 1020, the example RPD data collector 320 of the neural network-based demographic estimation system 300 collects RPD tuning data, as described above. At block 1025, the example feature generator 330 generates a feature vector for the RPD family (e.g., such as the vectors described in Table 1 above) based on the collected RPD tuning data, as described above. At block 1030, the feature generator 330 applies the RPD feature vector generated at block 1025 to the trained demographic prediction neural network 335 of the neural network-based demographic estimation system 300 to predict the demographic classification probability for the corresponding RPD family, as described above. At block 1035, the example household demographic assignment engine 340 of the neural network-based demographic estimation system 300 uses the demographic classification probability determined at block 1030 to assign a demographic category to the corresponding RPD family in the RPD household, as described above. At block 1045, the example ratings calculator 350 of the neural network-based demographic estimation system 300 uses the demographic category assigned to the corresponding RPD family in the RPD household at block 1045 and the RPD tuning data collected at block 1020 to augment / combine the panel tuning data collected at block 1005, as described above.

[0076] ​ is configured to execute ​ instructions to implement ​ is a block diagram of an example processor platform 1100 of an example neural network-based demographic estimation system that is configured to execute TM instructions to implement. The processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smartphone, a tablet such as an iPad [5]]

[0077] The exemplary example processor platform 1100 includes a processor 1112. The exemplary example processor 1112 is hardware. For example, the processor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor 1112 can be a semiconductor (e.g., silicon-based) device. In this example, the processor 1112 implements the exemplary group tuning data collector 310, the exemplary RPD data collector 320, the exemplary feature generator 330, the exemplary demographic prediction neural network 335, the exemplary household demographic assignment engine 340, and the exemplary ratings calculator 350.

[0078] The exemplary example processor 1112 includes local memory 1113 (e.g., cache memory). The exemplary example processor 1112 communicates with a main memory including volatile memory 1114 and non-volatile memory 1116 via a link 1118. The link 1118 can be implemented by a bus, one or more point-to-point connections, etc., or a combination thereof. The volatile memory 1114 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), dynamic random access memory and / or any other type of random access memory device. The non-volatile memory 1116 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1114, 1116 is controlled by a memory controller.

[0079] The exemplary example processor platform 1100 also includes an interface circuit 1120. The interface circuit 1120 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI expansion interface. In this example, the interface circuit 1120 implements the network interface 305.

[0080] In an illustrative example, one or more input devices 1122 are connected to interface circuit 1120. The input device 1122 permits a user to input data and / or commands into processor 1112. The input device may be implemented, for example, by an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touch screen, a touchpad, a trackball, a trackbar (such as an isopoint), a voice recognition system, and / or any other human-machine interface. Moreover, many systems (such as processor platform 1100) may enable a user to control the computer system and provide data to the computer using physical gestures (such as, but not limited to, hand or body movements, facial expressions, and facial markers).

[0081] One or more output devices 1124 are also connected to the illustrative example interface circuit 1120. The output device 1124 may be implemented, for example, by a display device (such as, a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a haptic output device, a printer, and / or a speaker. The illustrative example interface circuit 1120 thus generally includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0082] The illustrative example interface circuit 1120 also includes a communication device (such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface) to facilitate data exchange with an external machine (such as, any kind of computing device) via network 1126. The communication may be, for example, via an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0083] The illustrative example processor platform 1100 also includes one or more mass storage devices 1128 for storing software and / or data. Examples of such mass storage devices 1128 include: a floppy disk drive, a hard disk drive, an optical disk drive, a Blu-ray disk drive, a redundant array of independent disks (RAID) system, and a digital versatile disk (DVD) drive. In some examples, the mass storage device 1128 may implement the team member database 315, the RPD database 325, and / or the constraint database 345. Additionally or alternatively, in some examples, the volatile memory 1114 may implement the team member database 315, the RPD database 325, and / or the constraint database 345.

[0084] With ​The machine-executable instruction 1132 corresponding to the instruction can be stored in the mass storage device 1128, volatile memory 1114, non-volatile memory 1116, local memory 1113, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD 1136.

[0085] In accordance with the foregoing, it should be appreciated that example methods, apparatuses, and articles of manufacture for implementing neural network processing of set-top box return path data to estimate household demographics have been disclosed. The example neural network-based demographics estimation system 300 disclosed above uses a neural network having a time-distributed dense layer (TDDL) and a subsequent long short-term memory (LSTM) recurrent network layer to predict the demographics classification of a household (e.g., the panel households for training and the RPD households after training) based on viewing data (e.g., panelist tuning data for training and RPD tuning data after training). The example neural network-based demographics estimation system 300 groups the viewing data of a household into view blocks that describe the corresponding viewing sessions, where the view blocks indicate the day of the week, the day of the year, the quarter of the day, the channel change rate, and the number of minutes of viewing each possible network. In some examples, the time limit for a viewing block is 60 minutes. In some examples, the view blocks of a given household are combined and processed by the TDDL to generate a reduced feature set for the viewing session of that household. The reduced feature set is then processed by the LSTM to generate a reduced summary feature vector that summarizes the viewing history of that household. The reduced summary feature vector is combined with additional household features (such as total TV consumption, the number of recorded view blocks, and the number of TV tuners in the household) to generate a combined summary feature vector for that household. The combined summary feature vector is then applied to one or more additional hidden layers that output a classification vector indicating the probabilities that the household belongs to different possible demographics classes. Then, mixed integer programming is used to solve an objective function based on the demographics classification probabilities output from the neural network and subject to a series of constraints to assign one or more demographics classes to the corresponding RPD households in the RPD households that provide the RPD tuning data.

[0086] The disclosed methods, apparatuses, and articles of manufacture improve the efficiency of using a computing device by enabling RPD tuning data to be combined with panelist tuning data in an audience measurement processing system. Combining the RPD tuning data with the available panel data can significantly increase the amount of data that the audience measurement processing system can use to predict audience metrics (e.g., ratings). This increased amount of data can improve the statistical integrity of the input data, thereby reducing the associated statistical bias in the results generated by the audience measurement processing system. Accordingly, the disclosed methods, apparatuses, and articles of manufacture are directed to one or more improvements in computer functionality.

[0087] Although certain example methods, apparatuses, and articles of manufacture are disclosed herein, the scope of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatuses, and articles of manufacture that fall entirely within the scope of the claims of this patent.

Claims

1. A demographic estimation system, the demographic estimation system comprising: A processor; And A memory storing machine-readable instructions that, when executed by the processor, cause operations to be performed including the following: Generating features based on return path data reported from set-top boxes associated with return path data households, wherein the generating includes: for the respective return path data reported by each return path data household: Compiling the respective return path data into a set of temporally contiguous video blocks generated by the return path data household within a specific observation period, wherein each video block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective return path data into a plurality of video block-level features represented by a video block-level feature vector, the video block-level features including: A channel change rate determined based on a ratio of (i) the number of channel changes that occurred during the time interval to (ii) the duration of the time interval; For each of a plurality of stations to which the set-top box of the return path data household is configured to tune, the respective total number of minutes the station was watched; and A list of stations of the plurality of stations accessed during the time interval; and Generating a two-dimensional N×F video block-level feature vector representing a record of all video blocks of the return path data household within the specific observation period, where N is the total number of video blocks in the set of video blocks, and F is the total number of video block-level features; For each return path data household, aggregating household-level features into a 1×H household-level feature vector of dimension H, where the household-level features are different from the video block-level features, where H is the number of household-level features, and where the household-level features include the total number of video blocks reported for the return path data household within the specific observation period and the total number of tuners known to be included in the return path data household; training a neural network that processes the video block-level feature vector of each return path data household generated based on the return path data to predict the demographic classification probability of the return path data household based on panel data reported from meters monitoring media devices associated with panel member households, wherein the training includes: Creating video blocks for the panel member households based on the panel data, Generating a video block-level feature vector for a respective one of the panel member households based on the video blocks created for the respective panel member household, Applying the video block-level feature vector of the respective one of the panel member households to the neural network according to a training process of adjusting internal parameters of the neural network to reduce an error between a predicted demographic classification probability output by the neural network and the actual demographics of the panel member household known based on the panel data, and Shuffling the order of the video blocks of the return path data households fed into the neural network during each of a plurality of training cycles of the neural network, Among them, the neural network includes: A time-distributed dense layer configured to reduce, for each return path data family, the view block-level features to a compressed feature set having a number less than that of the view block-level features, wherein the time-distributed dense layer includes a set of weights to map the view blocks of the view block-level features into the compressed feature set; A recurrent neural network layer configured to generate, for each return path data family, a one-dimensional feature vector by processing the compressed feature set; and A merging layer configured to generate, for each return path data family, a merged feature vector by merging the features of the generated one-dimensional feature vector with the family-level feature vector of the return path data family; and Implement the neural network to process the view block-level feature vectors of each return path data family generated from the return path data to predict the demographic classification probability of the return path data family; and Based on the predicted demographic classification probability, assign one or more demographic categories to the corresponding return path data family in the return path data family.

2. The demographic estimation system according to claim 1, wherein, The neural network further includes: A hidden layer configured to process the merged feature vector for each return path data family; and An output layer in communication with the hidden layer and configured to output, for each return path data family, the predicted demographic classification probability associated with the return path data family.

3. The demographic estimation system according to claim 1, wherein Assigning the one or more demographic categories to the corresponding return path data family in the return path data family includes: solving an objective function subject to a set of constraints to assign the one or more demographic categories to the corresponding return path data family in the return path data family, the objective function being based on the predicted demographic classification probability.

4. The demographic estimation system according to claim 3, wherein: The first of the constraints is for constraining the corresponding demographic categories among the demographic categories assigned across the return path data families to generate a sum corresponding to the corresponding total estimate of the corresponding demographic categories specified by the service provider associated with the return path data; The second of the constraints is for constraining the corresponding possible family sizes among the different possible family sizes assigned across the return path data families to generate a sum corresponding to the corresponding total quantity of the corresponding possible family sizes specified by the service provider associated with the return path data; And The third of the constraints is for constraining the corresponding quantity of the demographic categories assigned to the corresponding return path data family in the return path data family to correspond to the corresponding family size assigned to the corresponding return path data family in the return path data family.

5. The demographic estimation system according to claim 3, wherein, The demographic categories correspond to respective second sets of demographic categories assigned to respective return path data households, and wherein the operations further include: Performing a simulated annealing process on a respective first set of demographic categories assigned to the respective return path data household to determine the second set of demographic categories.

6. The demographic estimation system according to claim 1, wherein, The generating further includes: for respective return path data reported by each return path data household, discarding from the set of view blocks those view blocks that include less than five minutes of viewing.

7. The demographic estimation system according to claim 1, wherein the operations further include: Generating features based on the panel data reported by meters monitoring media devices associated with the panelist households, wherein the generating includes: for respective panel data reported by each panelist household, compiling the respective panel data into a set of panel view blocks generated by the panelist household during a particular observation period, wherein each panel view block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective panel data into a plurality of panel view block-level features represented by a panel view block-level feature vector; and Training the neural network using the panel view block-level feature vectors generated for the panelist households, wherein the training reduces the error between the predicted demographic classification probabilities of the return path data households output by the neural network and the known actual demographics of the panelist households. [[ID=*6]]8. A non-transitory computer-readable storage medium including computer-readable instructions that, when executed, cause a processor to perform operations including the following: Generate features based on return path data reported from set-top boxes associated with a return path data household, wherein, The generating includes: for respective return path data reported by each return path data household: Compiling the respective return path data into a set of temporally contiguous view blocks generated by the return path data household during a particular observation period, wherein each view block is associated with a respective different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the respective return path data into a plurality of view block-level features represented by a view block-level feature vector, the view block-level features including: A channel change rate determined based on a ratio of (i) the number of channel changes that occur during the time interval to (ii) the duration of the time interval; For each of a plurality of stations to which the set-top box of the return path data household is configured to tune, the respective total number of minutes the station is viewed; and A list of stations of the plurality of stations accessed during the time interval; and Generating a two-dimensional N×F view block-level feature vector representing a record of all view blocks of the return path data household during the particular observation period, wherein N is the total number of view blocks in the set of view blocks and F is the total number of view block-level features; For each return path data household, aggregate household-level features into a 1×H household-level feature vector of dimension H, where the household-level features are different from the view block-level features, where H is the number of household-level features, and where the household-level features include the total number of view blocks reported for the return path data household during the specific observation period and the total number of tuners known to be included in the return path data household; train a neural network that processes the view block-level feature vector of each return path data household generated from the return path data to predict the demographic classification probability of the return path data household based on panel data reported by meters monitoring media devices associated with panelist households, where the training includes: Create view blocks for the panelist households according to the panel data, Generate a view block-level feature vector for a respective one of the panelist households according to the view blocks created for the respective panelist household, Apply the view block-level feature vector of the respective one of the panelist households to the neural network according to a training process that adjusts internal parameters of the neural network to reduce the error between the predicted demographic classification probability output by the neural network and the actual demographics of the panelist household known based on the panel data, and During each of multiple training epochs of the neural network, shuffle the order of the view blocks of the return path data households fed into the neural network, where the neural network includes: A time-distributed dense layer configured to reduce, for each return path data household, the view block-level features to a compressed set of features having a quantity less than the view block-level features, where the time-distributed dense layer includes a set of weights to map the view blocks of the view block-level features into the compressed set of features; A recurrent neural network layer configured to generate, for each return path data household, a one-dimensional feature vector by processing the compressed set of features; and A merge layer configured to generate, for each return path data household, a merged feature vector by merging the features of the generated one-dimensional feature vector with the household-level feature vector of the return path data household; and Implement the neural network to process the view block-level feature vector of each return path data household generated from the return path data to predict the demographic classification probability of the return path data household; and Assign one or more demographic categories to the respective return path data household in the return path data households based on the predicted demographic classification probability.

9. The non-transitory computer-readable storage medium according to claim 8, wherein, The neural network further includes: A hidden layer configured to process the merged feature vector for each return path data household; and An output layer in communication with the hidden layer and configured to output, for each return path data household, the predicted demographic classification probability associated with the return path data household.

10. The non-transitory computer-readable storage medium according to claim 8, wherein, Assigning the one or more demographic categories to corresponding return path data households in the return path data households includes: solving an objective function subject to a set of constraints to assign the one or more demographic categories to corresponding return path data households in the return path data households, the objective function being based on predicted demographic classification probabilities.

11. The non-transitory computer-readable storage medium according to claim 10, wherein: A first constraint among the constraints is for constraining a corresponding demographic category among the demographic categories assigned across the return path data households to generate a sum corresponding to a corresponding total estimate of the corresponding demographic category specified by a service provider associated with the return path data; A second constraint among the constraints is for constraining a corresponding possible household size among different possible household sizes assigned across the return path data households to generate a sum corresponding to a corresponding total quantity of the corresponding possible household size specified by a service provider associated with the return path data; And A third constraint among the constraints is for constraining a corresponding quantity of demographic categories assigned to a corresponding return path data household in the return path data households to correspond to a corresponding household size assigned to the corresponding return path data household in the return path data households.

12. The non-transitory computer-readable storage medium according to claim 10, wherein, The demographic categories correspond to a corresponding second set of demographic categories assigned to corresponding return path data households, and wherein the operations further include: Performing a simulated annealing process on a corresponding first set of demographic categories assigned to the corresponding return path data households to determine the second set of demographic categories.

13. The non-transitory computer-readable storage medium according to claim 8, wherein The generating further includes: for the corresponding return path data reported for each return path data household, discarding sets of the view blocks that include view blocks watched for less than five minutes.

14. A demographic estimation method, the demographic estimation method being executed by a computing system including a processor and a memory, the demographic estimation method including: Generating features based on return path data reported from a set-top box associated with a return path data household, wherein the generating includes: for the corresponding return path data reported by each return path data household: Compiling the corresponding return path data into a set of temporally contiguous view blocks generated by the return path data household during a specific observation period, wherein each view block is associated with a corresponding different time interval, and aggregating a set of one or more tuning events corresponding to the time interval and included in the corresponding return path data into a plurality of view block-level features represented by a view block-level feature vector, the view block-level features including: A channel change rate determined based on a ratio of (i) the number of channel changes that occurred during the time interval to (ii) the duration of the time interval; For each of a plurality of radio stations to which the set-top box of the return path data household is configured to tune, the corresponding total number of minutes the radio station was watched; and a list of radio stations of the plurality of radio stations accessed during the time interval; and generating a two-dimensional N×F video block-level feature vector representing records of all video blocks of the return path data household within the specific observation period, where N is the total number of video blocks in the set of video blocks, and F is the total number of the video block-level features; for each return path data household, aggregating the household-level features into an H-dimensional (1×H) household-level feature vector, where the household-level features are different from the video block-level features, where H is the number of household-level features, and where the household-level features include the total number of video blocks reported for the return path data household within the specific observation period and the total number of tuners known to be included in the return path data household; training a neural network that processes the video block-level feature vector of each return path data household generated from the return path data to predict the demographic classification probability of the return path data household based on panel data reported by meters monitoring media devices associated with panelist households, where the training includes: creating video blocks for the panelist households according to the panel data, generating a video block-level feature vector of a corresponding one of the panelist households according to the video blocks created for the corresponding panelist household, applying the video block-level feature vector of the corresponding one of the panelist households to the neural network according to a training process of adjusting internal parameters of the neural network to reduce the error between the predicted demographic classification probability output by the neural network and the actual demographics of the panelist household known based on the panel data, and shuffling the order of the video blocks of the return path data households fed into the neural network during each of a plurality of training cycles of the neural network; wherein the neural network includes: a time-distributed dense layer configured to reduce, for each return path data household, the video block-level features to a compressed set of features having a quantity less than the video block-level features, where the time-distributed dense layer includes a set of weights to map the video blocks of the video block-level features into the compressed set of features; a recurrent neural network layer configured to generate, for each return path data household, a one-dimensional feature vector by processing the compressed set of features; and a merging layer configured to generate, for each return path data household, a merged feature vector by merging the features of the generated one-dimensional feature vector with the household-level feature vector of the return path data household; and implementing the neural network to process the video block-level feature vector of each return path data household generated from the return path data to predict the demographic classification probability of the return path data household; and assigning one or more demographic categories to the corresponding return path data household among the return path data households based on the predicted demographic classification probability.

15. The demographic estimation method according to claim 14, wherein, The neural network further includes: A hidden layer configured to process the combined feature vectors for each returned path data household; and An output layer in communication with the hidden layer and configured to output, for each returned path data household, the predicted demographic classification probabilities associated with the returned path data household.

16. The demographic estimation method according to claim 14, wherein, Assigning the one or more demographic categories to corresponding returned path data households in the returned path data households includes: solving an objective function subject to a set of constraints to assign the one or more demographic categories to corresponding returned path data households in the returned path data households, the objective function being based on the predicted demographic classification probabilities.

17. The demographic estimation method according to claim 16, wherein: A first of the constraints is for constraining a corresponding demographic category among the demographic categories assigned across the returned path data households to generate a sum corresponding to a respective total estimate of the corresponding demographic category specified by the service provider associated with the returned path data; A second of the constraints is for constraining a corresponding possible household size among different possible household sizes assigned across the returned path data households to generate a sum corresponding to a respective total quantity of the corresponding possible household size specified by the service provider associated with the returned path data; And A third of the constraints is for constraining a respective quantity of the demographic categories assigned to a corresponding returned path data household in the returned path data households to correspond to a respective household size assigned to the corresponding returned path data household in the returned path data households.

Images