SYSTEM AND METHOD FOR COLLECTING DATA FROM USER DEVICES - Patent application

Abstract

A system and method for rapidly and scalably tracking user presence on a user device. The system determines whether a human is present at the device, i.e., whether a human is present in a position where they can interact with content displayed on the device. The ability to track user presence may be linked to the ability to measure attention. The system may operate to collect sensor data during a period of information output by the user device and map the sensor data to presence parameters to obtain presence data indicative of variations in the presence parameters over time. The presence data is synchronized with contextual attribute data to generate a validity data set that links changes in the presence parameters over time with corresponding contextual attribute data obtained during the period of information output.

Description

[Technical Field]

[0001] The present invention relates to techniques for collecting various data, for example from different sources or software, while a device is outputting information or other perceptible data, and the collected data is used to evaluate the impact of the output information.

[0002] In one example, outputting information may include displaying content, and data may be collected during the display of the content. Herein, the displayed content may be any information that may be consumed by a user. For example, the content may be any of media content (e.g., video, music, images), advertising content, and web page information.

[0003] In another example, the output information may be related to the provision of some kind of interactive content. For example, the device may be used to participate in a video conference or the like. Alternatively, the device may be associated with an automated service provider (e.g., a robot or other interactive machine) configured to engage in an interaction. The collected data may be used to evaluate whether to initiate an interaction and / or to evaluate the effectiveness of the interaction.

[0004] The device may be any consumer electronic device, such as a smartphone, tablet, desktop or laptop computer, etc. The displayed content may, for example, be stored on the device and / or generated locally on the device. Alternatively or additionally, the device may operate in a networked environment, and the content for display is available over a network.

[0005] In particular, the present invention relates to a scalable technique for detecting human presence at a user device when content is displayed on the user device. [Background technology]

[0006] Certain types of media content, such as advertisements, music videos, and movies, aim to provoke a change in a consumer's emotional state, e.g., to capture or otherwise engage a user. In the case of advertising, it may be desirable to translate this change in emotional state into an outcome such as increased sales. For example, a television commercial may attempt to increase sales of a product to which it is associated.

[0007] The proliferation of web-enabled consumer devices means that it is becoming increasingly difficult for marketers to capture consumer attention. For advertising messages to have an impact on consumers, it is desirable for consumers to pay attention. The fact that consumers are easily distracted means that it is increasingly desirable to accurately track or measure parameters that indicate audience attention or engagement.

[0008] Many current metrics associated with content delivery do not indicate any user interaction. Such metrics may include number of impressions, number of views, view-through rate, etc. They do not indicate user attention, and in fact may not even require a human being to be present. Summary of the Invention

[0009] At its most general, the present invention proposes a system and method for rapidly and scalably tracking user presence on a user device. The term "user presence" is used herein to mean a human being at a device, i.e., in a position where they can interact with content displayed on the device. The term "presence" is intended to indicate only that a human is present and does not, in itself, distinguish between a present human being who is paying attention to displayed content and a present human being who is distracted. However, as explained below, the ability to track user presence may be linked to the ability to measure attention.

[0010] The system includes means for collecting relevant data streams from multiple user devices while content is displayed on the user devices or during some other type of interaction; means for analyzing the collected data by an AI-driven module that can output presence data, such as one or more metrics indicative of user presence; and means for synchronizing the collected data with the presence data.

[0011] The system may be implemented entirely on a user device, or may be distributed across multiple entities in a networked environment.

[0012] As described above, the user device may be a consumer electronic device such as a smartphone, tablet, laptop, or desktop computer. The system may be deployed within one or more applications running on the user device. For example, the system's functionality may be provided in a software development kit (SDK) for application developers to incorporate into their applications. Thus, the applications may have built-in functionality for tracking user presence during operation. In another embodiment, the system's functionality may be provided in a standalone module that can run in the background of the user device. Other applications may be configurable to invoke the module to provide the system's functionality to them. Alternatively or additionally, the user device may be capable of communicating (e.g., over a network) with a remote server configured to provide some or all of the system's functionality. Thus, the user device may be configured to transmit collected data to the remote server for processing. In some cases, the remote server may return result data to the user device.

[0013] The system may be configured to aggregate data to enable useful reports of the effectiveness of displayed content or any other interactions on user devices to be generated. In particular, the ability to synchronize presence metrics with other data streams can make accessible types of events associated with a user's presence and assist in understanding the level of exposure content has across cohorts of users. This information makes it possible to generate recommendations that allow the delivery of content to be targeted in places, optimizing its effectiveness. Data may be aggregated across multiple consumers (e.g., a set of users with a common demographic or interest), or across multiple pieces of content (e.g., different video ads with a common theme or from the same advertiser), or across a particular marketing campaign (e.g., data from a range of different ads linked to a common advertising campaign), or across a brand (e.g., data from all content that mentions or is otherwise linked to the brand).

[0014] The systems and methods of the present invention can find use in facilitating the optimization of advertising campaigns. The collected data allows for effective real-time monitoring of presence sharing for a given advertising campaign, or for brands that actually appear within a campaign. The systems and methods of the present invention can provide the ability to report on the reasons driving presence, which can then assist in determining what steps are required to optimize a distribution strategy to achieve campaign goals. Campaign goals may be set against parameters measurable by the system. For example, an advertising campaign may have the goal of maximizing total user presence time for a given budget. In another example, the campaign goal may be, for example, maximizing a particular type of presence from a particular demographic group or within a particular geographic area, or maximizing presence in the context of a particular positive sentiment. In another example, the campaign goal may be reaching a particular level of user presence for the lowest cost. As discussed in more detail below, the system can not only use the data to report on performance against campaign goals, but also make predictions regarding how certain additional actions will affect that performance. Thus, the system provides a tool for optimizing advertising campaigns through the provision of recommended actions supported by predicted effects on performance against campaign goals.

[0015] Additionally or alternatively, the systems and methods of the present invention can report on the emotional state associated with a user's presence, particularly in situations where an image of the user's face is available, which can provide feedback on whether an advertisement or brand is perceived positively or negatively.

[0016] According to the present invention, there is provided a computer-implemented method for collecting data from a user device, the method including: outputting information from the user device; collecting contextual attribute data indicative of events occurring at the user device within a period of the output of the information; collecting sensor data within the period of the output of the information by a sensor at the user device; applying the sensor data to a classification algorithm to generate presence data, the classification algorithm being a machine learning algorithm operable to map the sensor data to presence parameters, the presence data indicative of fluctuations in the presence parameters over time within a period of display of content; synchronizing the presence data with the contextual attribute data to generate a validity dataset linking changes in the presence parameters over time with corresponding contextual attribute data obtained within the period of the output of the information; and storing the validity dataset in a data store.

[0017] Data from a sensor or sensors on a user device may be referred to herein as “sensor data.” Sensor data may be image data (e.g., a single captured image or a video stream) and / or audio data. In one embodiment, a system is configured to obtain presence data from collected sensor data. As described below, presence data may be obtained from the sensor data in an automated manner, for example, by applying the sensor data to one or more classification algorithms trained to recognize features associated with a user's presence. Features may be visual features, e.g., body parts such as the face, torso, arms, hands, legs, etc. Features may also be auditory features, e.g., voice. When the user device is portable, features may be movement patterns associated with walking, running, etc. Data from multiple sensors may be used in combination to obtain presence data. Using multiple sensor types can increase confidence in the presence data because ambiguity in one type of sensor data can be resolved by other types of sensor data.

[0018] The output of information may relate to any kind of interaction with a user device for which it may be useful to have information about user presence, for example, the output information may be a notification of an incoming phone or video call.

[0019] The step of outputting information may include displaying the content on a user device, although references herein to "displaying the content" may equally apply to outputting other types of information.

[0020] In one embodiment, the content to be displayed may include media content. Alternatively or additionally, the content may include information displayed (e.g., in a graphical user interface) during operation of the user device. Thus, validity data may relate to the operation or use of applications or other software programs running on the user device and / or content displayed through such applications or software programs.

[0021] In one embodiment, the method may further include executing an application on the user device and playing media content with the application running on the user device, wherein the contextual attribute data further indicates events occurring in the application during the playing of the media content.

[0022] The media content may be retrieved by the user device from local storage (e.g., on the user device itself) or from elsewhere by the application, for example, over a local or wide area network connection. In one embodiment, the application may be or link to a content sharing platform.

[0023] The context attribute data may include control analysis data about the application.

[0024] In one embodiment, the system functionality may be provided by the application itself. For example, an application developer may incorporate a software development kit (SDK) configured to provide the functionality discussed herein. Thus, the application may be configured to generate presence data and synchronize the presence data with contextual attribute data.

[0025] In another example, an application may be configured to communicate with an analysis module running in the background of the user device. The analysis module may be configured to generate presence data and synchronize the presence data with contextual attribute data. The analysis module may communicate with multiple applications. This means that the user device can have a single entity that handles presence data generation for various other applications.

[0026] In further embodiments, system functionality may be provided in a standalone application configured to collect data for all interactions with a given device. In other words, data is collected regardless of the type of interaction or application currently being used or the type of source of any content being displayed. A user may decide to install this type of application to obtain information about how effectively they use the device. Thus, effectiveness data may relate to different types of uses a user of a device makes. Effectiveness data may include, for example, a "health" report summarizing the user's engagement and / or emotional response to those interactions with the device. As discussed below, similar functionality may be provided by a browser plug-in, which can provide system functionality for all interactions a user has with the browser, regardless of the identity of the website publisher or the identity of the source of the displayed content.

[0027] The efficacy data generated by the system may be displayed to the user. The efficacy data can allow users to track or view how they interact with the application. Such information may be of immediate local value to users, who can then encourage them to allow the efficacy data to be shared more widely.

[0028] The application may include an adapter module configured to communicate with an analytics server over a network, which may allow effectiveness data from multiple user devices (or indeed, any data collected by user devices if appropriate permissions to share are obtained) to be collected. The collected data may be aggregated or otherwise analyzed to find patterns, which may then be used to improve the content.

[0029] References herein to "sensor data" may refer to detectable information related to the environment of a user device. The sensor data may include, for example, one or more images of a location in front of the user device. For example, a sensor may include a camera, such as a webcam built into the user device or provided separately. If a user is present, the sensor data may include visual aspects of the user's responses. For example, the sensor data may include data indicative of any one or more of facial responses, head and body gestures or postures, and eye tracking.

[0030] The displayed content may be generated locally on the user device (e.g., by software running thereon). For example, the displayed content may relate to a locally running game, mobile application, or desktop application. As discussed above, in one embodiment, an application running on the user device may be provided with built-in capabilities for acquiring presence data. That is, the application may be configured to continuously collect sensor data using one or more sensors on the user device and can acquire presence data (and preferably, emotion and attention data as well) from the sensor data. The application may run a classification algorithm locally to acquire the presence data. This data may be communicated to an analytics server to obtain the above-mentioned validity data set. An advantage of integrated presence data generation within an application is that the user's interaction with the application itself can be used in either or both of the contextual attribute data and the sensor data. The generated presence data may be displayed directly on the user device, for example, through a user interface provided by the application, to show the presence data in relation to application activity. Preferably, such information from multiple users may be shared with application developers to provide a richer understanding of how users interact with the application, e.g., to gain insight into which application features are strongly linked to presence or which features are linked to loss of presence.

[0031] Additionally or alternatively, the displayed content may be obtained from the web, for example, by downloading, streaming, etc. Thus, displaying the content may include accessing, by the user device over the network, a web page on a web domain hosted by a content server, and receiving, by the user device over the network, the content to be displayed by the web page. The content may be displayed directly on the web page, either separately from the web page or embedded in the web page, or may be displayed via a media player application.

[0032] In this embodiment, the method may operate to collect two or more of the following types of data from the user device: (i) contextual attribute data from the web page, (ii) contextual attribute data from the media player application (if used), and (iii) sensor data. Presence data is extracted from the collected data, and all data is synchronized to allow the cause or driver of presence to be investigated.

[0033] Accessing the web page may include obtaining a context data initiation script for execution on the user device. The context data initiation script may be, for example, machine-readable code located in a tag in the header of the web page.

[0034] Alternatively, the context data initiation script may be provided within the communication framework once the content has been served to the user device. For example, if the content is a video advertisement, the communication framework typically accompanies an advertisement request from the user device and a video advertisement response sent from the advertisement server to the user device. The context data initiation script may be included in the video advertisement response. The video advertisement response may be formatted in accordance with the Video Ad Serving Template (VAST) specification (e.g., VAST 3.0 or VAST 4.0) or may comply with any other advertisement response standard, such as the Video Player Ad Interface Definition (VPAID), the Mobile Rich Media Ad Interface Definition (MRAID), etc.

[0035] In further alternatives, the context data initiation script may be inserted into web page source code at an intermediary between the publisher (i.e., the originator of the web page) and the user (i.e., the user device). The intermediary may be a proxy server or a code insertion component in a network router associated with the user. In these examples, the publisher does not need to incorporate the context data initiation script into its version of the web page. This means that the context data initiation script does not need to be transmitted in response to every web page hit. Furthermore, this technique can allow the script to be included only in requests from user devices associated with users who have granted permission for their behavioral data to be collected. In some examples, such users can form a panel to evaluate the effectiveness of web content before it is released to a wider audience.

[0036] In yet a further alternative, system functionality may be provided by a browser plug-in that a user can install on their device. In this example, data may be collected for all interactions with the browser, i.e., regardless of the web page visited. Users may benefit from this configuration because the collected data may allow effectiveness data for different web pages to be directly compared. Thus, users may obtain information indicating how they engage with or pay attention to different web pages.

[0037] The method may further include executing a context data initiation script at the user device to perform one or more preliminary operations before the content is displayed. The preliminary operations may include determining permission to transmit context attribute data and sensor data to a remote analytics server, determining availability of sensors for collecting sensor data, or verifying whether the user is selected for sensor data collection. The method may also include terminating the sensor data collection procedure when the user device, using the context data initiation script, determines that (i) permission to transmit sensor data has been revoked, or (ii) sensors for collecting sensor data are unavailable, or (iii) the user is not selected for sensor data collection. Determining any one of these criteria may terminate the sensor data collection procedure. In this case, the user device may send only context attribute data to the analytics server.

[0038] Image or video data may be transmitted, e.g., streamed or otherwise transmitted, from the user device using any suitable real-time communication protocol, e.g., WebRTC or the like. The method may include loading code for enabling the real-time communication protocol upon determining by the user device using the context data initiation script that (i) permission to transmit sensor data has been granted, (ii) sensors are available for collecting sensor data, and (iii) the user has been selected for sensor data collection. To avoid slowing down initial access to the web page, the code for enabling the real-time communication protocol may not be loaded until all of the above conditions are determined.

[0039] In addition to the data collected from the user device, the analytics server may obtain additional information about the user from other sources. The additional information may include data indicative of demographics, user preferences, user interests, etc. The additional data may be incorporated into the effectiveness data set, for example, as labels to allow the presence data to be filtered or sorted by demographics, user preferences, or interests, etc.

[0040] The additional data may be obtained in various ways. For example, the analytics server may communicate (directly or over a network) with an advertising system, such as a demand-side platform (DSP) for running programmatic ads. The additional information may be obtained from a user profile maintained by the DSP or directly from the user, for example, as feedback from a quiz or through social network interactions. The additional information may be obtained by analyzing images captured by a webcam on the user device.

[0041] The media content may be a video, such as a video advertisement. The synchronization of the presence data and the contextual attribute data may be with respect to a timeline during which the video was played in a media player application. The sensor data and the contextual attribute data may be time-stamped in a manner that allows a time relationship between the various data to be established.

[0042] The display of media content on a web page may be triggered by accessing the web page or by taking some predetermined action on the web page. The media content may be hosted on a web domain, e.g., embedded directly in the content of the web page. Alternatively, the media content may be obtained from a separate entity. For example, a content server may be a publisher that provides space within a web page to advertisers. The media content may be advertisements transmitted from an advertisement server (e.g., as a result of an advertisement bidding process) that fill space within the web page. The contextual attribution data may further indicate events that occur on the web page within the period of display of the content.

[0043] Thus, the media content may be outside the control of the content server. Similarly, the media player application on which the media content is played may not be software resident on the user device. Thus, contextual attribute data associated with a web page may need to be obtained independently of contextual attribute data associated with the media player application.

[0044] The classification algorithm may be located on the analytics server. Having a central location can expedite the process of updating the algorithm. However, the classification algorithm can also be on the user device, where instead of transmitting sensor data to the analytics server, the user device is configured to transmit presence data and emotion data. The advantage of providing the classification algorithm on the local device is that it increases privacy for the user, as sensor data does not need to be transmitted away from their computer. Running the classification algorithm locally also means that the analytics server requires much less processing power, saving costs.

[0045] As described above, collecting sensor data may include capturing images using a camera, and the context data initiation script may be configured to activate the camera.

[0046] The contextual attribute data may include web analytics data about the web page and control analytics data about the media player application. The analytics data may include any conventionally collected and communicated information about the web page and media player application, such as viewability of any elements, clickstream data, mouse movements (e.g., scrolling, cursor position), keystrokes, etc.

[0047] Execution of the context data initiation script may be configured to trigger or initialize collection of web analytics data. Analytics data from the media player application may be obtained using an adapter module, which may be part of the media player application software or a plug-in forming a separate loadable software adapter that communicates with the media player application software. The adapter module may be configured to transmit control analytics data about the media player application to an analytics server over a network, the method including executing the adapter module upon receiving media content to be displayed. The adapter module may be launched through execution of the context data initiation script or may be loaded into a plug.

[0048] The context data initiation script may be executed as part of running a web page, as part of running a mobile application for viewing content, or as part of running a media player application. The control analytics data and web analytics data may be transmitted from the entity on which the context data initiation script is running to an analytics server.

[0049] If the sensor data includes multiple images showing a user's response over time, the classification algorithm may operate to evaluate a presence parameter for each image in multiple images captured within the period of display of the content.

[0050] In addition to presence data, sensor data may be used to obtain emotional state information about the user when the user is determined to be present, particularly when the user's face is visible in the captured image. Thus, the method may further include applying the sensor data to an emotional state classification algorithm to generate emotional state data for the user, the emotional state classification algorithm being a machine learning algorithm operable to map the sensor data to emotional state data, the emotional state data being indicative of variations over time in the probability that the user will have a given emotional state within a period of display of the content, and synchronizing the emotional state data with the presence data, wherein the efficacy dataset further includes the emotional state data.

[0051] The user device may be configured to locally respond to the detected emotional state data and / or presence data, for example, content may be retrieved and displayed by an application running on the user device, and the application is configured to determine actions based on the emotional state data and presence data generated at the user device.

[0052] The functionality described herein may be implemented as a software development kit (SDK) for use in creating applications or other programs that can utilize the presence parameters or availability data described above. The software development kit may be configured to provide a classification algorithm.

[0053] The methods discussed herein are scalable to networked computing environments including multiple user devices, multiple content servers, and multiple different pieces or types of content. Thus, the method may include receiving, by an analytics server, contextual attribute data and sensor data from the multiple user devices. The analytics server may be operable to aggregate multiple validity data sets generated from the contextual attribute data and sensor data received from the multiple user devices, for example, according to the processes described above. The multiple validity data sets may be aggregated with respect to one or more common dimensions shared by the contextual attribute data and sensor data received from the multiple user devices, for example, for a given piece of media content, or for a group of related pieces of media content (e.g., associated with an advertising campaign), or by web domain, website identity, date and time, type of content, or any other suitable parameter.

[0054] The result of performing the method discussed above may be a data store having a rich efficacy dataset therein linking user presence with other observable factors. The efficacy dataset may be stored in a data structure, such as a database, and the data structure may be queried to generate reports that allow relationships between presence data and other data to be observed. Thus, the method may further include receiving, by a reporting device over a network, a query for information from the efficacy dataset; extracting, by the reporting device from the data store, response data in response to the query; and transmitting, by the reporting device, the response data over the network. The query may be from a brand owner or a publisher.

[0055] The aggregated data may be used to update functionality at the user device. For example, if the content is retrieved and displayed by an application running on the user device, the method may further include using the aggregated validity data set to determine a software update for the application, receiving the software update at the user device, and adjusting the functionality of the application by performing the software update.

[0056] In another aspect, the present invention may provide a system for collecting data from a user device within a period of output of information from the user device, the system being configured to: collect from the user device contextual attribute data indicative of events occurring at the user device within the period of output of information; collect sensor data from one or more sensors on the user device within the period of output of information; apply the received sensor data to a classification algorithm to generate presence data, the classification algorithm being a machine learning algorithm operable to map the sensor data to presence parameters, the presence data indicative of fluctuations in the presence parameters over time within the period of output of information; synchronize the presence data with the contextual attribute data to generate a validity data set linking changes in the presence parameters over time with corresponding contextual attribute data obtained within the period of output of information; and store the validity data set in a data store. Features of the method discussed above may be equally applicable to the system.

[0057] As described above, the effectiveness data generated by the system may be used to make predictions regarding how certain additional actions will affect the performance of a given piece of content or a given advertising campaign. In another aspect of the invention, a method is provided for optimizing an advertising campaign in which recommended actions, supported by their predicted effect on performance against campaign goals, are used to adjust a programmatic advertising strategy.

[0058] According to this aspect, a computer-implemented method for optimizing a digital advertising campaign is provided, the method including: accessing an effectiveness dataset representing changes in presence parameters over time during a period of playing a portion of advertising content belonging to the digital advertising campaign to a plurality of users, the presence parameters being obtained by applying sensor data collected from each user during the period of playing the portion of advertising content to a machine learning algorithm operable to map the sensor data to the presence parameters; generating candidate adjustments to a target audience strategy associated with the digital advertising campaign; predicting an effect on the presence parameters of applying the candidate adjustments; evaluating the predicted effect against campaign objectives for the digital advertising campaign; and updating the target audience strategy with the candidate adjustments if the predicted effect improves performance against the campaign objectives by more than a threshold amount. The updating may be performed automatically, i.e., without human intervention. Thus, the target audience strategy may be automatically optimized.

[0059] The efficacy dataset may be obtained using the methods discussed above and may therefore have any of the characteristics described herein. For example, the efficacy dataset may further include user profile information indicative of the user's demographics and interests. In such examples, potential adjustments to the target audience strategy may change the target audience's demographic or interest information.

[0060] Indeed, the method may generate and evaluate multiple candidate adjustments. The method may automatically implement all adjustments that lead to an improvement of more than a threshold amount. Alternatively or additionally, the method may include presenting (e.g., displaying) all or a subset of the adjustments that lead to an improvement of more than a threshold amount. The method may also include selecting, e.g., manually or automatically, one or more of the adjustments to be used to update the target audience strategy.

[0061] The step of automatically updating the target audience strategy may include communicating the revised target audience strategy to a demand-side platform (DSP). Thus, a method according to this aspect may be performed in a network environment, for example, including a DSP, the above-discussed analytics server, and a campaign management server. The DSP may operate in a conventional manner based on instructions from the campaign management server. The analytics server may have access to an effectiveness dataset and may be the entity that runs campaign goal optimization based on information from the campaign management server. Alternatively, the campaign goal optimization may run on the campaign management server, which may be configured to send queries to the analytics server, for example, to obtain and / or evaluate the predicted effects of candidate adjustments to the target audience strategy.

[0062] Embodiments of the present invention will be discussed in detail below with reference to the accompanying drawings. [Brief explanation of the drawings]

[0063] [Figure 1] 1 is a schematic diagram of a data collection and analysis system according to an embodiment of the present invention; [Figure 2] 1 is a flowchart of a method for collecting and analyzing data according to an embodiment of the present invention. [Figure 3] FIG. 1 is a schematic diagram of a data collection and analysis system for generating a presence classifier suitable for use in the present invention. [Figure 4-1] 3 is a screenshot of a reporting dashboard presenting data resulting from execution of the method of FIG. 2. [Figure 4-2] 3 is a screenshot of a reporting dashboard presenting data resulting from execution of the method of FIG. 2. [Figure 5] 4 is a flowchart of an advertising campaign optimization method according to another aspect of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0064] Embodiments of the present invention relate to systems and methods for collecting and utilizing data from a user device while the user device is displaying web-based content. In the examples below, the displayed content is media content, e.g., video or audio. However, it will be understood that the present invention is applicable to any type of content that can be presented by a user device.

[0065] In this example, the system is configured to determine whether a user is present on a user device within a period of playback of media content. The determination may be made using data obtained from one or more sensors on the user device, such as any one or more of a camera (e.g., a webcam), a microphone, or a motion sensor (e.g., a gyroscope). The determination may be a binary decision, e.g., “user present” or “user not present,” or it may be a selection of multiple discrete states, e.g., “user present and face visible,” “user present and face not visible,” “user not present,” etc. Alternatively or additionally, the determination may involve obtaining a probability that the user is present or not present. The result of the determination may be referred to herein as “presence data.” The presence data may be characterized by “presence parameters” that indicate whether the user is present or not. Data from a sensor or sensors on the user device may be referred to herein as “sensor data.” The sensor data may be image data (e.g., a single captured image or a video stream) and / or audio data. In one embodiment, the system is configured to obtain presence data from collected sensor data. As described below, the presence data may be obtained from the sensor data in an automated manner, for example, by applying the sensor data to one or more classification algorithms trained to recognize features associated with a user's presence. The features may be visual features, e.g., body parts such as the face, torso, arms, hands, legs, etc. The features may also be auditory features, e.g., voice. When the user device is portable, the features may also be movement patterns associated with, for example, walking, running, etc.

[0066] Presence data may be an output of the system. Presence data may be associated with, for example, synchronized with, the media content that was playing when the presence data was collected. Presence data alone may be a useful parameter for assessing the usefulness or effectiveness of media content.

[0067] In other examples, if the presence data indicates that the user is present, further data may be collected, or further processing of already collected sensor data may be performed, to assess the impact or effect of the media content on the user. In one example, the system may suppress further data collection if the presence data indicates that the user is not present. This can inhibit the collection, processing, and possibly transmission of unnecessary data. In another example, if the collected sensor data includes image data and the presence data indicates that the user is present and a face is visible, the system may analyze the image data to determine the user's emotional state or to determine whether the user is attentive.

[0068] 1 is a schematic diagram of a data collection and analysis system 100 embodying the present invention. In the following discussion, the system is described in the context of evaluating media content 104 in the form of video advertisements, for example, that may be produced by a brand owner 102. However, it can be appreciated that the systems and methods of the present invention are applicable to any type of media content for which it is desirable to monitor its impact on users on a large scale. For example, the media content may be training or safety videos, online learning materials, movies, music videos, etc.

[0069] System 100 is provided in a networked computing environment in which several processing entities are communicatively connected through one or more networks. In this example, system 100 includes one or more user devices 106 configured to play media content, for example, via speakers or headphones and a software-based video player 107 on a display 108. User device 106 may also include or be connected to one or more sensors, such as a webcam 110, a microphone, etc. Examples of user devices 106 in the example include smartphones, tablet computers, laptop computers, desktop computers, etc.

[0070] User devices 106 are communicatively connected through a network 112 so that they can receive sourced content 115 to be consumed, for example, from a content server 114 (e.g., a web host), which can operate under the control of a publisher to distribute content on one or more channels or platforms. Publishers can sell "space" on their channels to brand owners to display video advertisements, either through an advertising bidding process or by embedding advertisements in the content.

[0071] Thus, the served content 115 may include media content 104 provided directly by the content server 114 or transmitted together with or separately from the served content by the ad server 116, for example, as a result of an advertising bidding process. The brand owner 102 may serve the media content 104 to the content server 114 and / or ad server 116 in any conventional manner. The network 112 may be any type of network.

[0072] In this example, the served content includes code for triggering the transmission of contextual attribute data 124 from the user device 106 to the analytics server 130 over the network 112. The code is preferably in the form of a tag 120 in the header of a main page loaded from a domain hosted by the content server 114. The tag 120 operates to load a bootstrapping script, which performs several functions that enable the delivery of information, including the contextual attribute data 124, from the user device 106. These functions are discussed in more detail below. However, in this example, the primary function of the tag 120 is to trigger the delivery of the contextual attribute data 124 to the analytics server 130 and, if appropriate, the delivery of a sensor data stream 122, such as a webcam recording containing video or image data from a camera 110 on the user device 106, to the analytics server 130.

[0073] The contextual attribute data 124 is preferably analytics data related to events that occur on the user device after the main page loads. The analytics data may include any conventionally collected and communicated information about the main page, such as viewability of any elements, clicks, scrolls, etc. This analytics data can provide a control baseline against which other metrics, such as presence metrics discussed below, are measured when the associated media content 104 is being viewed or played.

[0074] As mentioned above, the sensor data stream 122 transmitted to the analytics server 130 may include a video or set of images captured during the playback of the media content 104 .

[0075] In addition to the sensor data 122 and the contextual attribute data 124, the analytics server 130 is configured to receive the media content 104 itself and an auxiliary contextual attribute data stream 126, which includes analytics data from the video player on which the media content is displayed. The media content 104 may be provided to the analytics server 130 by the brand owner 102, or directly from the content server 114 or the user device 106. The auxiliary contextual attribute data stream 126 may be obtained by loading an adapter for the video player 107 on which the media content 104 is displayed. Alternatively, the video player 107 may have a plug-in to provide the same functionality within the native environment of the video player 107.

[0076] The auxiliary contextual attribute data stream 126 is acquired for the purpose of synchronizing the sensor data 122 with the playback position within the media content, thus providing brand measurement and creative level analysis. The auxiliary contextual attribute data stream 126 may include viewability, playback event, click, and scroll data associated with the video player.

[0077] In particular, when rendering the media content 104 is done via a third-party ad server 116, the video player 107 can be deployed within an iframe, thus providing a separate mechanism for generating the auxiliary contextual attribute data stream 126. In such a case, an adapter should be deployed inside the iframe, and the adapter can coordinate with the functionality of the main tag 120 to record and transmit data to the analytics server 130.

[0078] For example, the auxiliary contextual attribute data stream 126 may include information related to user commands such as pause / resume, stop, volume control, etc. Additionally or alternatively, the auxiliary contextual attribute data stream 126 may include other information regarding delays or disruptions in playback, for example, due to buffering or the like.

[0079] In combination, the context attribute data stream 124 and the auxiliary context attribute data stream 126 provide the analysis server 130 with rich background context that can be related to (and, in fact, synchronized with) the user's response to portions of the media content available from the sensor data stream 122.

[0080] The sensor data stream 122 may not be acquired from every user device on which the media content 104 is played because permission to share the information has not been obtained or because the appropriate sensors are not available. If permission to share the information is granted but sensor data has not been acquired, the main tag 120 may nevertheless transmit the context attribute information 124, 126 to the analytics server 130.

[0081] The bootstrapping script may operate to determine whether a sensor data stream 122 is to be acquired from a given user device, which may involve checking whether the user has opted in to participate, for example, based on a random sampling approach and / or based on publisher restrictions (e.g., because feedback from only some particular class of audience is required).

[0082] The bootstrapping script may first operate to determine or obtain permissions to share the context attribute data 124 and auxiliary context attribute data 126 with the analytics server 130. For example, if a permission management platform (CMP) exists for the target domain, the script operates to check for permissions from the CMP. The script may also operate to check for a global opt-out cookie associated with the analytics server or a particular domain.

[0083] The bootstrapping script may operate to check whether the sensor data stream 122 is to be acquired. If so (e.g., because the user was selected as part of the sample), the bootstrapping script may check the camera's 110 permission API to record and transmit the camera feed. Because the sensor data stream 122 is transmitted along with contextual attribute data from the primary domain page, it is important that the tag for running the bootstrapping script is in the header of the primary domain page and not in any associated iframes.

[0084] In one embodiment, the sensor data stream 122 is a recording of the full video from the camera 110 transmitted to the analytics server 130 over a suitable real-time communication protocol, such as WebRTC. To optimize page load speed, the code for WebRTC recording and on-device tracking is not loaded by a bootstrapping script before the relevant permissions are confirmed. In an alternative approach, the camera feed may be processed locally by the user device, resulting in the transmission of detected presence metrics (attention, emotion, and other signals, if appropriate) so that no images or video remain on the user device. In this approach, some functionality of the analytics server 130, discussed below, is distributed to the user device 110.

[0085] Generally, the function of the analytics server 130 is to transform the essentially free-form viewing data obtained from the user devices 106 into a rich data set that can be used to determine the effectiveness of media content. As an initial step, the analytics server 130 operates to determine presence data for each user. The presence data may be obtained from the sensor data stream 122 by using a presence classifier 132, which in this example is an AI-based model that returns the probability that a user is located within the camera's field of view. The presence classifier 132 may be configured to flag whether a user's face is visible in a given webcam frame, which can trigger further processing to determine whether the user is paying attention to the on-screen content.

[0086] The presence classifier 132 may output a time-varying signal indicative of changes in user presence during the playback of the media content 104. This may be synchronized with the media content 104 itself to allow the detected presence (and any associated attentive or distracted states) to be matched with the playback of the media content. For example, if the media content is a video advertisement, a brand may appear at specific time points or periods within the video. The present invention allows those time points or periods to be marked or labeled with presence and / or attention information.

[0087] Similarly, the creative content of a video may be represented as a stream of keywords associated with different time points or periods within the video. Synchronization of the keyword stream with presence metrics can allow correlations between keywords and presence (and corresponding attention or distraction) to be recognized.

[0088] Presence signals may also be synchronized with contextual attribute signals, thereby providing a rich data set of contextual data synchronized with user presence changes. These data sets, which may be obtained from each user consuming media content, are aggregated and stored in data store 136, from which they may be queried and further analyzed to generate reports, identify correlations, and make recommendations, as discussed below.

[0089] The contextual attribute data 124 may also be used to provide a confidence or trust to be applied to content to which the output from the presence classifier 132 relates, for example, by allowing a cross-check of what is visible on the screen or by interaction with the user device. For example, if the contextual attribute data 124 indicates that an input command is being received at the user device, then confidence in the presence data may be lost.

[0090] In situations where presence metrics indicate that a user is present, particularly in scenarios where the user's face is visible, the sensor data stream 122 may also be input to the attention classifier 134, which operates to generate a time-varying signal indicative of the user's attentiveness when consuming media content.

[0091] The sensor data stream 122 may also be input to an emotional state classifier 135, which operates to generate a time-varying signal indicative of the user's emotions when consuming media content. This emotional state signal may thus also be synchronized with an attention signal, which also allows emotions associated with attention (or distraction) to be assessed and reported.

[0092] In addition to generating the rich datasets discussed above, the analytics server 130 may be configured to determine specific presence metrics for a given portion of media content. One example of a presence metric may be presence volume, which may be defined as the average volume of presence detected during the duration of the media content's playback. For example, a presence volume score of 50% means that the viewer was present for half of the content, on average, throughout the entire video. The more seconds of presence the video managed to attract attention, the higher the score. Another example of a presence metric is presence quality, which may be defined as the proportion of media content to which a respondent was continuously present, on average. For example, a score of 50% means that a respondent was present for half of the video, on average, without interruption. This metric differs from presence volume because it is not the overall amount of presence that determines the score value, but rather how the presence was distributed along the view. Presence quality decreases as the respondent moves in and out of the camera's field of view, which may indicate that the respondent is periodically distracted.

[0093] The above metrics, or others, may determine the amount of contact between a user and a played instance of media content on a user device. From a brand owner or publisher's perspective, the benefit of this feature is that it not only allows them to report on the number of impressions and views of a particular piece of media content, but also allows them to distinguish between views where the user is present and views where the user is not present. If a user is present, further analysis may be performed to assess their attention and / or emotional state. The accompanying contextual attribute data then makes it possible to attempt to understand the levers that drive attention or distraction.

[0094] The system includes a report generator 138 configured to query the data store 136 to generate one or more reports 140 that can be provided to the brand owner 102, for example, directly or over the network 112. The report generator 138 may be a conventional computing device or server configured to query a database on the data store containing the collected and synchronized data. Examples of reports 140 are discussed in more detail below with reference to FIG. 4.

[0095] FIG. 2 is a flow chart illustrating steps taken by a user device 106 and an analytics server 130 in a method 200 that is an embodiment of the present invention.

[0096] The method begins by step 202 requesting and receiving web content by a user device over a network, where web content is intended to mean, for example, web pages that can be accessed and loaded from domains hosted by content server 114 as discussed above.

[0097] The web page includes in its header a tag containing a bootstrapping script configured to run several preliminary checks and processes that enable the collection of data from the user device. The method then continues with step 204 of running the bootstrapping script. One of the tasks performed by the script is to check for or obtain permission to synchronize the collected data with an analytics server. This may be done with reference to a content management platform (CMP), if applicable to the domain from which the web page is retrieved. In this case, the bootstrapping script is located after the code in the web page header that initializes the CMP.

[0098] The method continues with step 206 of checking or obtaining permission to share the data. This may be done in any conventional manner, for example, by checking the current status of the CMP or by providing an on-screen prompt. Permission is preferably requested at the domain level, so as to avoid repeated requests, for example, when accessing additional pages from the same domain. The method includes step 208 of checking camera availability and obtaining permission for data collected from the camera to be transmitted to the analytics server.

[0099] If a camera is available and permission to transmit data from the camera is granted, the method continues with step 210 of checking whether a user has been selected or sampled for sensor data collection. In other embodiments, this step 210 may occur before step 208 of checking camera availability.

[0100] In some situations, all users with available cameras may be selected. However, in other examples, users may be selected to either ensure an appropriate (e.g., random or pseudo-random) range of data is received by the analytics server 130 or to meet requirements set by a brand owner or publisher (e.g., to collect data only from one population sector). In another example, the ability to select users may be used to control the rate at which data is received by the analytics server. This may be useful when there are issues with or limitations on network bandwidth.

[0101] When the user provides permission to transmit sensor data from the camera and selects to transmit the sensor data, the method continues at step 212 with loading appropriate code that permits sharing of the camera data through a web page. In one embodiment, transmitting the behavioral data is performed using the WebRTC protocol. It is preferable to defer loading the code for transmitting the sensor data until after it has been determined that the sensor data will actually be transmitted. Doing so conserves network resources (i.e., unnecessary traffic) and facilitates rapid initial page load.

[0102] Sometime after accessing the web page and running the bootstrapping script, the method continues with step 214 of launching media content on the user device. Launching media content can mean initiating playback of media embedded in the web page, or encountering an advertising space on the web page that causes playback of a video advertisement received from an advertising server, for example, resulting from a traditional advertising bidding process.

[0103] Playback of the media content may be performed by executing a media player, e.g., a video player or the like. The media player may be embedded in a web page or configured to display the media content in an iframe within the web page. Examples of suitable media players include Windows® Media Player, QuickTime® Player, Audacious, Amarok, Banshee, MPlayer, Rhythmbox, SMPlayer, Totem, VLC®, and xine, or online video players such as JW Player, Flowplayer, VideoJS, and Brightcove®.

[0104] As discussed above, it may be desirable to transmit contextual attribute data relating to the behavior and control of the media player, i.e., analytical data about the media player, to an analytics server. To accomplish this, the method continues at step 216 with loading an adapter for the media player (or executing a media player plug-in, if present) that is configured to communicate the media player analytical data to a web page, which can then be transmitted to the analytics server.

[0105] The method continues with transmitting context attribute data 218 and, if applicable, transmitting sensor data to the analytics server 220. If a camera is available and permission is granted, this means that the data transmitted to the analytics server comes from three sources: (1) Sensor data from the camera—this is typically images or video from the camera itself. However, as discussed above, it is also possible for the user device itself to perform some preliminary analysis on the raw image data, for example, to measure presence and / or identify attention or emotion. In this embodiment, the sensor data transmitted to the analysis server may already be presence data, attention data, and emotional state data; image data need not be transmitted. (2) Contextual data from the web page - this is typically analytical data associated with the domain from which the web page is accessed. (3) Contextual Data from Media Players—This is typically analytical data associated with the media player on which the media content is displayed.

[0106] The method now moves to actions taken at the analytics server, which begin at step 222 of receiving the data discussed above from the user device. The method also includes step 224 of obtaining, by the analytics server, media content that is the subject of the collected sensor data and contextual attribute data. The analytics server may obtain the media content directly from a brand owner or from a content server, for example, based on an identifier transmitted by the user device. Alternatively, the analytics server may have a local store of media content.

[0107] The method continues with step 226 of classifying the sensor data for presence. In this step, each image from the data captured by the camera on the user device is fed to a presence classifier, which evaluates the probability that the user is present in the image. The output of the presence classifier may thus be a presence profile for the user relative to the media content, where the presence profile indicates changes in presence over time within the duration of the media content. In another embodiment, the classifier may be dichotomous, producing a frame-by-frame output that is either "present" or "not present." A presence profile may also be generated for such a two-state solution. In another embodiment, the classifier may be trained to include labels for the input data to qualify the presence signal. For example, the classifier may be able to distinguish between a state in which the user is present but the user's face cannot be read sufficiently to determine whether they are attentive, and a state in which the user is present and the face is visible, suitable for further analysis. Thus, the classifier may output labels such as "present and face visible," "present and face not visible," and "not present."

[0108] The presence classifier or analytics server may also be configured to generate one or more presence metrics for that particular viewing instance of the media content, which may be or may include the presence volume and presence quality metrics discussed above.

[0109] The method continues with step 228 of extracting attention or emotional state information from the sensor data stream. This may be done by an attention classifier and an emotional state classifier and may be performed in parallel with step 226. The output of this step may be an attention profile or an emotional state profile indicating changes in attention and / or emotional state over time within the duration of the media content.

[0110] As discussed above, the sensor data stream may include image data captured by a camera, the image data being multiple image frames representing a user's facial image. The image frames represent the user's facial features, such as the mouth, eyes, eyebrows, etc. The facial features result in descriptor data points indicating the position, shape, orientation, and / or shape of selected facial landmarks. Each facial feature descriptor data point may encode information indicative of the multiple facial landmarks. Each facial feature descriptor data point may be associated with a respective frame, e.g., a respective image frame from a time series of image frames. Each facial feature descriptor data point may be a multi-dimensional data point, with each element of the multi-dimensional data point representing a respective facial landmark.

[0111] The emotional state information may be obtained directly from the raw sensor data input, from descriptor data points extracted from image data, or from a combination of the two. For example, a plurality of facial landmarks may be selected to contain information capable of characterizing a user's emotion. In one embodiment, the emotional state data may be determined by applying a classifier to one or more facial feature descriptor data points within an image or across a series of images. In some embodiments, deep learning techniques may be utilized to derive the emotional state data from the raw data input.

[0112] The user emotional state may include one or more emotional states selected from anger, disgust, fear, happiness, sadness, and surprise.

[0113] The method continues with step 232 of synchronizing the presence profile 232 with corresponding context attribute data and emotional state data to generate a rich "availability" data set in which the context of periods of presence and absence in the presence profile are associated with various elements of the relevant context.

[0114] The method continues with step 234 of aggregating effectiveness data sets obtained for multiple viewed instances of the media content from multiple user devices (e.g., different users). The aggregated data is stored in a data store from which it may be queried to generate reports of the type discussed below with reference to FIG.

[0115] Figure 3 is a schematic diagram of a data collection and analysis system 300 for generating a presence classifier suitable for use in the present invention. It can be seen that the system in Figure 3 illustrates components for performing the collection and annotation of data and the subsequent use of that data in generating and utilizing a presence classifier.

[0116] System 300 is provided in a networked computing environment in which several processing entities are communicatively connected through one or more networks. In this example, system 300 includes one or more user devices 302 configured to play media content, for example, via speakers or headphones and a display 304. User device 302 may also include or be connected to one or more sensor components, such as a webcam 306, a microphone, etc. Example user devices 302 include smartphones, tablet computers, laptop computers, desktop computers, etc.

[0117] The user devices 302 are communicatively connected through a network 308 so that they can receive media content 312 to be consumed, for example, from a content provider server 310 .

[0118] The user device 302 may further be configured to transmit the collected sensor information over a network for analysis or further processing at a remote device, such as an analytics server 318 .

[0119] In this example, the information sent to the analytics server 318 may include a set of videos or images captured during playback of the media content. The information may also include links or other identifiers that enable the analytics server 318 to access the related media content 315 or the media content 312 consumed by the user. The related media content 315 may include information related to the manner in which the media content was played on the user device 302. For example, the related media content 315 may include information related to user commands such as pause / resume, stop, volume control, etc. Additionally or alternatively, the related media content 315 may include other information regarding delays or disruptions in playback, for example, due to buffering or the like. This information may correspond to (and may be obtained similarly to) the analytics data from the media player discussed above. Thus, the analytics server 318 may receive a data stream that includes information related to the playback of portions of media content on the user device.

[0120] In this example, the purpose of collecting sensor information is to annotate it with presence labels.

[0121] The system 300 provides an annotation tool 320 that facilitates the execution of the annotation process. The annotation tool 320 may include a computer terminal in communication (e.g., networked communication) with the analysis server 318. The annotation tool 320 includes a display 322 for presenting a graphical user interface to a human annotator (not shown). The graphical user interface may take many forms. However, it may significantly include several functional elements. First, the graphical user interface may present the associated media content 315 and the simultaneously collected sensor data 316 in a synchronized manner.

[0122] The graphical user interface may include a controller 324 for controlling the playback of the synchronized response data 316 and associated media content. For example, the controller 324 may allow the annotator to play, pause, rewind, fast forward, stop, scroll backward, scroll forward, etc. through the displayed material.

[0123] The graphical user interface may include one or more score applicators 326 for applying a presence score to a portion or portions of the response data 316. In one embodiment, the score applicator 326 may be used to apply a presence score to a period of a set of video or image frames corresponding to a given time period of collected sensor data. The presence score may have any suitable format. In one embodiment, it is dichotomous, i.e., a simple yes / no indication of presence. In other embodiments, the presence score may be selected from a set number of predefined levels or may be drawn from a numerical range (e.g., a linear scale) between end limits representing absence and presence, respectively, along with a clearly visible face.

[0124] Simplifying annotation tools can be desirable with respect to expanding the pool of potential annotators. The simpler the annotation process, the less training annotators will need to participate. In one embodiment, the annotated data may be harvested using a crowdsourcing approach.

[0125] Thus, the annotation tool 320 may receive time-series data indicative of a user's presence as well as represent a device for consuming portions of a media contact. The attention data may be synchronized with the response data 316 (e.g., via a scoring scheme). The analytics server 318 may be configured to collate or otherwise combine the received data to generate presence-labeled sensor data 330, which may be stored in suitable storage 328.

[0126] Presence data from multiple annotators may be aggregated or otherwise combined to obtain a presence score for a given response, for example, presence data from multiple annotators may be averaged over a portion of media content.

[0127] The analytics server 318 may be configured to receive presence data from multiple annotators. The analytics server 318 may generate combined presence data from different sets of presence data. The combined presence data may include a presence parameter indicative of a level of positive correlation between the presence data from the multiple annotators. In other words, the analytics server 318 may output a score that quantifies a level of agreement between dichotomous choices made by the multiple annotators across the response data. The presence parameter may be a time-varying parameter, i.e., the score indicating agreement may change over the period of the response data to indicate increasing or decreasing correlation.

[0128] In a development of this concept, the analytics server 318 may be configured to determine and store a confidence value associated with each annotator. The confidence value may be calculated based on how well the annotator individually scores correlation with the combined presence data. For example, an annotator who regularly scores in the opposite direction to the group of annotators as a whole may be assigned a lower confidence value than annotators who are more often in agreement. For example, the confidence value may be dynamically updated as more data is received from each individual annotator. The confidence value may be used to weight the presence data from each annotator in the process of generating the combined presence data. Thus, the analytics server 318 may exhibit the ability to "tune" itself for more accurate scoring.

[0129] The presence-labeled sensor data 330 may include presence parameters. In other words, the presence parameters may be associated with events in the data stream or media content, for example, synchronized or otherwise mapped to the events or linked to the events.

[0130] The presence-labeled sensor data 330 may include any one or more of the original collected data 316 from the user device 302 (e.g., raw video or image data, also referred to herein as response data), time-series presence data, time-series data corresponding to one or more physiological parameters from the physiological data 314, and emotional state data extracted from the collected data 316.

[0131] The collected data may be image data captured at each of the user devices 302. The image data may include multiple image frames showing a facial image of the user. Additionally, the image data may include a time series of image frames showing a facial image of the user.

[0132] The image frames may represent the user's facial features, such as the mouth, eyes, eyebrows, etc., each of which may include multiple facial landmarks, and the behavioral data may include information indicating the position, shape, orientation, shading, etc. of the facial landmarks for each image frame.

[0133] The image data may be processed on each user device 302 or may be streamed over the network 308 to an analysis server 318 for processing.

[0134] The facial features may result in descriptor data points that indicate the position, shape, orientation, sharing, etc., of a selected number of facial landmarks. Each facial feature descriptor data point may encode information indicative of a number of facial landmarks. Each facial feature descriptor data point may be associated with a respective frame, e.g., a respective image frame from a time series of image frames. Each facial feature descriptor data point may be a multi-dimensional data point, with each element of the multi-dimensional data point indicating a respective facial landmark.

[0135] The emotional state information may be obtained directly from the raw sensor data input, from extracted descriptor data points, or from a combination of the two. For example, a plurality of facial landmarks may be selected to contain information capable of characterizing a user's emotion. In one embodiment, the emotional state data may be determined by applying a classifier to one or more facial feature descriptor data points within an image or across a sequence of images. In some embodiments, deep learning techniques may be utilized to derive the emotional state data from the raw data input.

[0136] The user emotional state may include one or more emotional states selected from anger, disgust, fear, happiness, sadness, and surprise.

[0137] The creation of presence-labeled sensor data represents the first function of the system 300. The second function, described below, is the subsequent use of that data to generate and utilize presence models for the presence classifier 132 discussed above.

[0138] System 300 may include a modeling server 332 configured to communicate with storage 328 and access presence-labeling sensor data 330. Modeling server 332 may be directly connected to storage 328 or may be connected to storage 328 over a network, such as network 308 as shown in FIG.

[0139] The modeling server 332 is configured to apply machine learning techniques 334 to a training set of presence-labeled sensor data 330 to establish a model 336 for scoring presence from unlabeled response data, e.g., sensor data 316 as initially received by the analytics server 318. The model may be established as an artificial neural network trained to recognize patterns in the collected response data that indicate a high level of presence. Thus, the model may be used to automatically score the collected response data for presence without human input. An advantage of this technique is that the model is fundamentally based on direct measurements of presence that are sensitive to contextual factors that may be lost by measuring presence or engaging in a way that relies on specific predetermined proxies.

[0140] In one embodiment, the presence-labeling sensor data 330 used to generate the presence model 336 may also include information about the media content. This information may relate to how the media content is manipulated by the user, e.g., interrupted or otherwise controlled. Additionally or alternatively, the information may include data about the subject matter of the media content on display, e.g., to provide context to the collected response data.

[0141] As used herein, a portion of media content may be any type of user-consumable content for which information regarding user feedback is desirable. The present invention may be particularly useful when the media content is commercial (e.g., a video commercial or advertisement) and user presence is closely linked to an outcome, such as increased sales, or the like. However, the present invention is applicable to any kind of content, such as video commercials, audio commercials, movie trailers, movies, web advertisements, animated games, images, etc.

[0142] Figure 4 is a screenshot of a reporting dashboard 400 including a presentation of rich effectiveness data stored in the data store 136 of Figure 1 for a group of advertisements across different media content areas, e.g., a common field. The common field may be indicated by a main heading 401, shown in Figure 4 as "Sports Apparel," but may be changed by a user, for example, by selecting from a drop-down list.

[0143] Dashboard 400 includes an impression categorization bar 402 that shows the relative proportion of all served impressions that are (i) viewable (i.e., visible on the screen) and (ii) viewable with the user present, i.e., have a presence score above a predetermined threshold. Norms may be marked on the bar to show how the viewability and presence proportions compare to predicted outcomes.

[0144] The dashboard 400 may further include a relative emotional state bar 404 that indicates the relative strength of the emotional state detected from the audience present, from which information is available.

[0145] Dashboard 400 may further include a driver indicator bar 406, which in this example indicates the relative amount that different context attribute categories correlate to the presence detected thereby. Each of the context attribute categories (e.g., creative, brand, audience, and context) may be selectable to provide a more detailed breakdown of the factors contributing to that category. For example, the "creative" category may relate to information presented in the media content. The context attribute data may include a content stream describing the main items that are visible at any point in time in the media content. In FIG. 4, driver indicator bar 406 indicates the correlation between categories and presence. However, it may be possible to select other features, such as particular emotional states, whose relative strength of correlation with categories is of interest.

[0146] Dashboard 400 further includes a brand presence chart 408 that shows changes over time in the level of exposure (i.e., visibility to an existing audience) achieved by various brands in the common areas indicated in main heading 401.

[0147] Dashboard 400 further includes a series of charts that categorize impression categorizations by contextual attribute data, for example, chart 410 categorizes impression categorizations by viewing device type, and chart 412 categorizes impression categorizations using gender and age information.

[0148] The dashboard 400 further includes a map 414 in which the relative presence is illustrated using location information from the contextual attribute data.

[0149] Dashboard 400 further includes a domain comparison chart 416 that compares the amount of presence associated with the web domains from which impressions were obtained.

[0150] Finally, dashboard 400 further includes a summary panel 418 that categorizes the campaigns covered by common areas according to a predetermined presence threshold, which in this example is 10%, meaning that 10% of the impressions are detected as having a present audience.

[0151] The presence data collected by the disclosed system may be used to control programmatic advertising campaigns. Control may be performed manually, for example, by having a DSP adapt instructions based on recommendations provided in the reports. However, it may be particularly useful to implement automated adjustments to programmatic advertising instructions to effectively establish an automated feedback loop that optimizes programmatic advertising strategies to meet campaign goals.

[0152] The term "programmatic advertising" is used herein to refer to an automated process for purchasing digital advertising space, for example, on a web page, on an online media player, on a content sharing platform, etc. Typically, the process involves bidding on each ad slot (i.e., each available ad impression) in real time. In programmatic advertising, a DSP operates to automatically select a bid in response to an available ad impression. The bid is selected based in part on a determined level of correspondence between a campaign strategy provided to the DSP by the advertiser and contextual information about the ad impression itself. The campaign strategy identifies a target audience, and the bid selection process operates to maximize the likelihood of the ad being delivered to a portion of that target audience.

[0153] In this context, the present invention may be used as a means to adjust the campaign strategies provided to the DSP in real time, and preferably in an automated manner. In other words, the recommendations output from the analytics server may be used to adjust the target audience definition for a given advertising campaign.

[0154] In one embodiment, the system discussed above with respect to FIG. 1 may be used to provide information about presence in connection with a software platform or application from which various content can be consumed. The platform may be a content sharing platform or application such as YouTube®, Facebook®, Vimeo®, or TikTok®. Thus, publishers can obtain information related to presence on the platform, and the platform can inform or facilitate optimization of strategies for sharing or otherwise distributing content thereon. The presence information may be platform-wide or related to a specific dedicated channel provided by the platform. In one embodiment, the information about presence data may include variations in presence data by date, time, and / or geographic information.

[0155] Information about presence data across platforms or applications may be used to influence the provision of advertising space therein. For example, measured presence may be used as a metric to trigger the generation of advertising inventory, i.e., space for presenting advertisements. For example, if the measured presence for a particular channel or application exceeds a predetermined threshold, additional advertising inventory may be provided. Alternatively or additionally, presence may be used as a metric to adjust or otherwise control the cost of advertising inventory. In one example, a publisher (a provider of advertising inventory) may increase the cost of advertising inventory associated with a level of presence above a certain threshold. In another example, advertisers (looking to purchase advertising inventory to obtain advertising impressions) may adjust their bidding strategy, i.e., the amount they bid for advertising space, based on the presence metric associated with the advertising inventory.

[0156] 5 is a flowchart of a method 600 for optimizing a digital advertising campaign. The method is applicable to programmatic advertising techniques, where a digital advertising campaign has a defined goal and a target audience strategy aimed at achieving that goal. The target audience strategy can form an input to a demand-side platform (DSP) that is charged with delivering advertising content to users in a manner that meets the defined goal.

[0157] Method 600 begins by accessing 602 an effectiveness dataset representing changes in presence parameters over time during the playing of pieces of advertising content belonging to a digital advertising campaign to a plurality of users. The effectiveness dataset may be of the type discussed above, where the presence parameters are obtained by applying behavioral data collected from each user during the playing of the pieces of advertising content to a machine learning algorithm trained to map behavioral data to presence parameters.

[0158] The method continues with step 604 of generating candidate adjustments to the target audience strategy associated with the digital advertising campaign. The candidate adjustments may vary any applicable parameters of the target audience strategy. For example, they may change the demographics or interest information of the target audience. Multiple candidate adjustments may be generated. The candidate adjustments may be generated based on information from an effectiveness dataset for the digital advertising campaign. For example, the candidate adjustments may seek to increase the influence of portions of the target audience for which the presence parameters are relatively high, or may seek to decrease the influence of portions of the target audience for which the presence parameters are relatively low.

[0159] The method continues with step 606 of predicting the effect on the presence parameters of applying the candidate adjustments.

[0160] The method continues with step 608 of evaluating the predicted effect on campaign goals for the digital advertising campaign. The campaign goals may be quantified by one or more parameters. Thus, the evaluating step compares the predicted values ​​of those parameters to the current values ​​for the digital advertising campaign. In one embodiment, the campaign goal may relate to maximizing presence, such that improvements to the target audience strategy manifest as an increase in the presence parameter.

[0161] The method continues with step 610 of updating the target audience strategy with the candidate adjustment if the predicted effect improves performance against the campaign objectives by more than a threshold amount. In the above example, this can be an improvement in a presence parameter (e.g., a share of the existing audience achieved by the advertising campaign) above a threshold amount. The update may be performed automatically, i.e., without human intervention. Thus, the target audience strategy may be automatically optimized.

[0162] As discussed above, the present invention may find application in measuring the effectiveness of advertising, however, it may also find use in other areas.

[0163] For example, the present invention may find use in the evaluation of online educational materials such as video lectures, webinars, etc. It may also be used to measure presence to locally displayed written text, survey questions, etc. In this context, it may be used to assess the effectiveness of the content itself or the effectiveness of individual students, for example, if they were present within a period of viewing the training material sufficient to be allowed to take an exam.

[0164] In another embodiment, the present invention may be used in gaming applications, with a single participant or multiple participants, either running locally on a user device or running online. Any aspect of gameplay may provide displayed content for which presence is measurable. The present invention may be used as a tool to indicate and measure the effectiveness of changes to gameplay.

Claims

1. A method implemented by a computer of an analysis server that collects data from a user device used by a user within a period of time during which information is output from the user device, the method comprising: collecting contextual attribute data indicative of events occurring at the user device within a period of time for the output of the information; receiving sensor data from the user device collected by one or more sensors of the user device during a period of time during which the information is output; applying the sensor data to a classification algorithm to generate presence data, the classification algorithm being a machine learning algorithm operable to map the sensor data to presence parameters, the presence data being indicative of fluctuations in the presence parameters over time within a period of the output of the information; synchronizing the presence data with the contextual attribute data to generate a validity data set linking changes in the presence parameters over time with corresponding contextual attribute data obtained within the period of the output of the information; storing the efficacy data set in a data store; The computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the information includes content to be displayed on the user device.

3. the displayed content includes media content played by an application running on the user device; the contextual attribute data further indicates events occurring in the application during the time the media content is played.

3. The computer-implemented method of claim 2.

4. The computer-implemented method of claim 3 , wherein the contextual attribute data includes control analysis data for the application.

5. The computer-implemented method of claim 3 or 4, further configured to receive, from the application, presence data synchronized with the contextual attribute data.

6. the application is configured to communicate with an analysis module running in the background of the user device; The computer-implemented method of claim 3 or 4, wherein the analysis module is configured to generate the presence data and synchronize the presence data with the contextual attribute data.

7. The computer-implemented method of any one of claims 3 to 6, wherein the application includes an adapter module configured to communicate with an analysis server over a network.

8. the contextual attribute data further indicating events occurring during a time period during which the content is displayed on a web page of a web domain hosted by the content server; 3. The computer-implemented method of claim 2.

9. The method of claim 8 , further comprising transmitting a context data initiation script that describes a script to be executed on the user device.

10. The method of claim 9 , wherein the context data initiation script is inserted into the source code of the web page by an intermediary over a network between the content server and the user device.

11. the context data initiation script is included in a video ad response from an ad server; 10. The method of claim 9.

12. The method includes, by the user device using the context data initiation script: (i) permission to transmit sensor data has been revoked; or (ii) the unavailability of a device for collecting the sensor data; or (iii) the user is not selected for sensor data collection; and terminating the sensor data collection procedure upon determining A computer-implemented method according to any one of claims 9 to 11.

13. The method includes, by the user device using the context data initiation script: (i) permission to transmit sensor data has been granted; (ii) the availability of a device for collecting said sensor data; and (iii) determining that the user has been selected for sensor data collection; 12. The computer-implemented method of claim 11, further comprising loading a real-time communication protocol for transmitting the sensor data from the user device to the analytics server.

14. 14. The computer-implemented method of claim 12 or 13, wherein applying the sensor data to the classification algorithm occurs at the analytics server.

15. The computer-implemented method of any one of claims 8 to 13, comprising receiving presence data generated by a classification algorithm of the user device.

16. The computer-implemented method of any one of claims 8 to 15, wherein the contextual attribute data includes web analytics data about the web page.

17. The computer-implemented method of any one of claims 1 to 16, wherein the user sensor data comprises images captured using a camera.

18. 20. The computer-implemented method of claim 17, wherein the classification algorithm operates to evaluate the presence parameter for each image in a plurality of images of the user captured within a period of the output of the information.

19. applying the sensor data to an emotional state classification algorithm to generate emotional state data for the user, the emotional state classification algorithm being a machine learning algorithm operable to map the sensor data to emotional state data, the emotional state data indicating variations over time in the probability that the user will have a given emotional state within a period of output of the information; synchronizing the emotional state data with the presence data, wherein the efficacy data set further includes the emotional state data; The computer-implemented method of any one of claims 1 to 18, further comprising:

20. receiving, by the analytics server over a network, contextual attribute data and sensor data from a plurality of user devices; aggregating, by the analytics server, a plurality of validity data sets obtained from the contextual attribute data and sensor data received from the plurality of user devices; The computer-implemented method of claim 1 further comprising:

21. 21. The computer-implemented method of claim 20, wherein the multiple validity data sets are aggregated with respect to one or more common dimensions shared by the contextual attribute data and the sensor data received from the multiple user devices.

22. 22. The computer-implemented method of claim 21, wherein the common dimension includes any of a web domain, a website identity, a date and time, and a type of output information.

23. The output information is content acquired and displayed by an application running on the user device, and the method includes: using the aggregated validity data set to determine software updates for the application; receiving the software update at the user device; adjusting functionality of the application by performing the software update; The computer-implemented method of any one of claims 20 to 22, further comprising:

24. 1. A system for collecting data from a user device within a period of output of information from the user device, the system comprising: collecting contextual attribute data from the user device indicative of events occurring at the user device within a period of time of the output of the information; collecting sensor data from one or more sensors of the user device during the time period for outputting the information; applying the received sensor data to a classification algorithm to generate presence data, the classification algorithm being a machine learning algorithm operable to map the sensor data to presence parameters, the presence data being indicative of fluctuations in the presence parameters over time within a period of the output of information; synchronizing the presence data with the contextual attribute data to generate a validity data set linking changes in the presence parameters over time with corresponding contextual attribute data obtained within the period of the output of the information; storing the efficacy data set in a data store; The system is configured to:

Images