Attention tracking using sensors

Abstract

Some disclosed methods involve acquiring sensor data from a sensor system during content presentation and estimating user response events based on said sensor data. Some disclosed methods involve generating a user attention analysis based at least in part on estimated user response events corresponding to estimated user attention to content segments of the content presentation. Some disclosed methods involve modifying the content presentation and providing the modified content presentation based at least in part on the user attention analysis.

Description

Technical Field

[0003] , , , , ,

[0001] This application claims the benefit of priority of International Patent Application No. PCT / CN2023 / 105695 filed on July 4, 2023, US Provisional Application No. 63 / 513,318 filed on July 12, 2023, and US Provisional Application No. 63 / 568,378 filed on March 21, 2024, each of which is hereby incorporated by reference in its entirety. Technical Field This disclosure relates to devices, systems, and methods for estimating a user's attention level and related factors based on signals from one or more sensors, and to responses to such estimated user attention levels.

Background Art

[0002] Several methods, devices, and systems for estimating user attention, such as a user's attention to advertising content, are known. Previous implemented approaches for estimating a user's attention to media content involve evaluating the rating of the content by a person after the person has consumed the content, for example, after a person has finished watching a movie or an episode of a TV show, after a user has played an online game, etc. Existing devices, systems, and methods can provide benefits in some situations, but improved devices, systems, and methods would be desirable.

Summary of the Invention

Means for Solving the Problems

[0003] At least some aspects of this disclosure may be implemented by one or more methods. In some cases, one or more methods may be implemented at least in part by a control system and / or by instructions (e.g., software) stored in one or more non-temporary media. Some disclosed methods involve acquiring sensor data from a sensor system during content presentation and estimating user response events based on the sensor data. Some disclosed methods involve generating user attention analytics at least in part on estimated user response events corresponding to estimated user attention to content segments of content presentation. Some disclosed methods involve modifying content presentation and providing a modified content presentation at least in part on the user attention analytics.

[0004] Some or all of the operations, functions, and / or methods described herein may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include, but are not limited to, memory devices such as random-access memory (RAM) devices and read-only memory (ROM) devices, as described herein. Thus, some innovative aspects of the subject matter described herein can be implemented via one or more non-temporary media on which software is stored.

[0005] At least some aspects of this disclosure may be implemented through one or more devices. For example, one or more devices (e.g., a system comprising one or more devices) may be capable of performing at least partially the methods disclosed herein. In some implementations, the device is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof. The control system may be configured to perform some or all of the methods disclosed herein.

[0006] In some examples, the system may include a head unit, a loudspeaker system, a sensor system, and a control system. The control system may include one or more device analysis engines and a user attention analysis engine. One or more device analysis engines may be configured to estimate user response events based on sensor data received from the sensor system. The user attention analysis engine may be configured to generate user attention analysis based at least in part on estimated user response events received from one or more device analysis engines. The user attention analysis may correspond to estimated user attention to content segments of content presentation delivered via the head unit and loudspeaker system.

[0007] The control system may be configured to modify content presentation based at least partially on user attention analysis, and to provide the modified content presentation by the head unit, the loudspeaker system, or both the head unit and the loudspeaker system.

[0008] In some examples, the system may also include an interface system configured to provide communication between the control system and one or more other devices over a network. In some examples, the modified content presentation may be or include modified content received from one or more other devices over the network. According to some examples, modifying the content presentation may involve sending a user attention analysis from a user attention analysis engine via the interface system and receiving the modified content in response to the user attention analysis.

[0009] In some examples, modifying content presentation may include personalizing or enhancing content presentation. In some examples, personalizing or enhancing content presentation may involve changing one or more of the following: audio playback volume, audio rendering position, one or more other audio characteristics, or a combination thereof.

[0010] In some examples, the head unit may be a television or include a television. According to some such examples, personalizing or enhancing content presentation may involve modifying one or more television display characteristics. In some examples, the head unit may be a digital media adapter or include one.

[0011] As some examples show, personalizing or enhancing content presentation may involve altering the storyline, adding characters or other story elements, changing the time intervals in which characters are involved, changing the time intervals dedicated to other aspects of content presentation, or a combination of these.

[0012] In some cases, personalizing or enhancing content presentation may involve providing personalized advertising content. In some such cases, providing personalized advertising content may involve providing advertising content that corresponds to an estimated user's attention to one or more content segments relating to one or more products or services.

[0013] In some examples, personalizing or enhancing content presentation may involve providing or modifying the humor track.

[0014] In some examples, the head unit, loudspeaker system, and sensor system may be present in the first environment. According to some such examples, the control system may be further configured to modify content presentation based at least in part on sensor data, estimated user response events, user attention analysis, or a combination thereof, corresponding to one or more other environments.

[0015] According to some examples, the control system may be further configured to pause or replay content presentation.

[0016] In some examples, a sensor system may include one or more cameras. According to some such examples, sensor data may include camera data.

[0017] In some examples, the sensor system may include one or more microphones. In some such examples, the sensor data may include microphone data. In some such examples, the control system may be further configured to implement an echo management system to mitigate the effects of audio played back by the loudspeaker system and detected by one or more microphones.

[0018] In some examples, one or more first parts of the control system may be deployed in a first environment, and a second part of the control system may be deployed in a second environment. In some such examples, one or more first parts of the control system may be configured to implement one or more device analysis engines, and a second part of the control system may be configured to implement a user attention analysis engine.

[0019] Details of one or more implementations of the subject matter described herein are given in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative dimensions in the following drawings may not be drawn to scale. [Brief explanation of the drawing]

[0020] Similar reference numbers and names in various drawings indicate the same elements. [Figure 1A] A block diagram showing examples of components of a device that can implement various aspects of this disclosure. [Figure 1B] This document describes an environment that includes examples of components capable of implementing various aspects of this disclosure. [Figure 1C] Another environment is shown, including examples of components that can implement various aspects of this disclosure. [Figure 2] An example of an Attention Tracking System (ATS) component is shown. [Figure 3A] An example of the components of the ATS present in a regeneration environment is shown. [Figure 3B] This example shows the components of an ATS (Automatic Tracking System) that exist within the cloud. [Figure 4] This example shows the components of a neural network capable of performing real-time acoustic event detection. [Figure 5] An example of a device analysis engine configured to perform real-time pose estimation is shown. [Figure 6] Shows the components of ATS according to another example. [Figure 7] Shows an example of providing feedback from ATS during a video conference. [Figure 8] Shows an example of providing feedback from ATS during a video conference. [Figure 9] Shows an example of providing feedback from ATS during a video conference. [Figure 10] Shows an example of providing feedback from ATS during a video conference. [Figure 11] Shows an example of presenting feedback from ATS after a video conference. [Figure 12] It is a flowchart outlining an example of the disclosed method.

Mode for Carrying Out the Invention

[0021] Currently, we spend a lot of time consuming media content, which includes but is not limited to audio-visual content, interacting with media content, or a combination thereof. (For simplicity, both consuming media content and interacting with media content may be referred to herein as "consuming" media content.) Consuming media content may involve watching TV programs or movies, viewing or listening to advertisements, listening to music or podcasts, playing games, participating in video conferences, attending online learning lectures, etc. Thus, movies, online games, video games, video conferences, online advertisements, learning lectures, podcasts, streamed music, etc. may be referred to in this lecture as types of media content.

[0022] Previous approaches to estimating user attention to media content such as movies and television shows do not consider how a person reacts while they are in the process of consuming the media content. Instead, a person's impression may be evaluated according to their rating of the content after the user has consumed it, for example, after they have finished watching an episode of a movie or TV show, or after they have played an online game. Current systems often track which content a user selects, where they choose to stop that content, which sections of that content are replayed, etc. These metrics lack detailed information about how consumers engage with the content while it is being consumed. Current systems generally do not even recognize whether a user is present at any particular time. For this reason, current methods of estimating user attention generally do not provide any real-time information. Such real-time information, in addition to real-time feedback on content playback, would allow for much more detailed analysis, which could improve the user experience.

[0023] It would be beneficial to estimate one or more states of a person while they are in the process of consuming media content. Such states may include, or are related to, the user's attention, cognitive load, and interests. This real-time information, in addition to real-time feedback on content playback, allows for more detailed analysis, which in turn can improve the user experience.

[0024] Various disclosed examples overcome the limitations of previously implemented approaches to estimating user attention. Some disclosed technologies and systems utilize available sensors to detect user responses or absences in real time. Some such examples involve using one or more cameras, eye trackers, ambient light sensors, microphones, wearable sensors, or combinations thereof. Some such examples involve measuring levels of a person's engagement, heart rate, cognitive load, attention, and interest while they are consuming media content, such as watching television, playing games, participating in telecommunication experiences (such as video conferencing or video seminars), or listening to podcasts. Recent advances in AI, such as automatic speech recognition (ASR), emotion recognition, and eye tracking, have made such attention tracking systems possible. Furthermore, smart devices and diverse human-oriented sensors are becoming commonplace in our lives. For example, smart speakers, phones, and televisions (TVs) have microphones, game consoles have cameras, and smartwatches have electroskin response. By using some or all of these technologies in combination, it becomes possible to achieve improved user attention tracking systems.

[0025] Furthermore, the presence of multiple playback devices can lead to a richer playback experience, which may include audio playback from satellite smart speakers and haptic feedback from mobile phones. However, coordinating multiple devices for audio playback can make it difficult to detect audio events using microphones due to echoes. Therefore, some disclosed technologies and systems include descriptions of various approaches to performing echo management.

[0026] In addition to the increased volume of data supplied by sensors, advancements in deep neural networks (DNNs) have provided deeper insights into the meaning behind the data collected by these sensors. DNNs are now capable of tasks such as ASR (Autonomous Sensory Recognition), emotion recognition, and eye tracking. The output of such models can serve as indicators for real-time attention metrics.

[0027] Some disclosed examples involve using real-time sensor data to adapt content presentation for users in real time. Some such examples may involve modifying one or more media contents in response to estimated attention, arousal, cognitive load, etc. Some disclosed examples involve aggregating real-time user attention information at both the user and content levels to provide insights into future content options for that user and changes to that content for future consumers, respectively.

[0028] According to several examples, an attention tracking system consistent with the proposed technique may form the following feedback loop: • Content is presented and played from one or more devices. • Users pay a certain level of attention to the content presented. The sensor detects the user's attention signal. The attention analysis engine determines the user's attention level in real time. • Content presentation is adapted in real time to improve the user experience.

[0029] In some cases, user attention analysis can be aggregated over time to determine how people engage with content presentation and their affinity for different aspects of that content presentation. These user attention analyses can then be used in conjunction with traditional attention detection methods.

[0030] An exemplary application of instant, real-time attention detection is a virtual laugh track. A virtual laugh track allows users to laugh along whenever they find the content presentation humorous. Short-term uses of real-time user feedback may include personalized content selection and targeted advertising. In some cases, longer-term analysis may be used to adjust content presentation to appeal to a larger audience, such as for improved targeted user advertising.

[0031] What does "caution" mean? User attention, engagement, response, and reaction may be interchangeable throughout this disclosure. In some embodiments of the proposed techniques and systems, user response may refer to any form of attention to the content, such as audible responses, body posture, physical gestures, heart rate, or wearing of content-related clothing. Attention may take many forms, such as binary (e.g., the user saying "yes"), spectral (e.g., excitement, loudness, leaning forward), or open-ended (e.g., topic of discussion, multidimensional embedding). Attention may infer something related to the content presentation or an object within the content presentation. Attention to non-content-related information, on the other hand, may correspond to a low level of engagement with the content presentation.

[0032] As some examples show, the attention to be detected may be within a shortlist defined by any combination of user, content, content provider, user device, etc. (e.g., "Wow," "Ahh," "Red," "Blue," leaning forward, leaning out, raising left hand, raising right hand). You will understand that such a shortlist is not required. If the shortlist of possible responses is supplied by the content presentation, it may arrive through the metadata stream of the content presentation.

[0033] Figure 1A is a block diagram showing examples of components of an apparatus capable of implementing various aspects of the present disclosure. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 1A are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements. According to some examples, apparatus 150 may be configured to perform at least some of the methods disclosed herein. In some implementations, apparatus 150 may be one or more components of a workstation, one or more components of a home entertainment system, etc., or may include them. For example, apparatus 150 may be a laptop computer, a tablet device, a mobile device (such as a cellular phone), an augmented reality (AR) wearable, a virtual reality (VR) wearable, an automotive subsystem (e.g., an infotainment system, driver assistance or safety system), a game system or console, a smart home hub, a television, or another type of device.

[0034] According to some alternative implementations, device 150 may be a server or may include a server. In some such examples, device 150 may be an encoder or may include an encoder. In some examples, device 150 may be a decoder or may include a decoder. Thus, in some cases, device 150 may be a device configured for use in an environment such as a home environment, and in other cases, device 150 may be a device configured for use in a "cloud," for example, a server.

[0035] In some examples, the device 150 may be, or include, an orchestration device configured to provide control signals to one or more other devices. In some examples, the control signals may be provided by the orchestration device to coordinate aspects of displayed video content, audio playback, or a combination thereof. In some examples, the device 150 may be configured to modify one or more aspects of media content currently being provided by one or more devices in the environment in response to estimated user engagement, estimated user arousal, or estimated user cognitive load. Several examples are disclosed herein.

[0036] In this example, the device 150 includes an interface system 155 and a control system 160. In some implementations, the interface system 155 may be configured to communicate with one or more other devices in the environment. In some examples, the environment may be a home environment. In other examples, the environment may be a different type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, or an entertainment environment (e.g., a theater, a performance venue, a theme park, a VR experience room, an electronic game arena). In some implementations, the interface system 155 may be configured to exchange control information and related data with other devices in the environment. In some examples, the control information and related data may relate to one or more software applications that the device 150 is running.

[0037] In some implementations, the interface system 155 may be configured to receive or provide a content stream. In some examples, the content stream may include video data and corresponding audio data. The audio data may include, but is not limited to, audio signals. In some cases, the audio data may include channel data and / or spatial data such as spatial metadata. The metadata is provided, for example, by what is referred to herein as an “encoder”.

[0038] The interface system 155 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, the interface system 155 may include one or more wireless interfaces. The interface system 155 may include one or more devices for implementing the user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, a gesture sensor system, or a combination thereof. Thus, although some such devices are shown separately in Figure 1A, such devices may, in some examples, correspond to aspects of the interface system 155.

[0039] In some examples, the interface system 155 may include one or more interfaces between the control system 160 and a memory system, such as the optional memory system 165 shown in Figure 1A. Alternatively or additionally, the control system 160 may include a memory system in some cases. In some implementations, the interface system 155 may be configured to receive input from one or more microphones in the environment.

[0040] The control system 160 may include, for example, a general-purpose single or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0041] In some implementations, the control system 160 may reside in two or more devices. For example, in some implementations, part of the control system 160 may reside in a device within one of the environments referred to herein, while another part of the control system 160 may reside in a device outside the environment, such as a server, a game console, or a mobile device (such as a smartphone or tablet computer). In other examples, part of the control system 160 may reside in a device within one of the environments shown herein, while another part of the control system 160 may reside in one or more other devices within the environment. For example, the control system functions may be shared by a command device (such as one sometimes referred to herein as a smart home hub) and one or more other devices within the environment. In other examples, part of the control system 160 may reside in a device implementing cloud-based services, such as a server, while another part of the control system 160 may reside in another device implementing cloud-based services, such as another server or a memory device. The interface system 155 may also reside in two or more devices in some examples.

[0042] In some implementations, the control system 160 may be configured to perform at least partially the methods disclosed herein. In some examples, the control system 160 may implement one or more device analysis engines configured to estimate user response events based on sensor data received from a sensor system. In some examples, the control system 160 may implement a user attention analysis engine configured to generate user attention analysis based at least partially on the estimated user response events received from one or more device analysis engines. The user attention analysis may correspond to estimated user attention to content segments of content presentation. In some examples, the control system 160 may be configured to modify the content presentation based at least partially on the user attention analysis. In some examples, the control system 160 may be configured to have the modified content presentation delivered by one or more displays, by a loudspeaker system, or by one or more displays and a loudspeaker system. Some examples of these components and processes are described below.

[0043] Some or all of the methods described herein may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include memory devices such as those described herein, including but not limited to random-access memory (RAM) devices and read-only memory (ROM) devices. One or more non-temporary media may reside, for example, in an arbitrary memory system 165 and / or control system 160 shown in Figure 1A. Thus, various innovative aspects of the subject matter described herein may be implemented on one or more non-temporary media on which software is stored. The software may include, for example, instructions for controlling at least one device to execute some or all of the methods disclosed herein. The software may be executable by one or more components of a control system, such as the control system 160 in Figure 1A.

[0044] In some examples, the device 150 may include an optional microphone system 170, as shown in Figure 1A. The optional microphone system 170 may include one or more microphones. According to some examples, the optional microphone system 170 may include an array of microphones. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, for example, according to instructions from a control system 160. In some cases, the array of microphones may be configured for receiving beamforming, for example, according to instructions from a control system 160. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker in a speaker system or a smart audio device. In some examples, the device 150 may not include a microphone system 170. However, in some such implementations, the device 150 may nevertheless be configured to receive microphone data about one or more microphones in the environment via an interface system 160. In some such implementations, the cloud-based implementation of the device 150 may be configured to receive microphone data or data corresponding to microphone data from one or more microphones in the environment via the interface system 160.

[0045] In some implementations, the device 150 may include an optional loudspeaker system 175, as shown in Figure 1A. The optional loudspeaker system 175 may include one or more loudspeakers, which may also be referred to herein as “speakers” or more commonly as “audio playback transducers.” In some examples (e.g., cloud-based implementations), the device 150 may not include the loudspeaker system 175.

[0046] In some implementations, device 150 may include an optional sensor system 180, as shown in Figure 1A. The optional sensor system 180 may include one or more touch sensors, gesture sensors, motion detectors, cameras, eye-tracking devices, or a combination thereof. In some implementations, one or more cameras may include one or more standalone cameras. In some examples, one or more cameras, eye trackers, etc., of the optional sensor system 180 may be present in a television, mobile phone, smart speaker, laptop, game console or system, or a combination thereof. In some examples, device 150 may not include a sensor system 180. However, in some such implementations, device 150 may nevertheless be configured to receive sensor data about one or more sensors (cameras, eye trackers, monitors with cameras, etc.) present in or on other devices in the environment via the interface system 160. Although the microphone system 170 and the sensor system 180 are shown as separate components in Figure 1A, the microphone system 170 may also be called the sensor system 180 and may be considered part of the sensor system 102.

[0047] In some implementations, the device 150 may include an optional display system 185 as shown in Figure 1A. The optional display system 185 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, the optional display system 185 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 185 may include one or more displays of a television, laptop, mobile device, smart audio device, automotive subsystem (e.g., infotainment system, driver assistance or safety system) or another type of device. In some examples where the device 150 includes the display system 185, the sensor system 180 may include a touch sensor system and / or gesture sensor system adjacent to one or more displays of the display system 185. According to some such implementations, the control system 160 may be configured to control the display system 185 to present one or more graphical user interfaces (GUIs).

[0048] According to some such examples, device 150 may be or include a smart audio device such as a smart speaker. In some such implementations, device 150 may be or include a wake word detector. For example, device 150 may be configured to implement (at least partially) a virtual assistant.

[0049] Figure 1B shows an environment including an example of components that can implement various aspects of this disclosure. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 1B are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0050] In this example, environment 100A includes a head unit 101, which in this example is a television (TV). In some implementations, the head unit 101 may be or include a digital media adapter (DMA) such as Apple TV® DMA, Amazon Fire® DMA, or Roku® DMA. According to this example, content presentation is delivered via the head unit 101 and a loudspeaker system including the TV's loudspeaker and satellite speakers 102a and 102b. In this example, the attention levels of one or more people 105a, 105b, 105c, 105d, and 105e are detected by a camera 106 on the TV, by microphones on satellite speakers 102a and 102b, by microphones on a smart couch 104, and by microphones on a smart table 103.

[0051] In this example, the sensors in environment 100A are primarily used to detect auditory and visual feedback that can be detected by camera 106. However, in alternative implementations, the sensors in environment 100A may include additional types of sensors, such as one or more additional cameras, eye trackers configured to collect gaze and pupil size information, one or more ambient light sensors, one or more thermal sensors, and one or more sensors configured to measure electrocutaneous responses. According to some implementations, one or more cameras in environment 100A, which may include camera 106, may be configured for eye-tracking functionality.

[0052] The elements of Figure 1B include the following: 101: A head unit 101, which in this example is a TV, that provides audiovisual content and detects the user's attention using a microphone; 102a, 102b: Multiple satellite speakers that play content and detect attention using a microphone array; 103: A smart table that uses a microphone array to detect attention; 104: A smart couch that uses a microphone array to detect attention; 105a, 105b, 105c, 105d, 105e: Multiple users paying attention to content within environment 100; 106: A camera mounted on head unit 101.

[0053] Figure 1C shows another environment including an example of components that can implement various aspects of this disclosure. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 1C are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0054] In this example, Figure 1C illustrates a scenario in which an Attention Tracking System (ATS) is implemented in a car setting. Thus, environment 100B is the car environment in this example. Rear-seat passengers 105h and 105i are paying attention to content on their respective displays 101c and 101d. Front-seat users 105f and 105g are paying attention to the drive, to display 101b, and to audio content played from the loudspeaker. Audio content may include music, podcasts, navigation guidance, etc. The ATS utilizes all sensors in the car to determine the degree of each user's attention to the content.

[0055] The elements in Figure 1C include the following: 101b: Main display screen in a car with satellite navigation, reverse camera, music control, etc. 101c, 101d: Passenger screens designed to play entertainment content such as movies; 105f, 105g, 105h, 105i: Multiple users paying attention to in-vehicle content; 106b: A camera facing the outside of the vehicle that detects content such as advertising boards; 106c: A camera facing the interior of the vehicle that detects the user's attention; 301d, 301e, 301f, 301g: Multiple microphones that pick up noise inside the vehicle, including audio indications for content playback and user attention; 304d, 304e, 304f, 304g: Multiple speakers for playing content.

[0056] Figure 2 shows exemplary components of an Attention Tracking System (ATS). As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 2 are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0057] In this example, content presentation 202, also called content playback 202, is delivered via a device in environment 100C, and user responses are then detected by one or more sensors of a sensor system 180. In this example, an Attention Analytics Engine (AAE) 201 is configured to estimate user attention levels based on sensor data from one or more sensors. In this example, an analysis of how one or more people 105 pay attention to the content presentation can be aggregated over time, where a content analytics and user affinity module 203 is configured to estimate individual user affinity and group affinity to various aspects of the content presentation. The groups may correspond to specific demographics or populations. In this example, the content presentation module 205 is configured to adjust the content presentation in real time according to the attention analysis 206 received from the AAE 201. Adjustments made by the content presentation module 205 may include things such as content selection, virtual laughter, and cheers. Various additional examples of adjustments that may be made by the content presentation module 205 are disclosed herein. Real-time adjustments to the content made by the content presentation module 205, and corresponding user feedback detected by the sensor system 180, can create a feedback loop that focuses on a better experience for one or more people 105.

[0058] The elements in Figure 2 include the following: 105: One or more people who are paying attention to content playback 202 and have a reaction detected by the sensor; 180: A sensor system comprising one or more sensors configured to detect real-time user responses to content presentation 202; 201: An Attention Analysis Engine (AAE) that analyzes user attention information by referencing the current content, by acquiring data from the sensor system 180 (or, in some implementations, results from one or more Device Analytics Engines (DAEs) generated using measurements from the sensors). An example of a DAE is described below.

[0059] 202: Playback of content supplied by the content presentation module 205 through any combination of loudspeakers, displays, lights, etc. 203: Content analytics and user affinity module configured for aggregating attention analytics to provide insights into user attention to content playback 202; 205: A content presentation module configured to supply content to be played and to make real-time adjustments to content presentation based on attention analysis 206 supplied by AAE 201.

[0060] In the example shown in Figure 2, the Attention Tracking System (ATS) 200 includes an AAE 201, a content analysis and user affinity module 203, a sensor system 180, and a content presentation module 205. In this example, the AAE 201, the content analysis and user affinity module 203, and the content presentation module 205 are implemented by an instance of the control system 160 in Figure 1A. In some examples, the AAE 201, the content analysis and user affinity module 203, and the content presentation module 205 may be implemented as instructions, such as software, stored on one or more non-temporary and computer-readable media.

[0061] According to some examples, the ATS includes an attention analysis engine 201, sensors, and one or more device analysis engines (not shown in Figure 2), each corresponding to a sensor or group of sensors, all operating in real time. In some such examples, the ATS is • Collecting information from available sensors and • Passing sensor information through one or more device analysis engines, and • Determining user attention by passing the results of the device analysis engine through the attention analysis engine (201). It is configured for this purpose.

[0062] The output of ATS may vary depending on the specific implementation. In some examples, the output of ATS may be any type of attention analysis-related information, content presentation corresponding to said attention analysis-related information, etc. According to some examples, ATS may be implemented via an attention analysis engine 201 and a content presentation module 205 that reside on a device in the playback environment (for example, implemented by one of the playback devices in the playback environment), or via an attention analysis engine 201 and a content presentation module 205 that reside in the "cloud" (for example, implemented by one or more servers that are not in the playback environment), and so on.

[0063] Figure 3A shows the components of an ATS present in an example playback environment. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 3A are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0064] In the example in Figure 3A, the ATS 200 includes an Attention Analysis Engine (AAE) 201 and a Content Presentation Module 205 located in the head unit 101, which in this example is a television (TV). Multiple sensors, including the microphone 301a on the head unit 101, the microphone 301b on the satellite speaker 102a, and the microphone 301c on the satellite speaker 102b, provide the AAE 201 with sensor data corresponding to what is happening within the environment 100D. Other implementations may include additional types of sensors such as one or more cameras, one or more eye trackers configured to collect gaze and pupil size information, one or more ambient light sensors, one or more thermal sensors, and one or more sensors configured to measure electrocutaneous responses. Some implementations may include three or more satellite speakers, as indicated by the dot and element number 102c.

[0065] Microphones 301a-301c detect the audio of the content presentation. In this example, echo management modules 302a, 302b, and 302c are configured to suppress the audio of the content presentation, allowing for more reliable detection of sounds corresponding to the user's response to the content in the signals from microphones 301a-301c. In this example, the content presentation module 205 is configured to send echo reference information 306 to the echo management modules 302a, 302b, and 302c. The echo reference information 306 may include, for example, information about the audio being played by loudspeakers 304a, 304b, and 304c. As a simple example, local echo paths 307a, 307b, and 307c may be canceled by a local echo canceller using a local echo reference. However, any type of echo management system, such as a distributed acoustic echo canceller, could be used here.

[0066] In the example shown in Figure 3A, the head unit 101 includes a device analysis engine (DAE) 303a, the satellite speaker 102a includes a DAE 303b, and the satellite speaker 102b includes a DAE 303c. Here, DAEs 303a, 303b, and 303c are configured to detect user activity from sensor signals, which are microphone signals in this example. Different implementations of the DAE 303 may exist, and in some cases may even exist within the same attention tracking system. A particular implementation of the DAE 303 may depend, for example, on the type or mode of the sensor. For example, some implementations of the DAE 303 may be configured to detect user activity from microphone signals, while other implementations of the DAE 303 may be configured to detect user activity, attention, etc., based on camera signals. Some DAEs may be multimodal, receiving and interpreting inputs from different sensor types. In some examples, DAEs may share sensor inputs with other DAEs. The output of the DAE 303 may also vary depending on the specific implementation. DAE outputs may include, for example, detected phonemes, emotion type estimation, heart rate, body posture, and latent spatial representation of sensor signals.

[0067] In the implementation shown in Figure 3A, the outputs 309a, 309b, and 309c from DAEs 303a, 303b, and 303c, respectively, are supplied to AAE 201. AAE 201 is configured to combine the information from DAEs 303a-303c to generate an attention analysis. AAE 201 may be configured to use various types of data extraction techniques, such as neural networks and algorithms. For example, AAE 201 may be configured to use natural language processing (NLP) with speech recognition outputs from one or more DAEs. In this example, the analysis generated by AAE 201 allows for real-time adjustment of content presentation by the content presentation module 205. The content presentation 310 is then provided to actuators, including loudspeakers 304a, 304b, and 304c, and a TV display screen 305, and played back from these actuators. Other examples of actuators that may be used include lights and haptic feedback devices.

[0068] The elements of Figure 3A include the following: 301a, 301b, 301c: Microphones that pick up sounds within an environment of 100D, including audio for content playback and user responses. 302a, 302b, 302c: Echo management modules configured to reduce the level of content playback picked up by the microphone. 303a, 303b, 303c: Device analysis engines that can be configured to convert sensor measurements into probabilities of certain attentional events, such as laughter, gasps, or cheers. 304a, 304b, 304c: Loudspeakers within each device that play audio from content presentation module 205. 305: A TV display configured to display content from content presentation module 205. 306a, 306b, 306c: Echo reference information, and 307a, 307b, 307c: Echo paths between multiple speakers and multiple microphones in a duplex device.

[0069] In these examples, AAE 201, content presentation module 205, echo management module 302a, and DAE 303a are implemented by instance 160a of the control system 160 in Figure 1A; echo management module 302b and DAE 303b are implemented by another instance 160b of the control system 160; and echo management module 302c and DAE 303c are implemented by a third instance 160c of the control system 160. In some examples, AAE 201, content presentation module 205, echo management modules 302a-302c, and DAE 303a-303c may be implemented as instructions, such as software, stored on one or more non-temporary computer-readable media.

[0070] Figure 3B shows an example of the components of an ATS located in the cloud. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 3B are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0071] In this example, the ATS 200 includes the AAE 201 and content presentation module 205, which are implemented by the control system instance 160d of the control system 160 in Figure 1A. The control system instance 160d resides in the cloud 308. The cloud 308 may, for example, reside outside the environment 100E but may include one or more servers configured to communicate with the head unit 101 and at least the satellite speakers 102a and 102b via a network. The echo reference information 306 and local echo path 307 may be as shown in Figure 3A, but have been removed from Figure 3B to reduce the visual complexity of the figure. Those skilled in the art will understand that any type of echo management system can be used here. Two important differences between this implementation and the implementation shown in Figure 3A are as follows:

[0072] The DAE results 309a, 309b, and 309c are sent from DAE 303a, 303b, and 303c, respectively, to AAE 201 in Cloud 308 via the network. The content presentation module 205 is also implemented in the cloud 308 and configured to provide content 310 to devices within environment 100E via the network.

[0073] Acoustic event detection Figure 4 shows the components of a neural network capable of performing real-time acoustic event detection, as an example. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 4 are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0074] Figure 4 shows exemplary components of a neural network 400 that can implement a device analysis engine 303 for microphone 301. In this example, the neural network 400 is implemented by a control system instance 160e. According to this example, the microphone signal 410 from microphone 301 is passed to a banding block 401 to create a time-frequency representation of the audio within the microphone signal 410. The resulting frequency band 412 is passed to two-dimensional convolutional layers 402a and 402b configured for feature extraction and downsampling, respectively. In this example, a positional encode 403 is stacked on top of the features 411 output by the convolutional layers 402a and 402b, so that the real-time streaming transformer 404 can take temporal information into account. The embeddings 414 generated by these transformers are projected to several desired unit scores 406 using a fully connected layer 405. The unit score can represent any unit related to an acoustic event, such as a subword unit, phoneme, laughter, cheer, or gasp. In this example, the SoftMax module 407 is configured to normalize the unit score 406 to a unit probability 408, which represents the posterior probability of the acoustic event.

[0075] Other examples of attention-related acoustic events that may be detected for use in the proposed techniques and systems include:

[0076] Sounds that may indicate involvement: laughter, shouting, cheering, booing, crying, sniffing, groaning, oh, ah, shhh sounds, talking about the content, cursing, etc.

[0077] Sounds that may indicate non-involvement: typing, creaking doors, snoring, footsteps, vacuuming, washing dishes, cutting food, talking about something other than the content, different content playing on a device not connected to the attention system.

[0078] The following sounds may be used to indicate a specific type of content: ○Movies: Name the actors; ○Sports: Name the players or teams; ○If the content includes music: whistling, clapping, tapping feet, snapping fingers, singing along with the content, or making repetitive noises that correspond to the rhythm of the content; ○In children's shows: Children should make vocalizations that express emotions or respond to "call and response" prompts. ○Workout-related content: Groans, heavy breathing, moans, gasps.

[0079] Other noises that may help infer attention to content based on the context of the content: silence during dramatic time intervals, mentioning the names of objects, characters, or concepts in a scene, etc.

[0080] The elements in Figure 4 include the following: 400: An exemplary neural network architecture for a real-time audio event detector. 303d: A device analysis engine configured to detect attention-related events in microphone input data. 401: A bandaging block configured to process time-domain input into bandaged time-frequency domain information. 402a, 402b: 2-dimensional convolutional layers. 403: Position encoding, which overlays position information onto the features 411 output by convolutional layers 402a and 402b. 404: Multiple (six in this example) real-time streaming transformer layers. 405: A fully connected linear layer that functions as a projection onto the unit score 414 output by the real-time streaming transformer layer 404. 406: A unit score representing a different audio event class. A unit score can represent audio events such as laughter, moans, and cheers. 407: A SoftMax module 407 configured to normalize a unit score 406 to a unit probability 408 representing the likelihood of an acoustic event, and 408: The resulting unit probability.

[0081] Visual detection Figure 5 shows, as an example, the components of a device analysis engine configured to perform real-time pose estimation. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 5 are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0082] Figure 5 shows exemplary components of a device analysis engine (DAE) 303e configured to estimate user attention from visual information. In this example, the DAE 303e is implemented by control system instance 160f. Depending on the specific implementation, the DAE 303e may be configured to estimate user attention from visual information through a wide range of techniques. Some examples involve applying machine learning methods, using algorithms implemented via software, and so on.

[0083] In the example shown in Figure 5, the DAE 303e includes a skeletal estimation module 501 and a pose classifier 502. The skeletal estimation module 501 is configured to calculate the positions of major human bones from camera data 510, including a video feed, and to output skeletal information 512. The skeletal estimation module 501 can be implemented using a publicly available toolkit such as YOLO-Pose. The pose classifier 502 can be configured to implement any suitable process for mapping skeletal information to pose probabilities, such as a Gaussian mixture model or a neural network. According to this example, the DAE 303e—the pose classifier 502 in this example—is configured to output pose probabilities 503. In some examples, the DAE 303e can also be configured to estimate the distance of one or more parts of the user's body according to camera data.

[0084] Visual detection can reveal a wide range of attentional information. Some examples include:

[0085] Visuals that can show a person's positive engagement: leaning forward, lying on their back, moving in response to events in the content, wearing clothing that suggests admiration for something in the content, etc.

[0086] Visuals that can indicate a person's negative involvement: such as a disgusted facial expression or a person's hand "giving the middle finger."

[0087] Visuals that can indicate no human involvement: a person looking at a phone when it is not being used to provide content, a person holding a phone on their head when it is not being used to provide content, a person sleeping, no one paying attention in the room, no one at all, etc.

[0088] The elements in Figure 5 include the following: 303e: A device analysis engine configured to perform real-time pose estimation based on camera data 510. 501: A skeletal estimation module configured to calculate the position and rotation of major human bones from camera data 510 and output skeletal information 512. 502: A posture classifier configured to map skeletal information 512 to posture probability 503, and 503: The resulting posture probability.

[0089] System and Method for Measuring and Demonstrating User Attention During Video Interaction This section describes numerous concepts aimed at extracting sensor data captured during video consumption (e.g., watching videos such as movies and social media posts, and participating in video conferences) into attention metrics that are then presented as feedback via a graphical interface. This section includes both techniques for providing live feedback during meetings and techniques for providing offline feedback via reports.

[0090] Caution Model This section describes data collection via sensors typically included with display systems (accelerometers, IMUs, microphones, etc.) or via devices on or around the user (e.g., various sensors on smart devices such as smartwatches). All inputs from sensors, user interface interactions, and camera image processing are analyzed using attentional analysis models.

[0091] Figure 6 shows the components of an ATS in another example. As with other figures provided herein, the types, number, and arrangement of elements shown in Figure 6 are provided for illustrative purposes only. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0092] In this example, the ATS 200 takes independent measurements from various types of sensors within the sensor system 180 as input and outputs one or more extracted attention metrics. In the example shown in Figure 6, the ATS 200 includes device analysis engines (DAEs) 303f, 303g, 303h, 303i, and 303j, as well as the AAE 201, all of which are implemented by the control system instance 160g. In this example, each of the DAE 303f to 303j receives and processes sensor data from different types of sensors in the sensor system 180: DAE 303f receives and processes camera data from one or more cameras, DAE 303g receives and processes motion sensor data 602 from one or more motion sensors, DAE 303h receives and processes wearable sensor data 604 from one or more wearable sensors, DAE 303i receives and processes microphone data 410 from one or more microphones, and DAE 303j receives and processes user input 606 from one or more user interfaces. In this example, DAE 303f generates DAE output 309f, DAE 303g generates DAE output 309g, DAE 303h generates DAE output 309h, DAE 303i generates DAE output 309i, and DAE 303j generates DAE output 309j.

[0093] In this example, AAE 201 is configured to perform one or more types of data extraction methods, data combination methods, or both on DAE outputs 309f-309j. In some examples, data extraction and / or combination may be performed by implementing a fuzzy inference system. Other data extraction and / or combination methods may involve implementing a support vector machine or a neural network. According to some examples, AAE 201 may be configured to implement an attentional analysis model that is tuned or trained against subjectively measured ground truth. In some examples, AAE 201 may be configured to implement an attentional analysis model that is tuned or trained on research training data collected in a research laboratory setting. Such data may not be available in a consumer setting. Research training data may include sensor data from a broader array of sensors than is typical in a consumer setting, such as a complete set of sensors disclosed herein. In some examples, research training data may include measurements that are not typically available to consumers, such as EEG recordings.

[0094] In this example, AAE 201 is configured to output an estimated attention score 610 and individual estimated attention metrics 612. In this example, the estimated attention score 610 is a combined attention score that includes the responses of all people currently consuming the content presentation, and the individual estimated attention metrics 612 are the individual attention scores for each person currently consuming the content presentation.

[0095] Figures 7, 8, 9, and 10 illustrate examples of providing feedback from the ATS during a video conference. Figure 11 illustrates an example of presenting feedback from the ATS after a video conference. In some cases, the ATS may be one such as the one shown in Figure 6, or a similar ATS. According to some examples, the feedback shown in Figures 7–11 may be provided based on an ATS having fewer sensor types and fewer DAEs that provide input to the AAE 201. As with other figures provided herein, the types, number, and arrangement of elements shown in Figures 7–11 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0096] Figure 7 illustrates an example of providing feedback from ATS during a one-to-many video conference presentation. Figure 7 shows examples of various windows that may be displayed on the presenter's display during a video conference presentation. Window 705 shows the presenter's own video, window 710 shows the presented content, and windows 715a, 715b, and 715c show videos of participants who are not being presented. In this example, window 720 shows graphs illustrating three types of ATS feedback. Curve 725 shows estimated excitement, curve 730 shows estimated attention, and curve 735 shows estimated cognitive load. According to this example, the three types of ATS feedback are aggregated and based on data from all non-presenting participants.

[0097] Figure 8 illustrates an example of providing feedback from the ATS during a many-to-many video conference discussion. Figure 8 shows examples of windows that may be presented on each participant's display during a video conference discussion. Window 805 displays each participant's video. In this example, each window 805 contains an attention score 810 indicating estimated attention for that individual participant. In some examples, the ATS may estimate two or more types of feedback. In some such examples, the attention score 810 may show aggregated ATS feedback. In other examples, multiple attention scores 810 may be presented in each of the windows 805. In some alternative examples, the attention score 810 may not be presented on each participant's display during the video conference discussion. In some such examples, the attention score 810 may only be presented on the displays of a subset of participants (e.g., on the display of one participant) during the video conference discussion. The subset of participants may include, for example, supervisors, moderators, etc.

[0098] Figure 9 shows another example of providing feedback from ATS during a many-to-many video conference discussion. Figure 9 shows examples of windows that may be presented on each participant's display or on a subset of participants' displays during a video conference discussion. Window 905 shows each participant's video. In this example, window border 910a is shown with the thickest outline, indicating that the current speaker is shown in the corresponding window 905a. According to this example, window borders 910b and 910c are shown with thinner outlines than window border 910a but thicker outlines than the other window borders 905, indicating that the participants shown in windows 905b and 905c are currently looking at the current speaker.

[0099] Figure 10 shows another example of providing feedback from ATS during a many-to-many video conference discussion. Figure 10 shows examples of windows that may be presented on each participant's display or on the displays of a subset of participants during a video conference discussion. Window 1005 shows each participant's video. In this example, window boundary 1010a is shown with the thickest outline, indicating that the current speaker is shown in the corresponding window 1005a. According to this example, window boundaries 1010b and 1010c are shown with thinner outlines than window boundary 1010a but thicker outlines than the other windows 1005, indicating that the ATS is estimating a high level of attention to the participants shown in windows 1005b and 1005c.

[0100] Figure 11 shows an example of providing feedback from ATS after a many-to-many video conference discussion. The table shown in Figure 11 is the ATS feedback for each of the six video conference participants. In this example, row 1105 shows the utterance percentage for each participant, and column 1110 shows the individual attention score for each participant. The inner cells show the attention of individual participants to each other or to other stimuli. For example, cell 1115a shows user 1's attention to user 2, and cell 1115b shows user 6's attention to user 5. The cells in column 1120 show the individual participants' attention to stimuli or events other than other video conference participants. For example, cell 1115c shows user 6's attention to stimuli or events other than other video conference participants.

[0101] Attention Feedback Loop Any decision made based on information provided by the ATS may have its effectiveness evaluated using the ATS to detect the user's response. In some cases, this can form a closed loop in which decisions made using ATS information can improve over time. In some examples, this closed loop of user attention and responses to user attention can be taken a step further, allowing a decision to be passed on to another user for experience once it has been determined that the decision has a particular effect on the user. Specific example:

[0102] • One or more users are detected laughing at the comedy. This laughter is detected by the user's ATS-enabled device(s). Using this information, additional laughter is added to the comedy (for example, by the content presentation module 205 according to instructions from AAE 201). The ATS detects that this has increased the amusement for the users, and therefore the system (for example, the content presentation module 205 according to instructions from AAE 201) continues to add laughter whenever the ATS detects users laughing at the comedy. Over time, the system begins to optimize which type of added laughter each user finds most amusing.

[0103] • One or more users are detected laughing at a comedic advertisement after a TV program has ended. This laughter is detected on the user's ATS-enabled device(s). Using this information, it is assumed (e.g., by AAE 201) that the user will engage more with comedic advertisements after the program has ended. If this remains true for a user, the system (e.g., the content presentation module 205 following instructions from AAE 201) will continue to show comedic advertisements after the TV program has ended. Over time, the system begins to optimize on an individual basis for which types of comedic advertisements users find most amusing. In some examples, the affinity of individual users or groups to comedic advertisements after a TV program has ended may be determined over many viewing sessions. In some such examples, such affinity may be determined by the content analysis and user affinity block 203, while in others, such affinity may be determined by AAE 201.

[0104] • It is detected that most viewers find unintentionally funny moments within the content more humorous than the rest of the comedy. This humor is detected on the user's ATS-enabled device(s). This information is used to add additional humorous moments similar to this one to the same comedy. It is detected that this generally increased the sense of amusement for the user. Over time, the system begins to optimize how many of these types of gags should be placed in the content.

[0105] • One or more users are detected laughing at a comedy at home. This laughter is detected on the user's ATS-enabled device(s). This information is used to add additional laughter to the comedy. If it is detected that this increased the humor for the user, the system will begin adding laughter when it hears the user laughing at a comedic podcast in their car. Over time, the system will begin to optimize on an individual basis for what type of location and media type the user finds funniest.

[0106] Options for evaluating the effectiveness of decisions made and optimizing decisions based on attention responses apply to all use cases enumerated in this disclosure.

[0107] Personalization and enhancement using attention feedback use cases Modern personalization systems for enjoying content rely on information explicitly provided by the user through the use of control systems for the content, such as play and pause. For example, current content recommendation systems make decisions based on what content the user chooses to consume, which sections the user replays, and whether the user has left a "like" on a particular piece of content. This provides little insight into how the user is paying attention to the content or whether the user is even present during playback in the first place. Furthermore, slow personalization can come in the form of fan feedback through the explicit leaving of comments or reviews on content. This allows content creators to tailor their next content to the audience's preferences, but the feedback loop is slow and only represents users who leave comments and speak up.

[0108] The potential of personalized systems expands significantly with the use of richer user attention metrics, utilizing real-time microphones, cameras, and sensors such as accelerometers. Having such attention tracking systems makes it possible to determine what interests the user regarding content. For example, while watching a secret agent movie, the user might show particular interest in suits, watches, cars, locations, or action. Insights into such details allow for improved personalized systems.

[0109] Adapting to content consumption with richer attentional information can be done in real time and / or over time by aggregating attention metrics. These detailed attention metrics can also lead to better informed recommendations and provide content creators with real-time feedback on how users are paying attention to content. When any personalization decisions are made using attentional information, attention tracking systems may again be used to determine the impact on the user's(single or multiple) attention. The continued use of attention tracking systems can form a closed loop in which many types of personalization can be optimized or at least improved. This section details some possibilities and use cases of personalization systems that use richer attention metrics and sensors in real time.

[0110] As used herein, the term “content viewing experience” may refer to any actions that may affect the user’s experience while viewing content, including changes to playback (e.g., volume, TV backlight level), content (e.g., selected content, changes to the storyline, additions to elements), and control systems (e.g., pause, replay).

[0111] As used herein, the term “personalization” may refer to any content viewing experience tailored to the preferences of one or more users. “Experience augmentation” refers to real-time changes (including, but not limited to, real-time changes) to the content viewing experience. “Recommendation” refers to any content suggested for a user. Personalization, experience augmentation, and recommendation may all be based on cautionary information. These terms are explained in more detail in their respective sections.

[0112] "Linear content" refers to any form of content that is intended to be played from start to finish without branching paths or controls to the content flow (e.g., episodes of a Netflix® series, audiobooks, songs, podcasts, TV shows).

[0113] Detectable alert list and metadata This section provides a more detailed explanation of the potential shortlist of detectable attention, which was briefly described in the introduction. The scope of detectable attention may be specified in the list. An example of the type of attention that may appear in this list is one of the following: • A specific response that indicates the user is performing that action precisely. For example, it could be detected that the user says "yes," "no," or something that doesn't match, or that the user raises either their left or right hand, or neither. A response type in which the response has some degree of match to the type of response. For example, a user might be asked to "start moving," and the target attention type is moving away from the user. In this case, detecting that the user is moving around in small steps would be a strong match. Another example could be content saying "Are you ready?", to which the content is asking for an affirmative response. There could be many valid user responses that suggest affirmation, such as "Yes," "Of course," "Let's do this," or nodding. • Emotional responses are detected when a user's emotions or a subset of those emotions are detected. For example, a content provider might want to know how consumers felt about their latest release. They decide to add emotions to a shortlist of detectable attention. Users consuming content initiate conversations about the content, and only emotions are derived as attribution to the user's emotional response. Another example involves users who only want to share emotional levels on the uplift dimension. If a user is disgusted by the content they are watching, disgust will not be detected. However, low levels of uplift will be reflected in attention detection. • Topics of discussion. Here, ATS determines which topics arose in response to the content. For example, content creators might want to know what questions their film is raising for the audience. After listing the topics of discussion as attention-listed options, they might find that people generally talk about how interesting the film is, or about global warming.

[0114] There may be more types of attention. Attention lists can be provided by a wide range of different providers, such as device manufacturers, users, and content creators. If many discoverable attention lists are available, any way of combining these lists can be used to determine the resulting list, such as user lists only, the union of all lists, or the intersection of user lists and content provider lists.

[0115] The discoverable attention list may also provide users with a certain level of privacy, allowing them to provide their own list of what they wish to be discoverable and their own rules about how their list is combined with external parties. For example, users may be offered staged options (e.g., via a graphical user interface (GUI) and / or one or more audio prompts) about what can be discovered of them, allowing them to choose to have only emotional and specific responses discovered. This makes users feel comfortable using ATS-enabled devices.

[0116] A list of detectable alerts can reach a user's device in several ways. Two examples include: • A list of detectable alerts is provided to the device in the content metadata stream. There is a list of detectable alerts pre-installed on the user's ATS-enabled device, which may be applicable to a wide range of content and user alerts. In some cases, the user may be able to select from these detectable alerts, as described above.

[0117] A list of discoverable attentions associated with a particular segment of content may be learned from users so that their ATS-enabled devices can detect a larger set of attentions from them. In this way, content providers can discover how users pay attention to content and then add these attention types to a list of users they wish to detect who have a more constrained set of discoverable attentions. In some examples, there may be an upstream connection alongside the content stream that allows this learned metadata to be sent to the cloud for aggregation. This is briefly illustrated with reference to content analytics and user affinity block 203 in Figure 2.

[0118] The option to have a list of detectable warnings is applicable to all use cases listed in the "List of Use Cases" section.

[0119] Exemplary Use Cases Long-term personalization We use "personalization" to mean a way of adapting an experience to a user's preferences based on long-term user analysis. Personalized adaptations may be tested for their effectiveness by testing how the adaptation changes the user's experience using an ATS-enabled device. This forms a closed loop where the personalized adaptation can be improved and continue to track user preferences. In some alternatives, personalized adaptations may be applied in an open-loop manner where the effects of the adaptation are not measured. The changes applied to the experience do not necessarily follow user preferences in order to provide natural diversity and avoid creating isolated bubbles for the user, such as political isolation.

[0120] Determining user preferences Assume that one or more users consume a range of content on a playback device with an ATS (Article-to-Show) interface. In some cases, each user's content-related preferences can be determined over time by aggregating results from the ATS. The preferences that can be tracked can vary considerably and may include content type, actors, themes, effects, topics, locations, etc. User preferences can be determined in the cloud or on the user's ATS-enabled device. In some cases, the terms “user preference,” “interest,” and “affinity” may be used interchangeably.

[0121] Short-term estimates of what users are interested in can be established before long-term aggregates of user preferences become available. Such short-term estimates can be made through recent attentional information and hypothesis testing using attentional feedback loops.

[0122] Personalize content based on user preferences. Personalized adaptations to content may be based on user preferences. In some examples, if the personalized adaptation is informed by the preferences of multiple users, the adaptation may be optimized to take all users into consideration. Some exemplary methods of making personalized adaptations based on the preferences of multiple users include calculating one or more average attention association values, determining the maximum attention association value for a group, and determining learned combinations of preferences (e.g., via a neural network). Attention association values ​​may correspond to user preferences that include, but are not limited to, given / previously known user preferences.

[0123] Personalized adaptations (e.g., alternative scenes) may, in some cases, be delivered in the content stream as various playback options—for example, as user-selectable playback options. Alternatively, only a personalized version of the content may be streamed to the user. Another option is to generate the personalized adaptations via neural networks or the like. Depending on the specific implementation, the generation of the adaptations may occur on the user's ATS-enabled device or in the cloud.

[0124] Various forms of personalized adaptation are described in detail in the following sections.

[0125] Change the storyline of linear content based on user preferences. An example of an extension of "personalizing linear content based on user preferences" is when the content's storyline is adapted to satisfy user preferences. The storyline may be adapted by replacing, adding, or removing sections within the content. The content's storyline may also be adapted to suit the preferences of multiple users. This can be useful, for example, when many people are watching the content presentation together in the same viewing session. In some such examples, the storyline may be optimized to improve the overall enjoyment for the group by using a blend of those people's preferences.

[0126] Specific example: Bob has a hard time enjoying gory content and tends to be disgusted by it. He is watching a movie set on a battlefield, and it includes a scene where a soldier's leg is amputated. For Bob, the gory visuals of the scene are replaced with a shorter scene that only shows the face of the person performing the amputation. Alice is known to be a fan of Stephen Curry, as her gaze during the game was primarily focused on him, as detected by an ATS-enabled device with a camera. During the upcoming film, she will see the content personalized using generative technology so that Stephen Curry makes a cameo appearance. John has a short attention span and therefore prefers shorter content. This is detected by their ATS-enabled device, and in the next movie, he sees long, drawn-out scenes being cut into shorter sequences. Two parents and one child are watching a movie together. The child is easily frightened, so jump scares are removed from the movie. The parents enjoy movies without happy endings, so the usual happy endings for movies are left unresolved.

[0127] Generative augmentation of content based on user preferences Another example of an extension of "personalizing content based on user preferences" is when content is augmented with generated elements. Generated elements refer to machine-generated media and can include video, audio, lighting, or a combination thereof. "Augmentation" is used to mean changing one aspect of the content (e.g., objects in a scene, instruments that produce sound, lighting in a scene, narrator's voice). Some examples include superimposing generated images to replace objects in a scene (e.g., changing an apple to an orange) or changing sounds from one type to another (e.g., changing strings to a horn).

[0128] Specific example: John is watching the new movie "Top Gun: Maverick," in which Tom Cruise plays Colonel Pete Mitchell. John is a huge fan of Brad Pitt and would much rather see Pitt play Mitchell. Using generative machine learning, Brad Pitt's face is superimposed over Tom Cruise's face for the entire movie. Tom Cruise's voice is also replaced using generative techniques to make it sound like Brad Pitt's. From John's perspective, Mitchell is now played by Pete, not Cruise.

[0129] Personalized selection of linear content options based on user preferences Another extension of "personalizing linear content based on user preferences" is when the adaptation is a selection of personalized options for a content stream. Personalized options could include selecting a preferred commentator, or selecting a stream specifically adapted for one of two teams in a sport.

[0130] Specific example: John is watching the Super Bowl on his ATS-enabled device, and the device uses its camera to detect that John is wearing a red hat. The system (e.g., AAE 201) infers that he must be cheering for the Chiefs (the red team) and selects a version of the stream adapted for Chiefs fans. In some cases, John may be presented with additional replays of tries scored by the Chiefs and fewer replays for the opposing team. In the same scenario as above, John is not wearing a red hat, so the ATS initially doesn't know which team he supports. As the game progresses, John begins to get involved by cheering "Go, Chiefs," booing the opposing team, and cheering for points scored by the Chiefs. This information is detected by the ATS, and the appropriate stream is selected for John. In the same scenario as above, John is not wearing the red hat and does not comment on the game. John's user preference from the previous session, where he favored the Chiefs, is drawn out, and therefore the Chiefs version of the stream is selected for John. In the same scenarios listed above, the version of the stream selected may also include the choice of commentators for John's team. John and Alice are watching the Super Bowl together on an ATS-enabled device equipped with lights and cameras available to the system. It (for example, AAE 201) infers that John is a Chiefs fan because he is wearing a red hat, while Alice is a Lions fan because she is wearing blue and cheering for the Lions' goal. John and Alice are provided with balanced versions of the content presentation. In some examples, the lights may be red on the side of the room where John is sitting and blue on the opposite side of Alice.

[0131] Personalization of accents appearing in content based on user preferences Another extension of "personalizing content based on user preferences" is when the accent of voices within the content is changed to suit the user's preferences.

[0132] Specific example: Bob is listening to an audiobook playing on his ATS device. The device has detected that Bob is speaking with a British accent. While audiobooks are played with a US accent by default, the UK version of the audiobook is selected for Bob to make it more familiar and natural for him to listen to. Jane is watching a movie starring an Australian. She finds it difficult to understand the accent, and ATS detects that Jane has a French-influenced English accent. The Australian voice is generatively swapped out due to the French-influenced English accent. In the same scenario as above, Jane's preference for a French-style English accent is determined, and therefore, the voice assistant on Jane's ATS-enabled smart device changes to a French-style English accent.

[0133] Add personalized experience enhancements based on user preferences. Another extension of "personalizing content based on user preferences" is when the personalized adaptation is a preloaded experience augmentation. Examples of experience augmentation are described in detail under the following heading. In some examples, preloaded experience augmentation may be determined based on previous sessions in which users with similar preferences enjoyed the same experience augmentation. Such implementations can enhance the user experience and help cover cases where the ATS fails to detect attentional responses.

[0134] Specific example: Alice is watching a comedy on her ATS-enabled device. There is a joke in the comedy that Alice did not find funny. However, users with similar tastes to Alice typically find this joke funny. When the system receives a positive response from a user with similar tastes to Alice regarding this joke, a virtual laugh is played.

[0135] Adaptation of playback and experience based on user habits and characteristics Here, the term "user habits" refers to the everyday ways in which users engage with content (positively or negatively). This can include useful insights into how playback and experience can be optimized for the user. User habits can be determined by detecting things like which seat a user sits in when engaging with content using ATS, or what time of night a user starts sighing in annoyance when there is a noisy scene in the content.

[0136] The term "user characteristics" is used to refer to user attributes that can influence how a user experiences content, such as their hearing, vision, and attention span. User characteristics can be estimated by the ATS based on how users interact with the content, as follows: Missing a joke, especially one told quietly, suggests poor hearing and / or • When an actor only uses colors to describe something, it's difficult to follow what the actor is referring to. This suggests the possibility of color blindness.

[0137] User habits and characteristics can also be considered a form of user preference.

[0138] The term "playback adaptation" refers to changing the presentation of content. Playback adaptation can include changing the spatial rendering of content, adjusting the volume, and changing the color balance on the screen.

[0139] In some examples, adaptation to the user experience may be intended to ensure that the user's environment is in a desirable state for enjoying the content. Some such examples may include features such as automatically pausing content when there is an interruption in playback, or warning the user of things they should evaluate before starting the content. Playback and adaptation to the experience may be managed on the user's playback device(s).

[0140] In some cases, adaptations may be collectively selected to simultaneously improve the experience for multiple users of an ATS-enabled system. The following sections detail specific examples of user habits, user characteristics, and the resulting adaptations to replay and experience.

[0141] Personalized accessibility changes to content playback based on user characteristics. Another extension of "adapting playback and experience based on user habits and characteristics" is when playback adaptation is performed to improve content accessibility based on user characteristics. User characteristics that may be optimized for this may include nearsightedness, color blindness, hearing loss, dizziness, epilepsy, etc. Playback adaptations that may be applied to the mentioned characteristics to improve content accessibility may include increasing the font size of subtitles, enhancing the color or appearance of objects in a scene, applying compression to the audio of the stream, reducing camera shake, skipping scenes with flashing lights, etc.

[0142] Specific example: Jake watches his favorite comedians on an ATS-enabled device and laughs at almost every joke the comedians tell. The only jokes Jake doesn't laugh at are, most frequently, those that are told quietly. In some cases, dialogue enhancement may be enabled to improve the audibility of these jokes.

[0143] Personalized car notifications based on user habits Another extension of "adapting playback and experiences based on user habits and characteristics" is when the in-car experience is improved based on seat occupancy. Seat occupancy can be determined by sensors that show where people are located in the vehicle when engaging with content. In some cases, the ATS system can be configured to detect when there is an unusual seat occupancy. The improved in-car experience can come in the form of user warnings or notifications. These warnings or notifications can let the user know that they may have forgotten who normally goes on this drive. This improves the driving experience because the user is comfortable knowing that the usual participants in the ride have not been forgotten.

[0144] Specific example: Every weekend, the user drives their in-laws to dinner. The user starts backing up from the driveway to their in-laws' house, but their mother-in-law is not in the car. The car notifies the user of their mother-in-law's absence, and the crisis is averted.

[0145] Short-term experience enhancement Dynamically add elements to content based on user reactions. Assume that one or more users are consuming content on a playback device with an ATS. The ATS may be configured to detect the class of responses (e.g., laugh, cheer, shout "yes") according to metadata within the content stream. Additional elements may enhance the experience for one or more users in real time.

[0146] In some examples, there may be a fixed set of user sound events (e.g., cough, laugh, applause) or keywords (e.g., "bang," "hmm," "higher," "lower") specified in the metadata, along with details of the appropriate element to which it responds. In some such examples, there may also be a set of emotions (e.g., excitement, fear) associated with the appropriate action. In some examples, the element may also be delivered in the metadata stream alongside the content to enable such functionality.

[0147] In some cases, there may also be a library of elements stored locally—for example, on a TV or another playback device—which could be applicable to many content streams. In such cases, the library of elements may be more broadly applicable than elements specific to a particular content presentation. Alternatively or additionally, generative AI techniques may be applied to automatically generate elements, for example, based on a textual description of the content presentation.

[0148] According to some examples, the intensity of a response (e.g., the volume of a sound played, the strength of an audio effect, the size of an icon, the saturation of an icon, the opacity of an icon, the intensity of a visual effect) may be proportional to the intensity of a user reaction (e.g., the loudness of a shout, the length of laughter, the pitch of a song).

[0149] The following sections highlight examples of the types of elements that may be added.

[0150] Dynamically add auditory elements to content based on user reactions. Another extension of "dynamically adding elements to content based on reactions" is when the added elements are audibly played. In some examples, whenever one of the users produces a particular response, a corresponding audible element is added to the content. Audible elements can include playing sounds or adding audio effects. Examples of sounds that can be played include crowd noise, comedic sounds, impact hits, etc. Audio effects can be anything from reverberation, distortion, spatial movement, etc. In some examples, auditory elements can be these elements or a combination of similar elements. Sounds and audio effects can include the use of Dolby ATMOS® or other types of object-based audio systems. An example of sound using ATMOS® is playing a virtual laugh track by placing different laughs in different spatial locations. Another example of spatial audio effects could include hearing baseball move around the user after the batter hits the ball.

[0151] Specific example: Multiple users are watching a stream of cat videos. When a user's laughter is detected, a virtual laughter track is played through the speaker. This virtual laughter track can be created by placing different laughs in different spatial positions. During a baseball game, when the user's team hits the ball, a spatial "whoosh" sound and audio effect are added to the content. The spatial movement of the sound makes it sound as if the ball is flying past the user. The ball hit may also be encoded in the metadata provided with the content stream.

[0152] Dynamically add other users' auditory elements to the content based on their reactions. Another extension of "dynamically adding auditory elements to content based on reactions" is when the added auditory elements are supplied by other remote users of one or more other ATS systems. Such an example might allow a user in the local environment to hear how other users reacted to content when they consume it. For example, when a user (user A) reacts to content, that reaction may be recorded and sent to user A's friends (including user B). Later, when user B comes to see the same content, user A's reaction may be sent to user B's playback device through the content's metadata stream. User A's reaction may be played back for user B at the same point in the content where user A reacted. In some examples, the reactions played back for user B may include reactions from many users played back simultaneously. The reactions played back for user B do not necessarily have to be from user B's friends.

[0153] Specific example: Jane is watching a football game between Liverpool and Manchester. She cheers for every goal scored for Liverpool and boos for several goals scored by Manchester. Jane's friend Bob later watches the Liverpool vs. Manchester match. When Bob watches the game, he gets to experience watching the game along with Jane's cheering and booing. TV programs are often watched by millions of people. When a program is watched by a user, their reactions can be recorded and then sent to the cloud, where these reactions are mixed together to form a crowd reaction to the TV program. When another user watches the program, the crowd reaction audio can be played along with the content.

[0154] Dynamically add visual elements to content based on user reactions. Another extension of "dynamically adding elements to content based on responses" is when the added elements appear visually. Whenever one of the users produces a selected type of response, a visual element may be superimposed or composited on the display along with the content. The visual element may be an icon, animation, emoji, etc. Alternatively or additionally, the visual element may be a visual effect added to the content (e.g., vibration, color change).

[0155] Specific example: Jane is watching the Academy Awards on television on her ATS-enabled TV. While introducing the Best Actor in the film category, the show's host asks viewers at home to shout out the name of their favorite actor. Jane is a fan of Tom Cruise and enjoyed "Top Gun: Maverick" this year, so she shouts out "Tom Cruise." In response, Jane's TV displays an overlay image of a virtual Oscar statuette being held in mid-air by a cartoonish representation of Tom Cruise. Meanwhile, her neighbor Ben, a fan of Brad Pitt, sees an overlay image of a virtual Oscar statuette being held in mid-air by Brad Pitt. Injuries are common in ice hockey games. When watching an ice hockey game, an ATS-enabled TV can be configured to display an ambulance animation whenever it hears the user say "Call an ambulance." ·In Batman around the 1960s, whenever a character was hit in a fight scene, words like "Bang!", "Thud!", and "Bang!" were routinely superimposed on the action in a comic book font. Batman 2026, viewed on an ATS-enabled TV, could also display similar words based on user utterances during fight scenes. For example, if Penguin is hit by Robin and the user shouts "Bang!", the ATS-enabled TV could then be configured to overlay the text "Bang!" instead of the other available options "Bang!" and "Thud!".

[0156] Dynamically add lighting elements to content based on user reactions. Another extension of "dynamically adding elements to content based on reactions" is when the added elements are changes to the lighting. In some examples, changes to the lighting may be issued whenever the user interacts with the content in a predetermined way. These lighting changes may include bulk changes to the lighting or may trigger animations that use the lighting. Bulk changes may include changing the color and intensity of all lights. In some such examples, each light may change its own independent color and intensity. In some examples, animations may be issued to play on those lights in response to user reactions. Some exemplary animations are waves, strobes, and flickering flames.

[0157] Specific example: The user is watching a horror movie and verbally indicates that they are very scared by it. In response, the brightness of the lights is increased, reducing the sense of fear induced by the content. The family is watching a rugby game where the team colors are red and blue. The ATS detects that the family is responding positively to the red team, and the room lights undergo a bulk change to red.

[0158] Enhance content to improve comprehension based on user feedback. Assume that one or more users are viewing content on a playback device associated with ATS. Whenever one or more users engage loudly with the content (e.g., laugh, cheer), in some examples the content may be adjusted to improve the intelligibility of what might otherwise be missed by those users. Intelligibility may be improved by delaying the next scene until the user's reaction subsides, increasing the volume of dialogue, or turning on captions so that the next scene is still understandable. In some examples the content stream may include additional media to optionally extend scenes to achieve a time delay. Without such additional media, the time delay can be achieved by pausing the content until the intensity of the user's response decreases. Alternatively, generative AI may be used to generate additional media to stretch out scenes for as long as necessary to prevent the content from progressing until the user is ready to move on. In some examples the increase in dialogue volume may be proportional to the intensity of the user's reaction.

[0159] Specific example: Bob is watching a comedy with Jane. Bob finds the punchline in the scene hilarious and is laughing for a long time. The next scene in the comedy may be delayed until Bob can control his laughter to a certain level. • The family is watching the Super Bowl, and they are cheering loudly for extended periods about touchdowns scored by their team. During the period the user is involved, closed captions for the commentators are automatically turned on. This allows the user to follow what the commentators are saying. • The user is watching a stand-up comedy show and laughs for an extended period at a joke told by the comedian. The system increases the volume of the comedian's voice while the user continues to laugh. • Users are having difficulty understanding the dialogue in the movie they are watching. A dialogue enhancement feature can be turned on to improve the audibility of speech within the room. • In the same scenario as above, closed captions are turned on instead of improved dialogue.

[0160] Personalization through feedback to content creators This section shares similarities with long-term personalization and short-term experience augmentation, except that personalization is performed by content creators based on user attention metrics detected by one or more ATS systems. Attention feedback can occur either when a user explicitly responds to a content provider's request or when they implicitly pay attention to the content.

[0161] Examples of requesting and responding to explicit feedback include the following: The presenter says, "Turn on the sound!" and the audience either turns on the sound or remains silent. The presenter can ask, "What do you think, viewers at home?" and viewers can respond with emotions such as booing or cheering. The presenter asks, "Would you like to select the red object or the blue object?" and the audience responds with a set of "red" or "blue" responses, or with unmatched responses.

[0162] Examples of users who implicitly pay attention to content may include all types of attention referred to in this disclosure that do not respond to explicit requests within the content.

[0163] Personalization decisions may be made by the content creator or by an automated system implemented by the content creator. In some cases, personalization may involve personalizing content for the average audience. In some cases, personalization may involve generating multiple versions of the personalization adapted to the preferences of multiple audience categories (such as users in a particular country or region, or users known to have similar interests or preferences). In some cases, user attention responses may be sent to the service providing the content (e.g., the same server or data center) or to different services.

[0164] The next iteration process of content creation by creators based on feedback: personalization. Content creators provide content that consumers can view at any time. Content creators may also be provided with information on how users in an ATS-enabled environment engaged with their content. In some cases, this information can be used by content creators to provide a personalized twist to the next iteration of the content (e.g., episodes, albums, stories).

[0165] Specific example: The influencer receives attention metrics for their previous short video and finds that users are paying close attention and want to see more products similar to those they displayed. The influencer then decides to present similar products for their audience in their next short. A vlogger who creates a series of videos about video games asks viewers at the end of their videos, "Which character should I play next?" Viewers indicate which character they would like to see the vlogger play next (for example, by voice, pointing). The vlogger then takes these responses into consideration when deciding what to play next.

[0166] A TV show ends with an unresolved, suspenseful situation. Based on ATS user responses, there were four main types of reactions to the ending. The TV show producers use this information to decide which of the following four versions of the next episode to create. Users are shown which version of the episode is intended for them, based on how they reacted to the previous episode.

[0167] Response-based interactive live streaming media This section describes examples of what can be achieved when user attention information is sent back to content creators in real time. Based on this analysis, content creators can decide to adapt their content in real time.

[0168] In some cases, ATS-enabled systems may offer the option for other users to see how viewers are responding. For example, content creators can provide attentional information within their content. Alternatively, real-time attentional information may be sent back to the user in the content's metadata stream. Seeing how other users are responding in a particular way may increase the likelihood that a particular user will respond in the same way. Similarly, seeing where others are not responding may make a user realize that they have a greater influence on content presentation by responding in a certain way (e.g., cheering for a character).

[0169] Interactive live-streaming games based on reactions Other extensions of "responsive, interactive live streaming media" may include content that is or contains game content. Game content may involve individual users playing games, or it may involve competitive gaming, etc.

[0170] Specific example: Jane is live-streaming herself playing a video game. In the game, she is presented with the option to go either left or right. Jane asks her viewers to call out which way she should go, "left" or "right". The viewers' attention is sent back to Jane, showing that 70% want her to go "left". Jane decides to go right to confuse her viewers. Bob is hosting a live stream playing a trivia video game. He asks viewers which of several options is the correct answer. The answer chosen most often by viewers turns out to be incorrect. Bob then enhances the stream with camera shake, saying, "Since the answer was wrong, you have to go through camera shake together!"

[0171] Interactive live streaming music-related content based on user reactions. Another extension of "response-based interactive live streaming media" is when the media is music-related content, such as bands, disc jockeys (DJs), live music events, or one or more musicians playing various instruments while talking. Some attention responses that may be used for live music streams include clapping, singing along, dancing, and moving to the music. Attention responses, such as asking users to sing the most memorable line from a song, may also be explicitly requested by the content creator.

[0172] Specific example: The DJ is performing a show that includes participants joining in remotely using their own compatible devices. The DJ slowly builds up to more energetic tracks. Attention metrics show the DJ that the audience is already dancing more than usual at this point in the show. Based on this information, the DJ decides to move on to more energetic tracks earlier than originally planned. A musician is hosting a live stream from their own studio. While playing a guitar solo, the musician can see the reported low level of attention to the guitar solo. In addition, some viewers respond by saying "bongo." The musician decides to start playing bongo drums instead.

[0173] Interactive, user-generated live streaming content based on user reactions Another extension of "response-based interactive live streaming media" is when the media is user-generated content.

[0174] Specific example: A vlogger is hosting their live stream in a foreign country. The vlogger enters a store to buy snacks and shows the selection of snacks on the shelves. The vlogger asks viewers what they should buy. The vlogger uses cautionary information to help personalize the experience for viewers and decide what to buy.

[0175] A reaction-based interactive live streaming show Another extension of "response-based interactive live streaming media" is when the media is a program from a broadcaster (e.g., reality TV shows, episodic TV shows, award shows, live movies).

[0176] Specific example: During the People's Choice Awards program, users can vote for the best movie through their ATS-enabled devices. The presenter will ask, "What movie do you think deserves to be called Best Movie?" and users will shout out the name of the movie they want to win. The presenter will then announce the winner based on the votes cast using attention metrics. • A new interactive Jackass® movie is being filmed and broadcast live. It allows users of ATS-enabled devices to vote on who should wear a cape and jump off a ramp on a quad bike. In the Eurovision® finals, the winner is determined based on real-time alerts from ATS-enabled devices. During a Saturday Night Live (trademark) presentation, the producers are interested in knowing how people are paying attention to the content in real time. The producers discover that people find a joke the host just told particularly funny. From there, the producers encourage the studio audience to laugh louder (for example, via a sign visible to the audience but not to television viewers), and the host is told through their earpiece to hold onto the joke a little longer with the guest.

[0177] Response-based interactive live streaming media viewing sessions Another extension of "response-based interactive live streaming media" is when content presentation is a viewing session of a program, which may be a replay. Multiple participants watch the viewing session of a program through many ATS-enabled devices, and in some cases simultaneously. This viewing session may be hosted by a human producer or by an automated system. Some examples of suitable viewing session programs include "choose your own adventure" shows, betting game shows, trivia game shows, and children's television shows (such as shows that include phone calls and Q&A).

[0178] Specific example: • Hug Man has a segment in the show where he hugs celebrities. In one viewing session, multiple participants vote on how long they think Hug Man can hug a celebrity before the celebrity taps out to break the hug. A game is hosted by an automated system where viewers must find the locations of items hidden within a scene on a given list.

[0179] Recommendation Some disclosed examples include recommending user content based on user preferences and / or current user status determined by analytics using an ATS. Recommendations may be calculated in the cloud or on the user's ATS-enabled device. Recommendations may be determined based on short-term or long-term information, or a combination of both. Recommendation systems may also be optimized in the cloud to better understand what content a larger group of people (other than people in specific environments such as a particular home) enjoys in certain situations (e.g., user mood, viewing session length, news events).

[0180] Content can be recommended before, during, or after content playback. One example of recommending content during playback is when a user is not responding positively to the currently playing content, and therefore, different content that better suits their preferences is recommended.

[0181] Mood-based content recommendations Users may be recommended content based on their mood, as detected by an ATS-enabled device. ATS can detect things like the user's laughter, sighs, emotional valence and / or arousal level in speech, and whether they react to content that would normally evoke a positive response.

[0182] In some cases, the ATS system can also leverage the closed loops that can be created as described above. For example, the ATS system may recommend content based on whether a particular type of content had a positive effect on the user the last time it was consumed when it was in the user's current mood. Some examples include exploring whether uplifting music or sad music would be more helpful in lifting this user out of a negative mood, or whether watching a sports game is more enjoyable than watching a comedy when the user is in an euphoric mood. Recommendations may also take the form of short-term mood estimations that do not take user preferences into account, so as not to form a closed loop of recommendation improvement.

[0183] Mood-based content recommendations may be determined using long-term analysis. Assume that the ATS is not yet aware of the user's mood (e.g., the user has just woken up). In some cases, the ATS may refer to long-term mood information to infer a likely mood. Such inference may take the form of finding patterns (e.g., the user is generally happy in the morning), detecting trends (e.g., the user's mood has been improving towards summer), or using filtered mood information (e.g., the user was in a good mood this week, so they are likely in a good mood now).

[0184] Specific example: Bob is in a gloomy mood, which is detected by ATS because he is not laughing at the jokes in lighthearted comedies that match his usual taste. Dark comedy is recommended to Bob as the next content. ATS detects that Bob's mood improves slightly through light chuckles. This information is stored (possibly on Bob's device or in the cloud, etc.) to help notify the recommendation system when Bob finds himself in a similar mood again. • A child is watching a television program and starts to get scared. The child is then advised to switch to a calmer program.

[0185] Content recommendations based on user focus on specific views The term "view" can refer to any visual element that represents a particular type of content. Some examples of views include thumbnails while browsing content options, different screens displaying content simultaneously (for example, a TV playing a program and a tablet playing a game), and multiple windows on a screen.

[0186] Focus on a particular view can be determined by an ATS-enabled device(s)

[0187] Specific example: As a user scrolls through a list of content options, an ATS-enabled device may report which content thumbnails the user is most engaged with. As the user continues to scroll for further options, recommendations are directed towards those the user implicitly shows interest in and pays attention to.

[0188] Content recommendations based on music content Music attention can be determined by an ATS-enabled device through detection of singing along, drumming in the air or on an object (e.g., a car steering wheel), or shouting a song or artist's name when music begins. Music attention indications can then be used to determine the user's musical preferences.

[0189] Content recommendations may be provided in the form of recommended or automatically curated playlists based on the user's musical attention metrics. Recommended content may be similar in terms of genre, era, artist, song structure, etc. Other forms of media may be recommended based on musical preferences. For example, a movie may be suggested based on the fact that it has a soundtrack that matches the user's musical taste.

[0190] Specific example: Alice is on a road trip and has a habit of tapping along to her favorite songs on the steering wheel. This information is used to determine her musical preferences and recommend a curated road trip playlist for the remainder of her journey.

[0191] Exercise use case Exercise use cases may involve long-term personalization, short-term experience enhancement, personalization through feedback to content creators, or a combination of these, sometimes simultaneously. Exercise content may include yoga classes, workout classes, and music users want to listen to while working out at home. Specific sounds that may be heard to detect exercise-related attention indicators may include the following: • To be out of breath; • Take a deep breath; • Count aloud during repetition; • Increased respiratory rate; Body swaying; • Visually detecting exercise movement; • Visually detecting the user's technical skill level; • Emotion recognition, etc.

[0192] In some cases, the ATS can provide a human or virtual coach with attention-related information in real time. This information may correspond to the attention of one or more users to their movements. Alternatively or additionally, the ATS can provide the coach with user track records, performance trends, attention information from previous sessions, and so on.

[0193] Specific example: Jane uses her ATS-enabled device when she works out every Monday. Her progress is tracked time-wise, and the class difficulty is matched to her progress. Jane is also provided with an overview of her progress on the ATS-enabled device, including a dashboard on her phone's display. Bob wants to do an exercise class that is at his level and suits his preferences. Bob watches pre-recorded exercise classes that are personalized for him by having video "tracks" selected for him to suit his needs. Their "tracks" may refer to different sections of pre-recorded videos, so that those preferred exercises may be selected for the user in the order most desirable for the user. • Multiple users, for example, two users, decide to take an exercise class together. They have different preferences regarding exercise movements and different levels of ability in areas such as flexibility, strength, and aerobics. They provide input to the ATS indicating that they want to complete a core workout, and the ATS automatically generates an exercise class for them that is adapted to their abilities and preferences. • The exercise class is hosted by a virtual coach. The ATS detects that the user is having difficulty completing the final repetitions due to an asthma problem. The ATS instructs the virtual coach not to encourage the user to complete the repetitions. • An exercise class hosted by a virtual coach has a participant who is not actively pushing themselves to complete the set of exercises. This is detected by an ATS-enabled device, and the virtual coach encourages that participant to work harder and push themselves further. • Live exercise classes are hosted by coaches with many participants across multiple ATS-enabled devices. The coach is informed that John is not keeping up with the current exercise. However, the coach is also provided with information on how John has done in previous sessions. The coach offers encouragement, for example, by saying, "Come on John, you did it last week. You can do it!" • In another live workout class hosted by a coach, ATS detects that one of the participants has particularly poor form during the current workout. Based on the feedback from ATS, the coach decides to speak directly with that participant through the user's content stream to offer some tips on how to perform that workout better.

[0194] Regeneration optimization and control use cases This section provides examples of long-term personalization and short-term experience enhancement. “Playback optimization” in this section refers to optimizing playback across controlled devices based on an objective function, in some examples. This objective function may include, for example, maximizing attention, maximizing intelligibility, or maximizing spatial quality with respect to listening position. Updates necessary to achieve playback optimization may be calculated on the user’s ATS-enabled device or in the cloud and potentially transmitted to the user’s device within a metadata stream provided with the content. “Playback control” refers to functions such as play, pause, rewind, and next episode.

[0195] Specific example: Six users are in an ATS-enabled car playing music. Two users in the back row are having a conversation. They are not paying attention to the music content, but rather to their discussion. The music playback is drowned out in the back row of the car, but kept louder for the other users who are enjoying the song. Alice prefers to watch TV in the living room adjacent to the kitchen while Bob eats dinner. Bob enjoys listening to music while preparing dinner. Playback is jointly optimized for Alice and Bob so that both hear spatial audio centered on their respective areas of the room. Furthermore, audio from Alice's content is much quieter in Bob's zone, and vice versa. Many users were watching a movie in the room, but then everyone left to greet a visitor. The ATS (Activity Tracking System) detected that no one was in the room (for example, based on camera data), and the attention level for the content decreased significantly. The ATS paused the movie. Bob is watching a movie when he receives a call. An ATS-enabled device detects that Bob is not paying attention to the content because his posture indicates that he is holding the phone to his head. ATS pauses the movie for Bob while he answers the phone. In the same scenario as above, Bob has shown on previous occasions that he prefers to play content while on the phone by resuming it after an automatic pause. The system notices that Bob has started making another phone call while a movie is in progress and, based on Bob's learned preference, does not pause the content for him.

[0196] Advertisements using attention feedback use cases Previously deployed advertising systems have limited ways of determining user attention. Some current methods include having users click on ads, giving users the option to skip ads, and having trial viewers fill out questionnaires about the ads. With such methods, advertisers do not necessarily know whether a user is even present when the ad is presented.

[0197] Using an ATS enables better-informed advertising. Better-informed advertising can improve current techniques such as ad performance, ad optimization, audience sentiment analysis, user interest tracking, and informed ad placement. Furthermore, better-informed advertising opens up new advertising opportunities, including the following: Interactive advertising; • Personalized advertising; • Attention-driven shopping; and • Motivating attention for improved advertising.

[0198] These ideas, along with a list of use cases, will be explored in detail throughout this section.

[0199] Exemplary Use Cases Ad analytics based on user attention When users consume content on an ATS-enabled device, the advertisements played on the system can also leverage the ATS. Extensive ad analytics can be derived from the ATS. Some examples of analytics that can be provided by an ATS include: • Overall user engagement; • Which aspects of the advertisement did the user engage with? • Did the user engage positively or negatively? • Who was involved in the advertising (for example, based on demographics, user preferences, and interests); • When did the user interact with the advertisement in a specific way (e.g., morning / evening, Monday / Tuesday, etc.)? • Whether the particular actor in the aforementioned advertisement attracted more attention or less attention; • Which sections of the content trigger higher or lower attention (e.g., advertisements at the beginning, middle, or end of the content; or after a suspenseful situation, joke, or romantic scene).

[0200] Specific example: The marketing team for a new 3D-printed basketball wants to know the performance of their latest ads to determine if they're getting their money's worth. The marketing team launched the ads on ATS-enabled devices and now receives an analysis of the ad performance. Following the 2023 Super Bowl, a ranking of the best advertisements from the event is desired for the benefit of public novelty. Since the Super Bowl was broadcast across ATS-enabled devices, analysis of each advertisement will be collected and compared to determine the ranking.

[0201] Optimize ads based on user attention. Advertisements can be optimized for desired attentional responses using "ad analytics based on user attention." Such optimization can be achieved through sequentially and iteratively improving the advertisement after the release of new versions based on the analytics. Alternatively or additionally, such optimization may be achieved by simultaneously releasing multiple versions of the advertisement to see how user responses differ.

[0202] Specific example: A company wants to optimize its ads for maximum engagement before fully launching a marketing campaign. The company decides to conduct a soft release of the ad to 1000 ATS users. Using the analysis from this soft release, the company creates another version of the ad and conducts another soft release, repeating this process until they are satisfied with the ad. Finally, the company confidently launches the advertising campaign. • In the same scenario as above, instead of a soft release of content, there are sequential iterations of focus group testing. The focus groups view the content using ATS-enabled devices. A company has many great ideas for bringing a new product to market. They decide to create many versions of their advertisement and release them simultaneously. Using analysis from ATS-enabled devices, they find that one version of the advertisement is performing significantly better than the others. They decide to continue advertising using only the best-performing advertisement.

[0203] Product sentiment analysis based on user attention Product sentiment can be determined using users' ATS-enabled devices during content and advertising playback. This sentiment may be evaluated on a user or population basis. For example, a population could be users within a certain age group, users who share an interest (e.g., cars), users with a certain attentional characteristic (e.g., users who laugh in a typical frequency range for adult women), a learned segment of a population (e.g., users who sing in a certain frequency range with a specific tone), or the entirety of all ATS users.

[0204] Demographics and user interests are a subset of factors that can be used to select a population for evaluating sentiment analysis. Demographics and user interests may be specified by the user, estimated by the ATS, or both. User interests may be estimated, for example, in the manner specified in the "Determining User Preferences" section. Demographics can be estimated from cues such as the characteristics of their responses (e.g., the frequency range of their laughter, the height of their jumps, their visual appearance indicating age), the type of attentional responses they use (e.g., the words they use and the type of dance movements), etc.

[0205] Sentiment can be determined by how the user responds to the product, either implicitly or explicitly. Implicit responses can include yawning, staring at the product when it appears on the screen, etc. Explicit responses can include positive responses such as saying "I love <product name>", or negative responses such as leaving the room after saying "Another <product name>?"

[0206] Specific examples: · The latest James Bond movie features James wearing an Acme watch. Acme wants to know how the watch is perceived during the placement of this product and is provided with sentiment analysis as determined by the ATS for that watch on a population scale. Acme decides to enter into the same type of product placement agreement for the next James Bond movie. · Expanding on the last example, before the release of the movie, Acme had already run an advertising campaign on TV. The sentiment of the watch in the TV ads is captured using ATS-enabled devices after the James Bond movie is released. As the movie gains more viewers, Acme finds higher attention and sentiment towards its watch in the TV ads. ·A company is trying to expand the age demographics of its product. They want to know how people aged 40 - 50 perceive the product. They determined the sentiment towards the product for ages 40 - 50 using a mix of attention analysis from self - reported users within that group and users who did not provide their age but were estimated to be within that demographic by ATS. From this data, the company was given a large amount of sentiment analysis information from the ads it launched, which was available on ATS - compatible devices. ·A company is trying to target its product to introverted people. The company has hypothesized that introverted people respond more quietly compared to other groups of people. The company decides to filter the sentiment analysis of its product to people who do not engage loudly and do not react with large movements.

[0207] Interactive advertising using ATS Advertisements can be interacted with using an ATS - compatible device. The interactive components of an advertisement may be sent through the advertisement's metadata stream, which includes information such as which types of attention to respond to and the respective actions when the advertisement should respond. The types of actions when an advertisement can respond can include actions such as changing the content of the advertisement, activating control mechanisms such as skipping the advertisement, and storing user responses (e.g., within the user's device or in the cloud) for when an advertisement of the same product is played next. These actions can help determine whether the user is engaged and whether to allow gamification of the advertisement. For example, since the user displayed a level of engagement with the device, they may be given a reward to skip the advertisement.

[0208] Specific example: ABCD Inc. has launched a marketing campaign for its new car. They have decided to place their ads on ATS-enabled devices to allow for an interactive experience. Their ads feature a car racing game. The ads allow users to control the vehicle by leaning left or right, or by shouting "left" or "right." Users are more responsive to and more aware of the new car than other cars advertised in a traditional way. Insurance companies want to show how minor accidents can happen at any point in life, and how having insurance makes minor accidents less serious. To achieve this, insurance companies are launching "Choose Your Own Adventure" ads on ATS-enabled devices. Each time an ad appears, users can choose a new path for that story, resulting in a consistent conclusion that "you should get insurance." The company releases a new game and, along with it, releases advertisements to engage people in the release and get them excited about it. These advertisements are hosted on ATS-enabled devices and involve multiple ad placements to conclude a series of ads that resemble the game's features. Each time an ad from this campaign is played, the user is given a mechanism to personalize the ad, such as customizing the appearance of the main character. The next time an ad from this campaign is played, the character will have the customized appearance and the story will progress with any other dialogue the user has previously had. • Users are consuming content in which ad interruptions are embedded. Users are rewarded for not skipping ads and actively engaging with them (e.g., making strong eye contact or discussing the ad). Examples of rewards may include not receiving another ad interruption for the remaining 30 minutes of content playback.

[0209] Personalized advertising based on user preferences Advertisements can be personalized in a similar way to personalized content adaptations detailed in the Long-Term Personalization section. As mentioned in the Long-Term Personalization section, adaptations may be shipped in the metadata stream, sent directly as selected versions, or generated generatively. In some cases, advertisements may simply be selected to match the user's interests.

[0210] Specific example: • Two users watching TV together are known to enjoy comedy, as detected by ATS-enabled devices, and therefore, a humorous version of the advertisement is shown. Conversely, users in different households prefer the content up to that point, and therefore, a more serious version is shown. Users watching basketball on ATS-compatible devices are known to be fans of Stephen Curry because they respond positively to him. Advertisements for tickets to the next game will be personalized by selecting a version featuring Stephen Curry. In the same scenario as above, the user never mentions Stephen Curry, but the ATS recognizes that Stephen Curry is the user's favorite player because the user's gaze is always following Stephen Curry on the court. Jane and Bob are watching a show about world travel. They discuss how much they want to visit Greece, and this topic is detected by their ATS-enabled device playing the show. In the next ad break of the show, Jane and Bob are shown an advertisement about a Greek resort. While Bob is watching the latest James Bond movie, he says, "I really like his suits," indicating his interest in suits related to James Bond. During the next ad break, Bob is shown ads for suits with a similar appearance. As mentioned above, this example could be improved by asking if there are any pre-populated words ("suit" or "ABCD") that are identified along with the content. In a scenario similar to the one described above, the user does not mention their preference for the suit. Instead, a camera connected to the ATS allows the system to determine that the user has a special interest in James Bond's watch. For this reason, the user is presented with an advertisement for that watch during the next ad break.

[0211] Attention-driven shopping Many shopping-related opportunities can be made possible through the use of ATS-enabled devices. This can span many types of content. Some examples include shopping-related content channels and virtual assistants. New shopping-related opportunities may include: • Similar to what is detailed in "Advertising Analysis Based on User Attention," but especially for product sentiment analysis for shopping-related content (e.g., online shops, shopping TV channels); • Direct interaction with shopping materials through ATS-compatible devices; • Optimize the shopping experience, products, and offers presented using user preferences and interests; Further attention-driven shopping opportunities may also be possible.

[0212] Specific example: While a user is browsing an online shop, eye tracking or the user verbally saying "Oh, I like that" indicates their interest. The item is then automatically added to the cart. • The interactive ATS-enabled shopping experience explicitly tells viewers, "If you like this product today, give us a round of applause!" A subset of ATS-enabled viewers who engage with the product will receive it for free. The remaining users who did not receive the product will have the item added to their cart and will be notified of the updated price. Shopping TV channels utilize attention analytics to determine which products users are most interested in. The user's device also recognizes their emotional connection to the product. Shopping TV channels use attention analytics to determine which aspects of an advertised product (e.g., features, price) contribute to converting users into purchasing the product, or to causing them to lose interest in buying it. For example, if a user engages when the product is presented but loses attention when the price is presented, then the price is misplaced. • Users can interact with a virtual assistant using the ATS. The virtual assistant may have access to update the user's shopping dialogue. The user can refer to an item on the screen that they are paying attention to, for example, saying, "Hey Dolby, add that item to my cart," and the virtual assistant can automatically add that item to the user's cart. The item the user was paying attention to may have been non-advertising related content such as movies or podcasts. While a user is consuming shopping-related content, the model of each product displayed on the screen is a version selected to suit the user's preferences. For example, John insists that all of his possessions be yellow. If a yellow option is available, he will be shown the yellow version of each product. ATS detected that Alice responds better to shopping content when items are sorted by price from lowest to highest. The next time Alice enters an online shop, this consumer preference will be automatically selected for her. The user shows interest in a car appearing on a shopping channel. The playback system responds, "Shall we book a test drive for you?" The user engages by saying "Yes," and the test drive is automatically arranged using an ATS-enabled device. • A user shows interest in a vacuum cleaner on a shopping TV channel. The user says, "Wow, that's amazing," and the ATS-enabled device adds, "Do you want to order it now?" to the playback. The user responds to the question, "No, I'll think about it," and the order is declined based on the user's response.

[0213] Information-driven ad placement based on user attention. Ad placements can be informed using attentional information detected by the ATS. The term "advertising placement" is used to mean when and which ads to display. Ad placement selection can be determined using long-term trends and optimizations, or in real-time using information about how users are engaging at that particular moment. Furthermore, a combination of these two approaches can be used to optimize real-time ad placement decisions over the long term. Examples of decisions that may be made using this information include: To minimize ad intrusion, ads are placed where users are least engaged with the content. To maximize ad attention, place ads where users are most engaged with the content. • Place ads related to a specific topic only when that topic is present within the content. • Place advertisements for a particular product when users interested in that product type are engaging with the content. • Use the closed loop enabled by ATS to optimize ad placements based on their impact on ad performance or content performance. • Any combination of the above.

[0214] Learned decision-making for ad placement can be performed at different levels, as follows: ·For each user (e.g., prefers ads at the start of content); ·For each population type (e.g., Gen Z viewers, or French viewers, jazz fans); ·For each scene (e.g., certain users are more likely to engage with ads when a particular user is on screen: an ad for a tag hooyer featuring Ryan Gosling for users who respond to that scene, following a scene where Ryan Gosling is on screen); ·For each episode (e.g., in this episode, the least intrusive part for receiving ads is learned); ·For each series (e.g., the greatest attention is generally within the last 5 minutes before an episode ends); and so on.

[0215] Specific examples: ·An original equipment manufacturer (OEM) that manufactures ATS-compatible device components sells attention information to broadcasters. Then, the broadcaster sells advertising space based on the level of attention for that space. ·A mobile phone video game generates revenue by advertising other games during play. The game studio that produces the video game uses an ATS-compatible mobile phone to determine that users are least engaged after a battle within the game. The game studio wants to minimize the annoyance of ads and thus decides to place ads after the battle ends. ·TV program production companies value the viewing experience of their own programs. For this reason, they want to optimize ad placement to maximize the performance of the content. The production company uses the attention level after an ad break to determine the impact of ad placement on the program. Some of the types of attention they may look for include that excited users in the program have returned, that all users have left the room and are not present, that users are currently more engaged with the phone, etc. • While watching a TV program, placements may be delayed until the user has at least a certain level of attention. Alternatively, ads may be reduced while the user is engaged with the content.

[0216] Predictive ad placement based on user attention Building on the previous section, "Informed Ad Placement Based on User Attention," in some examples, models can be formed to predict the type of attention, the level of attention, the best time to place an ad on content, or a combination of these. When a model is trained, it may not require any additional attentional information to determine where to place an ad. The model may learn by referencing one or more content presentations (e.g., audio, video, text). If the model is given the task of predicting the level of attention, the content provider or user can use this information to determine where to place the ad. A model trained to predict the best time to place an ad may be trained to predict different ad placements based on user attributes such as age, user interests, user preferences, and location.

[0217] Specific example: An authoring tool is created that attempts to predict the best time to place ads. TV broadcasters use this tool to automatically determine where to place ads within their programs 24 hours a day, 7 days a week.

[0218] Attention motivation Users of ATS-enabled devices may be offered incentives to engage with content or advertisements. These incentives may be offered by content providers, content creators, TV manufacturers, etc. An example of an incentived attention type is an expression of how one feels about the content (e.g., voice, gestures). Exemplary reward types may include monetary value, bonus content, or something similar.

[0219] Specific example: Jane listens to a lot of music by her favorite artists, and ATS determines that Jane consistently sings along to those artists' music. Based on the ATS data, the artist's label offers Jane a free ticket to one of their shows for being one of their biggest fans. Users watch every football game Manchester plays and cheer loudly for every goal. They also receive discounts on tickets to see Manchester play live. A group of children watches an animated film. Clear enjoyment of the film, based on positive vocal emotions from the group, unlocks bonus content for the movie. At the end of the film, the children are offered intentionally produced bloopers and additional "behind-the-scenes" clips. The user receives an advertisement for a speaker system and responds positively. The user is then presented with an offer to receive a free TV if they purchase the speaker system.

[0220] Content performance evaluation using attention feedback use cases Current content performance evaluation methods generally involve having test audiences preview the content. Obtaining metrics through test audiences has several drawbacks, including the need for manual work (e.g., reviewing questionnaires), the lack of representativeness of the end audience, and the potential high cost. This section details how these issues can be overcome through the use of an Attention Tracking System (ATS).

[0221] Having an ATS allows users to determine how they actually respond to content being played. Used on end-user devices, ATS can make all content consumers test audiences, reduce content evaluation costs, and eliminate the problem of having non-representative test audiences. Furthermore, the analytics generated by an ATS require no manual intervention. Since the analytics are collected automatically in real time, content can be automatically improved by the machine. However, the option of manually optimizing content remains. Additionally, using an ATS during the content improvement process can create a closed loop where the effectiveness of decisions made using attentional information can be tested by utilizing the ATS at another time. Examples of how ATS can be leveraged for content performance evaluation and content improvement are detailed in this section.

[0222] This section of the disclosure refers to types of metadata that specify where a certain attentional response from the user is expected. For example, laughter may be expected at a certain timestamp or during a certain time interval. In some examples, a certain mood may be expected for an entire scene.

[0223] In some implementations, a performance analysis system can take in the expected level of response to content specified by the content creator and / or statistics of responses detected by the ATS, and then output a score that can serve as a content performance metric.

[0224] Event analyzers can incorporate attentional information (events, signals, embeddings, etc.) to determine key events within content that elicited a particular response from a user(s). For example, an event analyzer might perform clustering on response embeddings to determine areas or events within content that users responded to with similar responses. In some examples, probe embeddings might be used to find the time periods when similar attentional indications occurred.

[0225] Exemplary Use Cases Added value for content creators The "Content Performance Assessment Using Attention Feedback Use Cases" section focuses on the added value to content creators through ATS. There are several aspects in which implementing ATS can benefit content creators and content providers. These include: • Content performance evaluation; • Content improvement; and • Real-time content improvement

[0226] Content Performance Evaluation Evaluate content performance based on user attention. Content performance can be determined using attention metrics derived from user ATS. For example, a user leaning forward while viewing the screen presenting content demonstrates user interest in the content. However, a user talking about a topic unrelated to the content may indicate a lack of interest. Such information regarding user attention to content can be aggregated to gain insights into how users are responding as a whole. Aggregated insights can be compared to results from other parts or sections of the content to compare performance. Some examples of parts or sections of content include episodes, shows, games, levels, etc. Differences in attention levels can reveal useful content performance insights. Furthermore, attention information may indicate what users are paying attention to (e.g., themes, objects, effects, etc.). It should be noted that any content performance assessments obtained using ATS can be used in conjunction with traditional assessment methods such as surveys.

[0227] Evaluate content performance using authored metadata. Another extension of "evaluating content performance based on user attention" is when potential user responses are listed in metadata. Suppose one or more users are watching content (e.g., an episode of a Netflix series) on a playback device with an associated ATS. The ATS may be composed of metadata within the content stream to detect certain classes of responses (e.g., laughing, shouting "yes," "oh my god").

[0228] In some cases, content creators or editors can specify what the expected response from the audience is. They can also specify when the response is expected, for example, at a specific timestamp (e.g., at the end of a punchline, during a funny visual event such as a cat smoking a cigarette), within a specific time interval, for a certain category of event type (e.g., a specific type of joke), or for the entire piece of content.

[0229] According to some examples, expected responses may be delivered in a metadata stream along with the content. Alternatively, there may be a library of user response types applicable to many content streams, stored, for example, on the user's device, another local device, or in the cloud, which can be more broadly applicable. The metadata indicating which attention metrics are expected may only include those that are listened to and permitted by the user. This is to give users more privacy and to provide content creators and providers with desired attentional analysis.

[0230] Next, user reactions to the content (in some cases, consistent with the aforementioned metadata) can be collected. Statistics based on these reactions and metadata can be used to evaluate the content's performance. Examples of evaluating specific types of content include:

[0231] joke Content creators add metadata to specify where they expect a "laugh" response from the audience. Laughter can even be categorized into different types, such as "loud laughter," "chuckles," "suffocating laughter," and "hahahaha laughter." In addition, content creators may choose to detect other linguistic responses, such as people repeating a joke or trying to predict the punchline.

[0232] During content streaming, in some cases, metadata may notify the ATS to detect whether a particular type of laughter response occurred. Response statistics can then be collected from different audiences. The performance analysis system can then use these statistics to evaluate the content's performance, which can be useful feedback for the content creator. For example, if statistics show that a particular segment of a joke or skit did not receive many laughter responses from the audience, it means that the performance of this segment needs improvement.

[0233] fear During a horror movie, audiences are often expected to react with exclamations like "that's scary," visible jumps, or gasps. Analysis of ATS information gathered from this authored metadata may reveal that certain segments of a scary scene receive little to no frightening response. This could indicate that those parts of the scary scene need improvement.

[0234] Controversial topic Several channel streams discuss different groups, events, and policies that may receive numerous comments and discussions. During content streaming, metadata notifies the ATS to detect whether supportive or debatable responses are being presented. Statistics of user responses can then be collected from different audiences. Such ATS data can help content creators analyze topic reception.

[0235] provocative scene Content creators can add metadata that specifies locations within content presentations where they expect a strong negative response from the viewer, such as "Ew, gross" or turning their face away. This can be useful in horror movies, user-generated "bad taste" content, etc. In some cases, during content streaming, such as a video of someone eating a spider, metadata could notify the ATS to detect whether a provocative reaction is likely to occur. The aggregated data may indicate that the user did not respond with disgust during the scene. The content creator can then decide that additional work is needed to make the scene more provocative.

[0236] A magnificent scene Content creators can add metadata that can specify locations within content presentations where they expect strong positive responses from viewers, such as "Wow" or "So beautiful." This technology can be useful for movies, sports broadcasts, and user-generated content. For example, in a snowboarding broadcast, slow-motion highlights are expected to receive strong positive reactions. Receiving user response information and analysis from aggregated data from ATS can provide content creators with insights. Content creators can determine whether viewers like the content and then adjust the content accordingly.

[0237] Evaluate content performance using learned metadata. Another extension of "evaluating content performance using authored metadata" is when metadata is learned from user attention information. During content playback, the ATS can collect responses from the audience. Statistics of responses can be fed into an event analyzer to help create meaningful metadata for the content. Metadata may be generated according to one or more specific dimensions (e.g., excitement, tension). In some examples, the event analyzer can decide which metadata to add to determine where events are likely to occur, using techniques such as peak detection. Authored metadata may already exist for the content, but additional learned metadata can still be generated using such disclosed methods.

[0238] Specific example: • Live stand-up comedy shows have all jokes marked up in metadata through information coming from ATS-enabled devices. • A show that has already been authored with metadata has additional metadata learned using an ATS-enabled device. The additional metadata reveals that viewers laughed at moments that were unintentionally funny. The content creator decides to highlight these moments.

[0239] Creating highlight reels based on viewer reactions Another extension of "using learned metadata to evaluate content performance" is when highlight reels are created for content based on the most engaged segments. For example, during a game broadcast, viewers may react excitedly, perhaps shouting "Go, go!" or saying "Ahhh" when their favorite team is losing a fight. These reactions may be detected by ATS and may be predefined reaction types. Collectively, statistics from ATS data may highlight the relatively most prominent moments in the broadcast. These prominent moments can be used to create highlight reels, either automatically or with some effort from the editor. Furthermore, for each individual user who watched the game using an ATS-enabled device, a highlight reel could be created based solely on that user's attention indications. Similarly, highlight reels could be created for subsets of ATS users, such as a group of friends or supporters of the "Red" team.

[0240] User enjoyment can be enhanced by allowing them to hear how other users reacted to highlighted segments within a Reel. This can be done in the same way, or similarly, as detailed in the "Dynamically adding other users' auditory elements to content based on reactions" section of "Personalizing and augmenting with attentional feedback."

[0241] Top occurrences of events based on user attention Another extension of "creating highlight reels based on viewer reactions" is when the highlights derive from a set of specified events (e.g., jokes, tackles, catches). These events can occur across different content, such as episodes, games, books, or songs.

[0242] Specific example: The top 10 best football goal reels are determined based on responses detected by ATS-enabled devices. Responses sought by ATS may include supporting responses such as applause, cheers, and shouts of "Got it!". • Baseball games will receive automatically generated highlights from the sequences that users found most exciting overall. • The top 10 funniest jokes from stand-up comedians are automatically selected for the comedians' promotional videos based on user attention. Reactions that ATS may seek may include laughter, applause, cheering, etc.

[0243] Real-time content improvement Integrating polls into podcasts In some cases, podcasts can obtain real-time audience polls by collecting a few predefined words, such as "yes" or "no." This process can give listeners a greater sense of interactivity and being part of the streaming experience, rather than passively receiving content.

[0244] Example: A musician streams music in real time for their audience. The host asks the audience to choose which song to play from four options. By saying "A," "B," "C," or "D," or by pointing to an option, listeners can select their option and have their attention detected by their ATS. The responses are then aggregated immediately. The host is notified by the results from the audience. The results report that 67.5% of the audience prefer A, 20.1% want to hear C, and the other options are less than 10%. The host decides to play song A for the audience.

[0245] Managing user attention across media locations Obtaining real-time user attention metrics opens up many possibilities. This section lists some use cases of what can be achieved with such attention when combined with contextual information and / or by sharing attention information across devices, media types, etc. While the focus is primarily on location as a form of contextual information, other contextual information types such as weather, local holidays, and events are equally applicable. Many of the examples covered in this section revolve around the idea of ​​continuously customizing the experience across devices, media types, and locations using attention information, such as that detected by ATS.

[0246] This section covers use cases of user attention across media and location. “Content” is used to mean any form of consumable media, such as television programs, radio, bus stop advertisements, billboards, and satellite navigation. It discusses what can be achieved when sharing attentional information across different media types. When discussing the use of attentional information across media types, we are referring to the idea that user responses can be considered in conjunction with user responses observed in other media. This combination can help determine user preferences and interests more robustly. Furthermore, such methods may allow ATS-based systems to use previously determined user preferences and interests to inform decisions in different media. Geographic information is a valuable resource, providing an additional dimension for geo-sensitive actions, such as recommendations for shops, restaurants, and events. In some examples, information collected using an ATS can be combined with location information to enhance existing settings and / or to create new applications that would otherwise not be possible.

[0247] There are at least three elements of the system that can support the use cases outlined below. These are as follows: • Information obtained: This can include a set of content, geographical information, and detected attention information. For example, an advertising board is a form of content, the user's location is a form of geographical information, and the user's attention is captured by the ATS. • Device Platform: Exemplary device platforms include automobiles, TVs, and mobile devices. • Actions: Recommendations (for example, as described in the "Recommendations" section), highlights, purchases, etc.

[0248] Based on these three elements, different scenarios and use cases are listed below.

[0249] Exemplary Use Cases Tracking user preferences across devices and media. As detailed in the section "Determining User Preferences," individual users can have their interests and preferences tracked using ATS-enabled devices. This can be extended when a user's interests may be determined across multiple ATS-enabled devices and media types. Furthermore, when user preferences are tracked across devices and modes, learned user interests can inform decisions on new devices and / or media. User preferences may be determined on the user's device or in the cloud. User preferences may be synchronized, in some cases, using content metadata streams or otherwise. Such preferences may also be stored on the user profile being used while ATS is running (e.g., Google profile, FB profile, Dolby ID, etc.).

[0250] User preferences, such as those determined by ATS, may also be shipped to non-ATS-enabled devices (for example, using cloud-based profiles) to allow for a personalized experience when ATS is unavailable.

[0251] Specific example: Jane watches a movie featuring a music sequence. She sings along with the songs playing, which is detected as positive attention to the music on her ATS-enabled device. Her preferences are updated accordingly. The next day, while she is in her car, the automatically generated playlist takes into account songs similar to those she positively responded to the previous day.

[0252] In a virtual reality (VR) game, Bob decides to give his character a medieval appearance. The VR system is ATS-compatible and detects that Bob enjoys the game more while having a medieval appearance. This suggests that Bob may have an interest in medieval themes. When Bob is looking for his next audiobook, several books with medieval themes are added to his recommendation list. While Steve is listening to a professional basketball game in his car, he seems to get very annoyed every time the Lakers score. When he gets home and turns on the TV, it automatically switches to the TV channel playing the aforementioned game and, since Steve doesn't seem to like the Lakers, automatically selects a commentary stream designed to appeal to Warriors fans.

[0253] Highlight local interests based on user attention. User interests and preferences, as determined by ATS, can be used to provide users with highlights of nearby interests (e.g., landmarks, shops, venues) (e.g., notifications, pins on maps). User interests and preferences can be determined using the methods described in the "Determining User Preferences" section or the "Tracking User Preferences Across Devices and Media" section. Combining these user preferences with location information may allow for recommendation-based systems. Some examples of such recommendations include highlighting locations on satellite navigation to notify users that they are driving past locations that match their interests.

[0254] Specific example: • The user has been detected as a fan of the artist. The user will be notified when driving past a location where the artist has previously performed or will soon perform. While the user is watching the four-wheel drive show, they react particularly strongly to vehicles of one particular brand. When the user drives past a dealership that sells that car, the dealership is highlighted on the map. The car's voice assistant also suggests setting up a test drive reservation. • The user has watched many historical programs about Pompeii with interest. When the user is driving past an exhibition about Pompeii, the vehicle will notify the user, and the satellite navigation will highlight museums near the route. Users showed interest in Omega watches while watching several movies and TV shows where Omega watches were visible. Users were shown the nearest Omega store when searching for GPS directions on their phones.

[0255] Location-linked ads paired with user attention By combining location and user attention information, new geo-linked advertising strategies can be created and existing ones improved. Current examples of location-based advertising include billboards and advertisements on public transport. This strategy can use ATS to help advertisers determine whether their real-world-located ads are successful. Furthermore, this strategy offers new advertising opportunities where ads can be virtually placed in locations.

[0256] Real-world advertisements can have their attention tracked through the use of ATS-enabled devices. For example, many current advertising strategies, such as billboards, are tied to location, such as where they are placed. ATS can further refine which geographic-based locations receive more attention, as opposed to generalized data such as traffic data. Users can be detected as looking at the billboard and may have other user responses associated with that billboard. Interaction with a particular physical advertisement can be determined based on GPS data, camera data, microphone data, etc. Furthermore, physical advertisements can encourage attentional responses through interactive mechanisms such as trivia. After responding to trivia, an action can be taken even if the answer is incorrect.

[0257] Virtual advertising can be achieved in a similar way to the idea detailed in "Highlighting local interests based on user attention," except that local interests are motivated by the advertisement. New advertising strategies are possible. For example, all vehicles on the road are advertisements for their own models, and if it is detected that a user is paying attention to another vehicle, it is known that the user may be interested in that vehicle. Alternatively, markers could appear on the user's map indicating locations with geographically linked advertisements that match the user's preferences.

[0258] Users may connect with advertisers in the following situations: ·When a user interacts with one of the geographically linked advertisements, • When a user is near a geographically linked advertisement that is of interest to them, as determined based on previous warnings.

[0259] In some cases, as a result of successful client connections from ATS-enabled ads, an automotive OEM may be offered discoverer compensation by the ad sponsor.

[0260] Specific example: A trivia question, "What was the first movie to use Dolby Audio?", is displayed on a geographically linked advertising board. The answer is "A Clockwork Orange." A free movie ticket is offered for the correct answer, while an incorrect answer presents the user with a "Learn More" link. The user walks past a bus with humorous geographical-linked advertisements on its side. The system detects that the user is laughing using their ATS-enabled mobile device. The next advertisement that appears on the phone is for the same product advertised on the bus. • When a user begins discussing a topic presented in an ad, it is detected that they are engaging with the ad on the car. Statistics on the effectiveness of the ad board are sent back to the advertiser or ad board manager, and different prices are set for the ad board based on viewer attention. Alice drives past an advertisement for a live music event. She responds positively to the advertisement and receives an offer to purchase tickets for the event. In the same example as above, instead of receiving an offer to buy tickets, Alice is offered the opportunity to have some of the band's new music played for her in the car. A user is driving and sees an advertisement for a concert on the bus, saying, "[Name] is coming to [Location] for a concert. I'm definitely going." An ATS-enabled vehicle uses the vehicle's current location to capture the user's response to the advertisement. This is then used to offer the user a ticket. • While driving past a bus stop advertisement for Acme watches, the user says, "Acme watches are nice." Then, the next time the user is near an Acme store, that store will be highlighted on the map for the user. The musical show paid for advertisements linked to a virtual geography tailored to the user's preferences for its upcoming show. If a user has interests aligned with the musical, or the topics covered by an upcoming musical, the user will have the venue location pinned on their satellite navigation. • Car brands can decide to use the cars they manufacture as geographically linked advertisements that they can engage with. A user might see someone else driving a car they like and say, "That's nice," which is then detected by an ATS-enabled device. The user then receives personalized advertisements tailored to their preference for that particular car model. • Potential interest in the content of an advertisement on an advertising board is determined by how users perceive and pay attention to the advertisement each time they drive past it.

[0261] Continuous attention and customization across devices and locations This section provides examples of what can be achieved using combinations of the following sections. • Track user preferences across devices and media. • Highlight local interests based on user attention. • Location-based ads paired with the user's attention.

[0262] Dynamic, continuous attention tracking optimization may be possible. For example: Device A detects positive attention to something of interest to the user and distributes this knowledge to other devices. Device B creates a highlight reel for topics of interest. Device C advertises things that are of interest to the user.

[0263] Specific example: The user engages with geographically linked advertisements for watches on an ad board in their car. Following this, the user is presented with advertisements for the same watch brand on their smart TV during commercial breaks. Finally, when the user starts searching for restaurants on their map app for lunch, watch brand stores are highlighted on the user's map.

[0264] User-attention-based navigation and routing Routes determined by the navigation system may use attentional analysis to help inform the decision. This can occur both over short periods (e.g., guiding a user on a scenic route because they are in a relaxed mood and not in a hurry) and over long periods (e.g., optimizing urban traffic based on what holistically brings the highest level of positive attention to driving). Routes may also be personalized to guide the user onto a path that better suits their interests or preferences. User interests and preferences may be determined by methods described, for example, in the sections “Determining User Preferences” or “Tracking User Preferences Across Devices and Media.” Recommended routing may also suggest departure times that take into account the user’s emotional state, as determined by the ATS.

[0265] Specific example: • Peak or "rush hour" times in suburban areas are particularly problematic. Routing is generated for each user based on the preferences of all ATS users driving in suburban areas during peak hours. In some cases, an ATS-enabled navigation system may suggest that some people leave early in the morning, as they are known to often wake up early, and vice versa. The navigation system may also suggest that users take different routes based on what kind of roads they enjoy and the proximity of their location to their destination. In these ways, traffic routing in suburban areas can be optimized in a holistic manner. Since traffic and weather conditions are favorable for driving, the user may be advised to leave a little earlier today. Furthermore, in this case, the user is in a good mood and may be more willing to leave earlier. Navigation routing based on ATS information is given this reasoning and informs the user to take an earlier route. The user is looking to drive to a destination where there are two different routes that take roughly the same amount of time. The user is recommended the route that has shops and / or advertisements that match their interests. Chung is planning a road trip. He wants to take a route with "good scenery." The ATS-based navigation system suggests a route that maximizes coastal driving, as Chung regularly watches marine documentaries and surfing videos. Yvonne is planning a similar road trip. Her "scenic" route will pass by top-rated restaurants and wineries along the way, as she often watches cooking shows and talks about wine in the car.

[0266] Trivia about using ATS The use of ATS allows for a wide range of trivia experiences. Trivia is a form of content. By utilizing location information, context-relevant trivia can be arbitrarily derived. Trivia questions can also be supplied by advertisers.

[0267] Specific example: While John drives around town in an ATS-enabled vehicle, trivia is automatically generated for him in games like Eye-Spy. Nearby landmarks and towns can form the basis for trivia questions such as, "When was this town founded?" In the same scenario as above, a light advertising pitch is incorporated into the mix of trivia questions John receives. For example, "What is the only wrist-like watch that is...?" The answer could be a watch brand John likes, such as Acme. • A completely ad-centered trivia game can be offered to users. This game can serve as entertainment for users and / or may offer them rewards. An example reward after a trivia ad session could be that the user does not receive any further ads across their devices for the rest of the day.

[0268] User attention-based geographic sentiment mapping By using attentional information in conjunction with location information, it becomes possible to map areas to individual user emotions. This type of mapping can prove to be insightful for geographical decisions. In some cases, such mappings can be post-processed by computer programs to help users digest the information. For example, the program could create summaries, pin locations, etc.

[0269] Specific example: Users looking to buy a new home are trying to determine areas that might be suitable for them. Geographic sentiment mapping data can prove useful because it can provide insights into areas where users have previously responded positively (e.g., laughing, excitedly talking) compared to areas where users have given negative responses (e.g., swearing, complaining). Users may be provided with a list of recommended areas where they might want to buy a home. A couple is looking to buy a house. Each time they view a property, they return to their car and receive explanations while driving away. The emotions at each location are tracked by their ATS-enabled vehicle. When the couple is about to make a final decision, they have the ATS information displayed to them as a dashboard, a heatmap, along with location pins of the places they visited during their house-hunting adventure. Using such an information summary, the couple can make more informed decisions about their feelings regarding the homes they might buy without having to take detailed notes about their previous impressions.

[0270] Improving safety and well-being through attention feedback In recent years, products aimed at welfare and other related purposes have become increasingly popular. Applications intended to monitor and improve user well-being would greatly benefit from information that strongly reflects the user's current state in order to predict well-being. This type of information can be collected from Attention Tracking Systems (ATS).

[0271] Attentional analysis can be used to enhance user safety. Safety systems can withstand information from many types of attentional information, such as mood, attention level, and positive / negative emotions. This section lists examples of how ATS technology can improve user safety and well-being.

[0272] Exemplary Use Cases User utterances detected by ATS as a substitute for attention An ATS (Automatic Train Stop) system installed in a vehicle can be used to determine whether the driver is focused on the driving task and in a calm and composed state of mind. This assessment may be performed in real time and / or may improve over time as the ATS-enabled system learns the user's behavior. Learning the user's behavior can help determine what is normal for a particular user and detect deviations from that normality. For example: Singing along to music playing in the car or laughing at jokes on a podcast may indicate that the driver is alert and attentive. • Abusive language can indicate that the driver is irritated or suffering from "road rage," which may lead to inappropriate judgments. • A certain user is known to constantly utter insults. When this user is detected to be using abusive language, the attribution to "road rage" may be less likely. Snoring can indicate that the driver has fallen asleep! Blinking or head bobbing may indicate that the driver is about to fall asleep.

[0273] Adaptive acoustic driver mindfulness application Several smartphone applications targeting mindfulness, meditation, sleep, calmness, and mental well-being exist in the prior art. Some disclosed examples provide mindfulness applications configured to enhance concentration on the task of driving a car. In proposed technologies and systems, feedback from the ATS (Automatic Trainer System) is used to inform the reproduction of diverse acoustic responses, for example, from the in-car audio system.

[0274] Here is an example of a generated voice response that a driver might hear from such a system. "This morning, you cursed at other drivers more often than usual. You sound a little irritated and not concentrating on driving. Here's a calming whale song to help you concentrate."

[0275] Improving driver concentration by gamifying attention. In this example, the driver may be asked to participate in a simulated trivia quiz or other game designed to reward careful driving. The insurance company may offer a reduction in car insurance premiums to drivers who consistently achieve good scores in such games.

[0276] For example, drivers may be periodically asked to answer questions about road conditions, such as the following: "What color was the car that just merged from the right lane?" • "What is the current speed limit?" • "How many meters does the left lane end?" These examples can be extended to additional use cases such as operating heavy machinery and air traffic control.

[0277] Automatically recommend breaks An ATS-enabled system may be configured to detect that the driver is becoming more distracted over time, more irritated as traffic deteriorates, more fatigued, and less focused on the road than before. In some examples, such a system may detect that the driver is becoming more fatigued over time, more irritated as traffic deteriorates, and less focused on the road than before, using methods specified in "user utterances detected by ATS as a substitute for attention." Based on this, the system may recommend that the driver take a break for the safety of themselves and their passengers (if any).

[0278] Such a system may also personalize recommendations, as described in "Personalization and Enhancement Using Attentional Feedback," which includes, for example, "Long-Term Personalization" and "Recommendations" sections, so that the user is shown preferred rest options. These may include preferred foods, coffee, service stations, etc. Preferences may be determined using the user's attention to driving after taking a rest at a particular location. Insurance companies may offer reduced auto insurance premiums to drivers who consistently follow such recommendations and refrain from driving while disabled.

[0279] Keep the children in the back seat quiet. Children yelling or arguing in the back seat while a car is in motion can be distracting to the driver. In some cases, an ATS (Automatic Child Safety System) can be configured to detect the sounds of a child yelling, arguing, or being bored and automatically switch the content playing from the in-car audio system to something more entertaining for the child. In some cases, an ATS can be configured to ensure a certain target attention level (e.g., by laughing at a joke, or by singing along or responding during a call-and-response segment in a child's podcast). If the child's attention level falls below a certain level, in some cases, the system can automatically try different content until the child calms down and helps the driver to concentrate.

[0280] Measuring mental health using surrogate speech. A person's utterances and other responses, when analyzed over several hours or days, can reveal something about their mental health. For example, an ATS configured to listen for laughter may be able to calculate a metric for average laughter per week. This could then be translated into a score indicating the person's general mental health or overall well-being. The user's previous attention patterns or deviations from the average may also be detected, which could indicate that something has changed in the user's well-being.

[0281] Other examples include: Frequent verbal abuse can be an indicator of poor mental health. • Monotonous speech, especially when compared to a baseline of a person's normal speaking style, can indicate poor mental health. The frequency with which people smile may indicate good well-being. • The possibility that a person sings along to uplifting songs may indicate positive mental health.

[0282] Figure 12 is a flowchart outlining an example of the disclosed method. Method 1200 may be performed, for example, by the control system 160 in Figure 1A, by the control system 160 in Figure 2, or by one of the other control system instances disclosed herein. The blocks of Method 1200, as with other methods described herein, are not necessarily performed in the order shown. According to some examples, one or more blocks may be performed in parallel. Furthermore, some similar methods may include more or fewer blocks than those illustrated and / or described.

[0283] Method 1200 may be performed by an apparatus or system including the control system 160 in Figure 1A, the control system 160 in Figure 2, or one of the other control system instances disclosed herein. In some examples, blocks of Method 1200 may be performed by one or more devices in an audio environment, for example, by an audio system controller (which may be referred to herein as a smart home hub), or by another component of the audio system, such as a television, a television control module, a laptop computer, a game console or system, or a mobile device (such as a cellular phone). However, in some implementations, at least some blocks of Method 1200 may be performed by one or more devices configured to implement cloud-based services, such as one or more servers.

[0284] In this example, block 1205 involves the control system acquiring sensor data from the sensor system during content presentation. Content presentation may be, for example, television programs, movies, advertisements, music, podcasts, game sessions, video conferencing sessions, or online learning courses. In some examples, the control system may acquire sensor data in block 1205 from one or more sensors of the sensor system 180 disclosed herein. Sensor data may include sensor data from one or more microphones, one or more cameras, one or more eye trackers configured to collect gaze and pupil size information, one or more ambient light sensors, one or more thermal sensors, one or more sensors configured to measure electrocutaneous responses, and the like.

[0285] In this example, block 1210 involves the control system in estimating user response events based on sensor data. In some examples, block 1210 may be performed by one or more device analysis engines (DAEs). Response events estimated in block 1210 may include, for example, detected phonemes, emotion type estimation, heart rate estimation, body posture estimation, and one or more latent spatial representations of sensor signals.

[0286] In this example, block 1215 involves the control system generating user attention analysis based at least in part on estimated user response events corresponding to estimated user attention to content intervals of content presentation. In some examples, block 1215 may be performed by the attention analysis engine. A “content interval [interval]” may or may not be of uniform size. In some examples, a content interval may be of uniform size, such as 1 second, 2 seconds, or 3 seconds. Alternatively or additionally, a content interval may correspond to an event in content presentation, such as the presence of a particular actor, the presence of a particular object, the presentation of a theme such as a musical theme, a joke, a dramatic event, a romantic event, a frightening event, or a beautiful event.

[0287] In this example, block 1220 involves the control system modifying the content presentation based at least in part on user attention analysis. In some examples, block 1220 may be performed by the content presentation module in accordance with input from the attention analysis engine. Modifying the content presentation may involve, for example, adding or changing a laugh track, adding or changing audio corresponding to the responses of one or more other people to the content presentation (or a similar content presentation), extending or shortening the time that at least a portion of the content is presented, or adding, removing, or changing at least a portion of the visual content presentation and / or audio content presentation. In some examples, modifying one or more aspects of the audio content may involve modifying one or more spatial properties of the audio content. In some examples, modifying one or more spatial properties of the audio content may involve rendering at least one audio object at a different location than where it would normally be rendered.

[0288] In some cases, content presentation may be modified based at least partially on one or more user preferences or other user characteristics. In some examples, modifying content presentation may involve personalizing or enhancing content presentation, for example, based at least partially on one or more user preferences or other user characteristics. In some examples, personalizing or enhancing content presentation may involve changing audio playback volume, one or more other audio characteristics, or a combination thereof. In some examples, personalizing or enhancing content presentation may involve changing one or more display characteristics, such as brightness or contrast. In some examples, personalizing or enhancing content presentation may include changing the storyline, adding characters or other story elements, changing the time intervals in which a character is involved, changing the time intervals dedicated to other aspects of content presentation, or a combination thereof.

[0289] In some examples, personalizing or enhancing content presentation may include providing personalized advertising content. In some such examples, providing personalized advertising content may include providing advertising content that corresponds to an estimated user's attention to one or more content segments relating to one or more products or services.

[0290] In this example, block 1225 is involved in causing a control system to provide the modified content presentation. In some examples, block 1225 may be involved in providing the modified content presentation on a television screen, one or more loudspeakers, etc.

[0291] Some aspects of this disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed method, and a tangible computer-readable medium (e.g., a disk) for storing code to perform one or more examples of the disclosed method or its steps. For example, some disclosed systems are or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed in software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed method or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed method (or its steps) in response to asserted data.

[0292] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on an audio signal(s), including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) programmed and / or otherwise configured with software or firmware to perform any of the diverse operations including one or more examples of the disclosed methods. Or, elements of some embodiments of the system of the present invention may be implemented as a general-purpose processor or DSP configured (e.g., programmed) to execute one or more examples of the disclosed methods, and the system may also include other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to execute one or more examples of the disclosed methods may be coupled with input devices (e.g., a mouse and / or keyboard), memory, and display devices.

[0293] Another aspect of this disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) for storing code (e.g., an executable coder for execution) for performing one or more examples of the disclosed method or its steps.

[0294] While specific embodiments and applications of the Disclosure have been described herein, it will be apparent to those skilled in the art that many modifications are possible to the embodiments and applications described herein without departing from the scope of the Disclosure described herein and claimed herein. While specific forms of the Disclosure have been shown and described, it should be understood that the Disclosure should not be limited to the specific embodiments or methods described herein.

[0295] Various aspects of this disclosure can be understood from the following enumerated example embodiments (EEEs).

[0296] [EEE1] Head unit and; Loudspeaker systems and; Sensor systems and; Control system and A system having the following control system: One or more device analysis engines configured to estimate user response events based on sensor data received from the aforementioned sensor system; A user attention analysis engine configured to generate user attention analysis based at least in part on estimated user response events received from one or more device analysis engines, wherein the user attention analysis corresponds to estimated user attention to content segments of content presentation provided through the head unit and the loudspeaker system. It has, The aforementioned control system: The content presentation is modified based at least partially on the aforementioned user attention analysis; The modified content presentation is configured to be provided by the head unit, by the loudspeaker system, or by both the head unit and the loudspeaker system. system. [EEE2] The system according to EEE1, further comprising an interface system configured to provide communication between the control system and one or more other devices over a network, wherein the modified content presentation is or includes modified content received from the one or more other devices over the network. [EEE3] The system according to EEE2, wherein modifying the content presentation includes transmitting a user attention analysis from the user attention analysis engine via an interface system and receiving the modified content in response to the user attention analysis. [EEE4] The system described in any one of the EEE1 to 3, wherein modifying the content presentation includes personalizing or enhancing the content presentation. [EEE5] The system described in EEE4, which personalizes or enhances the content presentation by changing one or more of the following: audio playback volume, audio rendering position, one or more other audio characteristics, or a combination thereof. [EEE6] The system according to EEE4 or 5, wherein the head unit is a television, or includes a television, and personalizing or enhancing the content presentation includes changing one or more television display characteristics. [EEE7] Personalizing or enhancing the content presentation includes any one of the systems described in EEE3 to 6, which includes altering the storyline, adding characters or other story elements, altering the time intervals in which a character is involved, altering the time intervals dedicated to other aspects of the content presentation, or a combination thereof. [EEE8] Personalizing or enhancing the content presentation includes providing personalized advertising content, as described in any one of the EEE3 to 7. [EEE9] Providing personalized advertising content includes providing advertising content that corresponds to the estimated user's attention to one or more content segments relating to one or more products or services, as described in EEE8. [EEE10] Personalizing or enhancing the aforementioned content presentation includes providing or modifying the laughter track, as described in any one of the EEE3 to 9 systems. [EEE11] The head unit, the loudspeaker system, and the sensor system are in the first environment; The control system is further configured to modify the content presentation based at least in part on sensor data, estimated user response events, user attention analysis, or a combination thereof, corresponding to one or more other environments. A system described in any one of the items EEE1 through EEE10. [EEE12] The control system is further configured to pause or replay the content presentation, as described in any one of EEE1 to 11. [EEE13] The sensor system includes one or more cameras, and the sensor data includes camera data, as described in any one of EEE1 to 12. [EEE14] The sensor system includes one or more microphones, and the sensor data includes microphone data, as described in any one of EEE1 to 13. [EEE15] The system according to EEE14, wherein the control system is further configured to implement an echo management system to mitigate the effects of audio reproduced by the loudspeaker system and detected by one or more microphones. [EEE16] The system according to any one of EEE1 to 15, wherein one or more first parts of the control system are deployed in a first environment, and a second part of the control system is deployed in a second environment. [EEE17] The system according to EEE16, wherein one or more first parts of the control system are configured to implement one or more device analysis engines, and the second part of the control system is configured to implement the user attention analysis engine. [EEE18] The head unit is a digital media adapter, or includes a digital media adapter, as described in EEE4 or 5.

Claims

1. Head unit and; With a loudspeaker system; Sensor systems and; Control system and A system having the following control system: One or more device analysis engines configured to estimate user response events based on sensor data received from the aforementioned sensor system; A user attention analysis engine configured to generate user attention analysis based at least in part on estimated user response events received from one or more device analysis engines, wherein the user attention analysis corresponds to estimated user attention to content segments of content presentation provided through the head unit and the loudspeaker system. It has, The aforementioned control system is: Modify the content presentation based at least partially on the aforementioned user attention analysis; The modified content presentation is configured to be provided by the head unit, by the loudspeaker system, or by both the head unit and the loudspeaker system. system.

2. The system according to claim 1, further comprising an interface system configured to provide communication between the control system and one or more other devices over a network, wherein the modified content presentation is or includes modified content received from the one or more other devices over the network.

3. The system according to claim 2, wherein modifying the content presentation includes transmitting a user attention analysis from the user attention analysis engine via the interface system and receiving the modified content in response to the user attention analysis.

4. The system according to any one of claims 1 to 3, wherein modifying the content presentation includes personalizing or enhancing the content presentation.

5. The system according to claim 4, wherein personalizing or enhancing the content presentation includes changing one or more of the following: audio playback volume, audio rendering position, one or more other audio characteristics, or a combination thereof.

6. The system according to claim 4, wherein the head unit is a television or includes a television, and personalizing or enhancing the content presentation includes changing one or more television display characteristics.

7. The system according to claim 3, wherein personalizing or enhancing the content presentation includes altering the storyline, adding characters or other story elements, altering the time intervals in which a character is involved, altering the time intervals dedicated to other aspects of the content presentation, or a combination thereof.

8. The system according to claim 3, wherein personalizing or enhancing the content presentation includes providing personalized advertising content.

9. The system according to claim 8, wherein providing personalized advertising content includes providing advertising content that corresponds to an estimated user's attention to one or more content segments relating to one or more products or services.

10. The system according to claim 3, wherein personalizing or enhancing the content presentation includes providing or modifying the laughter track.

11. The head unit, the loudspeaker system, and the sensor system are in the first environment; The control system is further configured to modify the content presentation based at least in part on sensor data, estimated user response events, user attention analysis, or a combination thereof, corresponding to one or more other environments. The system according to claim 1.

12. The system according to claim 1, wherein the control system is further configured to pause or replay the content presentation.

13. The system according to claim 1, wherein the sensor system includes one or more cameras, and the sensor data includes camera data.

14. The system according to claim 1, wherein the sensor system includes one or more microphones, and the sensor data includes microphone data.

15. The system according to claim 14, wherein the control system is further configured to implement an echo management system to mitigate the effects of audio reproduced by the loudspeaker system and detected by one or more microphones.

16. The system according to claim 1, wherein one or more first parts of the control system are deployed in a first environment, and a second part of the control system is deployed in a second environment.

17. The system according to claim 16, wherein one or more first parts of the control system are configured to implement one or more device analysis engines, and the second part of the control system is configured to implement the user attention analysis engine.

18. The system according to claim 4, wherein the head unit is a digital media adapter or includes a digital media adapter.

Images