Media content playback is modified based on the user's mental state.

Abstract

The embodiments of this disclosure describe a computer-implemented method for playing media content, the computer-implemented method comprising: determining a first mental state associated with a user accessing the media content based on sensor data; and modifying the playback of the media content based on the first mental state.

Description

Technical Field

[0001] The various implementation schemes generally involve psychophysiological sensing systems, and more specifically involve modifying media content playback based on the user's mental state. Background Technology

[0002] When interacting with a given environment, users may divide their attention between multiple tasks. In various environments, users may perform primary tasks, such as driving, working, or exercising, while simultaneously performing secondary tasks, such as consuming media content. In some situations, the primary task requires more attention, which may reduce a user's ability to focus on secondary tasks. For example, when a user is focused on driving to navigate complex driving situations, they may miss all or part of the media content being consumed as a secondary task.

[0003] Some multimedia systems allow users to increment their input in fixed time increments ( For example Users can manually rewind media content in 15-second increments. Users who have missed all or part of the media content can manually rewind in fixed time increments. However, rewinding in fixed time increments can be inaccurate because it doesn't account for the actual amount of time the user missed while distracted from the media. Typically, users would want to rewind the media content back to the point where they began to miss it. However, manually identifying the correct amount of time to rewind can be difficult and requires multiple attempts and / or overshooting.

[0004] As mentioned earlier, improved techniques for modifying media content playback after a user becomes distracted would be useful. Summary of the Invention

[0005] One implementation describes a computer-based method for playing media content. The method includes determining a first mental state measure associated with a user accessing the media content based on sensor data. The method also includes modifying the playback of the media content based on the first mental state measure.

[0006] Other embodiments, in particular, provide a method and system configured to implement the computer-readable storage medium described above.

[0007] At least one technical advantage of the disclosed technology over the prior art is that the disclosed technology can rewind and replay or pause and resume media content based on a determined user mental state. The disclosed technology can more accurately rewind media content to a point in time when a user was distracted from the media content, relative to conventional techniques that rewind media content in fixed time increments. Additionally, the disclosed technology can automatically pause playback of media content when a user is distracted from the media content and resume playback when the user is no longer distracted. These technical advantages represent one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to provide a more thorough understanding of the above-mentioned features of the various embodiments, the inventive concept briefly outlined above can be understood by reference to the following more detailed description in connection with the accompanying drawings, in which some of the embodiments are shown. It should be noted, however, that the accompanying drawings are not intended to limit the inventive concept in any way and are merely intended to illustrate one or more typical embodiments of the inventive concept.

[0009] Figure 1 A block diagram of a system configured to implement one or more aspects of the present disclosure is shown.

[0010] Figure 2 A view from the perspective of a passenger cabin of a vehicle is shown in accordance with various embodiments.

[0011] Figure 3 A media controller application in accordance with various embodiments is shown in more detail. Figure 1

[0012] Figure 4 An example vehicle system including a media controller application in accordance with various embodiments is shown. Figure 1

[0013] Figure 5 A flowchart of method steps for playing media content based on a mental state metric in accordance with various embodiments is shown.

[0014] Figure 6 A flowchart of method steps for playing media content based on a mental state metric in accordance with various other embodiments is shown. DETAILED DESCRIPTION

[0015] In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0016] SYSTEM OVERVIEW ​​

[0017] Figure 1 A block diagram of a system 100 configured to implement one or more aspects of the present disclosure is shown. As shown, the system 100 includes, but is not limited to, a computing device 110, one or more sensors 120, one or more input / output (I / O) devices 130, and a network 150. The computing device 110 includes a processing unit 112 and a memory 114. The computing device 110 can be a device including one or more processing units 112, such as a system on a chip (SoC). In various embodiments, the computing device 110 can be a mobile computing device or a host unit included in a vehicle system. Embodiments disclosed herein contemplate any technically feasible system configured to implement the functionality of the system 100 via the computing device 110. Various examples of the computing device 110 include a mobile device For example , a phone, a tablet, a laptop, etc.), a wearable device For example , a watch, a ring, a bracelet, an earpiece, etc.), a consumer product For example , a game, etc.), a smart home device For example , a smart lighting system, a security system, a digital assistant, etc.), a communication system For example , a teleconference system, a videoconference system, etc.), and the like. The computing device 110 can be located in various environments, including but not limited to a road vehicle environment For example , a consumer car, a commercial truck, etc.), an aerospace and / or aviation environment For example , an airplane, a helicopter, a spacecraft, etc.), a nautical and submarine environment, and the like.

[0018] The processing unit 112 can include one or more central processing units (CPUs), digital signal processors (DSPs), microprocessors, application-specific integrated circuits (ASICs), neural processing units (NPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), combinations thereof, and the like. The processing unit 112 generally includes one or more programmable processors that execute program instructions to manipulate input data. In some embodiments, the processing unit 112 can include any number of processing cores, memory, and other modules for facilitating program execution.

[0019] The memory 114 can include a memory module or a collection of memory modules. The memory 114 generally includes a memory chip, such as a random-access memory (RAM) chip, that stores application programs and data processed by the processing unit 112. In various embodiments, the memory 114 can include non-volatile memory, such as an optical drive, a magnetic drive, a flash drive, or other storage device. In some embodiments, a separate data storage area, such as a data storage area 152 accessible via the network 150 (“cloud storage”), can supplement the memory 114.

[0020] As shown, the memory 114 stores a media controller application 140 and a multimedia application 145. In operation, the processing unit 112 executes the media controller application 140 and the multimedia application 145. As discussed in greater detail below, the media controller application 140 receives and processes sensor data from one or more sensors 120 to determine a value of a mental state metric associated with a user. In some embodiments, the mental state metric can represent a cognitive load associated with brain activity the user is using, an emotional load associated with an emotional state of the user, an amount of mind wandering of the user, or any combination thereof. The media controller application 140 controls playback of media content based on such mental state metric through the multimedia application 145.

[0021] In some embodiments, the media controller application 140 can control playback of media content based on the determined mental state metric by: (1) storing a playback time when the mental state metric begins to satisfy a threshold and, optionally, when the mental state metric stops satisfying the threshold; and (2) rewinding the media content and replaying from the playback time when the mental state metric began to satisfy the threshold, either automatically after the mental state metric stops satisfying the threshold or in response to a user request. For example, the playback time can be stored as a time code (e.g., a time stamp and / or time index within the media content) in the memory 114 of the computing device 110 or elsewhere (e.g., in a data store 152 accessible via the network 150) along with any other suitable metadata (e.g., metadata regarding a file name and / or a URL associated with and / or otherwise identifying the media content). For example For example In some embodiments, the media controller application 140 can control playback of media content based on the determined mental state metric by: (1) storing a playback time when the mental state metric begins to satisfy a threshold and, optionally, when the mental state metric stops satisfying the threshold; and (2) rewinding the media content and replaying from the playback time when the mental state metric began to satisfy the threshold, either automatically after the mental state metric stops satisfying the threshold or in response to a user request. For example, the playback time can be stored as a time code (e.g., a time stamp and / or time index within the media content) in the memory 114 of the computing device 110 or elsewhere (e.g., in a data store 152 accessible via the network 150) along with any other suitable metadata (e.g., metadata regarding a file name and / or a URL associated with and / or otherwise identifying the media content). For example

[0022] ​​In other implementations, the media controller application 140 can control the playback of media content based on a determined user mental state by pausing playback when a mental state metric meets a threshold and resuming playback when the mental state metric no longer meets the threshold. In some implementations, the user may have the option to enable or disable control over media content playback based on mental state.

[0023] Multimedia application 145 can be any technically feasible application capable of playing media content. As used herein, media content can include audio and / or video content. For example, multimedia application 145 can be a conventional media player application. Although shown as different from multimedia application 145, in some embodiments, the functionality of media controller application 140 can be included in multimedia application 145, and vice versa.

[0024] As described, one or more sensors 120 may include one or more means of performing measurements and / or acquiring data that can be used to determine a measure of mental state associated with the user. In various embodiments, one or more sensors 120 may generate sensor data relating to cognitive load, emotional load, mental wandering, and / or any combination thereof experienced by the user. For example, one or more sensors 120 may collect biometric data relating to the user ( For example This includes heart rate, respiratory behavior, brain activity, skin conductance, blood oxygenation, pupillary measurement, eye fixation and saccades, skin conductance response, blood pressure level, mean blood glucose concentration, etc. Alternatively or additionally, one or more sensors 120 may generate sensor data relating to objects in the environment that are not the user. For example, one or more sensors 120 may generate data relating to vehicle operation (…). For example Sensor data (such as pedal position, steering wheel position, etc.) including objects around the vehicle, vehicle speed, ambient temperature inside the vehicle, and amount of light inside the vehicle. In some embodiments, one or more sensors 120 may be coupled to and / or included therein in a computing device 110.

[0025] In some implementations, one or more sensors 120 may acquire data processed by the media controller application 140 to calculate the cognitive load experienced by the user as a measure of mental state. For example, one or more sensors 120 may include a pupil sensor ( For example A camera (focused on the user's eye) is used, and the pupil sensor acquires image data about at least one pupil of the user. The media controller application 140 can then perform various pupil measurement techniques, such as time-series-based techniques using time-series wavelet transform, which detect eye parameters (…). For example, pupil diameter fluctuations, direction in which the pupil is gazing, eyelid position, etc.), and estimate the user’s cognitive load based on the eye parameters. As another example, the media controller application 140 can feed the image data into a machine learning model that has been trained to classify cognitive load, emotional load, and / or amount of mind wandering by analyzing eye saccades and fixations, blink rate duration, and / or other parameters. As yet another example, the one or more sensors 120 can include a heart rate sensor and / or other biometric sensor that acquires a user’s biosignals and / or physiologic signals For example , heart rate, respiration rate, eye movements, GSR, neural brain activity, etc.). In such cases, the media controller application 140 can compute the cognitive load that the user is experiencing from one or more of the acquired biosignals and / or physiologic signals.

[0026] In some embodiments, the one or more sensors 120 can acquire sensor data that the media controller application 140 processes in order to determine the emotional load that the user is experiencing as a mental state metric. The emotional load can be defined and computed in various ways. In some embodiments, parameterized measures of emotion associated with different aspects of emotion can be computed, such as a valence measure that represents positive or negative emotion, an arousal measure that represents the intensity of emotion, and / or a dominance measure that represents the level of emotional control. The parameterized measures can then be combined to determine the emotional load that the user is experiencing. For example, the one or more sensors 120 can include sensors that acquire a user’s biosignals and / or physiologic signals For example , perspiration, heart rate, heart rate variability (HRV), blood flow, blood oxygen level, respiration rate, galvanic skin response (GSR), sounds produced by the user, behavior of the user, etc.). In such cases, the media controller application 140 can compute one or more quantified parameterized measures of emotion based on the sensor data in order to determine the emotional load that the user is experiencing. Examples of using parameterized measures to determine emotional load are further described in U.S. Patent Application entitled “Techniques for Separating Driving Emotion from Media Induced Emotion in a Driver Monitoring System” having serial number 16 / 820,533, filed March 16, 2020, which is incorporated by reference herein in its entirety. In other embodiments, a discrete classification of emotion For example may be made using a machine learning model Example For example, joy, anger, sadness, etc.). For example, one or more sensors 120 can include a user-facing camera that records a user’s face in image data. In such cases, the media controller application 140 can analyze the image data using a machine learning model (or otherwise) to determine the user’s facial expression, which is then mapped to a particular emotion. In still other embodiments, the media controller application 140 can perform various voice tonality analyses on audio data recorded via a sound sensor (microphone) to determine the emotional load experienced by the user. For example, when a user begins speaking over a phone call, the media controller application 140 can determine that the user is distracted from the recreational media content. As another example, the media controller application 140 can detect a high emotional load (or cognitive load) in the voice of the user speaking, indicating that the user is likely distracted from the recreational media content. For example

[0027] In some embodiments, one or more sensors 120 can acquire data that is processed by the media controller application 140 in order to compute an amount of mind wandering experienced by the user as a mental state metric. Similar to the cognitive load computations described above, the amount of mind wandering can be determined based on pupil sensor data or other data using time series-based techniques, machine learning techniques, etc. In some embodiments, the amount of mind wandering experienced by a user can be determined by comparing the amount of a user’s eye saccades and fixations that have changed relative to the user’s baseline. In some embodiments, fewer eye saccades than the baseline can be detected, indicating that the user is staring. In some embodiments, more eye saccades than the baseline can be detected, indicating that the user is daydreaming. In other embodiments, when a user’s eyes diverge to infinity, the eye fixation angle of the eyes is the same and the eye fixation direction of the eyes is parallel, indicating that the user is focusing on something very far away, mind wandering can be identified. In still other embodiments, the amount of mind wandering can be determined by identifying certain brain wave patterns in neural activity sensor data, such as electroencephalogram (EEG) data. For example, an increase in theta (4 Hz to 7 Hz) or delta (2 Hz to 3.5 Hz) EEG data can be associated with mind wandering. For example That is

[0028] ​​​In some implementations, the media controller application 140 can process sensor data acquired via one or more sensors 120 to determine a mental state metric representing the combination of cognitive load, emotional load, and / or mental wandering. Any technically feasible combination can be calculated, such as: an additive combination, where a weighted sum of the measures of cognitive load, emotional load, and / or mental wandering is used as the mental state metric; a machine learning-based combination, where the measures of cognitive load, emotional load, and / or mental wandering are input into a machine learning model that outputs an overall mental state metric; and so on.

[0029] When determining mental state metrics, the media controller application 140 can also remove the mental state metric component caused by the entertainment media content. This avoids modifying the playback of media content when the media content itself is causing the user to experience high cognitive load, high emotional load, and / or high mental wandering. In some implementations, the media controller application 140 can determine when a specific value of the mental state metric is caused by environmental factors rather than the media content itself, and in such cases, ignore the mental state metric. For example, the media controller application 140 can receive sensor data from one or more sensors mounted on the vehicle, such as cameras, lidar (light detection and ranging), radar, etc., and use such sensor data to identify complex driving situations that are causing high cognitive workload. In this case, the media controller application 140 can determine that the high cognitive workload is not caused by the media content itself. In other implementations, the media controller application 140 can use... For example Speech-to-text and / or sentiment analysis is used to analyze media content or associated metadata to determine the value of a mental state metric caused by the media content item. In this case, the media controller application 140 can remove (the current value of the mental state metric) For example The values ​​of mental state measures not caused by media content are determined by subtracting the contribution of mental state measures caused by media content items.

[0030] In some embodiments, one or more sensors 120 may include optical sensors, such as RGB cameras, infrared cameras, depth cameras, and / or camera arrays, said camera arrays comprising two or more such cameras. Other optical sensors may include imagers and laser sensors. In some embodiments, one or more sensors 120 may include physical sensors that record a user's body position and / or movement, such as touch sensors, pressure sensors, position sensors (…). Example For exampleAccelerometers and / or inertial measurement units (IMUs), motion sensors, etc. In this case, the media controller application 140 can analyze the acquired sensor data to determine the user's motion and then correlate this motion with the cognitive load, emotional load, and / or mental wandering experienced by the user.

[0031] In some implementations, one or more sensors 120 may include physiological sensors, such as heart rate monitors, electroencephalography (EEG) systems, radio sensors, thermal sensors, and skin conductance sensors. Example Sensors that measure changes in skin resistance caused by emotional stress, non-contact sensor systems, magnetoencephalography (MEG) systems, etc. In various implementations, the media controller application 140 can perform spectral entropy, weighted average frequency, bandwidth, and / or spectral edge frequencies to determine cognitive load based on the acquired sensor data.

[0032] In some embodiments, one or more sensors 120 may include acoustic sensors, such as microphones and / or microphone arrays that acquire sound data. Such sound data may be processed by the media controller application 140 using various natural language (NL) processing techniques, sentiment analysis, and / or speech analysis to determine the semantics of spoken phrases in the environment and / or infer emotional parameterization measures based on the semantics. In another example, the media controller application 140 may use speech tone analysis to analyze the acquired sound data to infer emotional load and / or cognitive load based on the speech signals included in the sound data. In some embodiments, the media controller application 140 may perform various analysis techniques related to the spectral centroid frequency and / or amplitude of the sound signal to determine the cognitive load, emotional load, and / or mental wander experienced by the user based on the sound signal.

[0033] In some implementations, one or more sensors 120 may include one or more environmental sensors (such as cameras), thermal imaging sensors, infrared sensors, ultrasonic sensors, lidar sensors, radar sensors, vehicle instrument sensors, etc. For example, in a driving context, sensor data may be acquired via cameras and / or other sensors mounted on the vehicle for detecting other vehicles, external events, weather events, etc. The media controller application 140 may estimate the cognitive load, emotional load, and / or mental wandering experienced by the user based on the sensor data acquired by the environmental sensors. Returning to the driving example, when the media controller application 140 identifies a complex driving situation or abnormal weather based on vehicle sensor data, the media controller application 140 may determine that the user is experiencing high cognitive load.

[0034] In some implementations, one or more sensors 120 may include behavioral sensors that detect user activity within an environment. Such behavioral sensors may include means for acquiring relevant activity data, such as means for acquiring application usage data, mobile device usage data, and / or data associated with user and environment interactions. In this context, the media controller application 140 can estimate the cognitive load, emotional load, and / or mental wandering experienced by the user by determining the activity the user is currently engaged in. For example, a given activity may be categorized as an enjoyable social activity that the user would engage in when happy and active. In this case, the media controller application 140 can correlate the use of a given application with predefined emotions (…). For example Example This data can be correlated with (e.g., arousal) and / or predefined parametric measures of emotion (high arousal value and positive valence value). As another example, in a driving context, behavioral sensors can include various sensors that detect vehicle control inputs (such as how the steering wheel is turned or how the accelerator / brake pedal is pressed). Such sensor data can also be correlated with the cognitive load, emotional load, and / or mental wandering experienced by the user.

[0035] One or more I / O devices 130 may include devices capable of receiving input, such as a keyboard, mouse, touchscreen, microphone, and other input devices for providing input data to computing device 110. In various embodiments, one or more I / O devices 130 may include devices capable of providing output, such as a display screen, speaker, etc. One or more of the I / O devices 130 may be incorporated into computing device 110 or may be external to computing device 110. In some embodiments, computing device 110 and / or one or more I / O devices 130 may be components of an advanced driver assistance system.

[0036] Network 150 enables communication between computing device 110 and other devices via wired and / or wireless communication protocols, including Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi, cellular networks, satellite networks, vehicle-to-vehicle (V2V) networks, and / or Near Field Communication (NFC). As described, media controller application 140 can access remote data storage 152, such as cloud storage, via network 150. In some implementations, network 150 can also be used to retrieve sensor data from other computer sources. For example (Traffic congestion data, weather data, etc.). In this case, the media controller application 140 can also use sensor data retrieved via the network 150 to calculate a mental state metric, which is used to determine whether the playback of media content should be modified.

[0037] ExampleA view of a passenger compartment 200 of a vehicle according to various embodiments is shown. In some embodiments, the passenger compartment 200 may correspond to the environment associated with system 100. As shown, the passenger compartment 200 includes, but is not limited to, an instrument panel 210, a windshield 220, and a host unit 230. In various embodiments, the passenger compartment 200 may include any number of additional components to achieve any technically feasible functionality. For example, the passenger compartment 200 may include a rearview camera (not shown).

[0038] As shown in the figure, the main unit 230 is located in the center of the instrument panel 210. In various embodiments, the main unit 230 can be installed anywhere within the passenger compartment 200 in any technically feasible manner without obstructing the windshield 220. The main unit 230 can include any number and type of instruments and applications and can provide any number of input and output mechanisms. For example, the main unit 230 can enable the user ( Figure 2 The driver and / or passengers can control the entertainment functions. In some embodiments, the host unit 230 may include navigation functions and / or advanced driver assistance systems (ADAS) designed to improve driver safety, automate driving tasks, etc.

[0039] The host unit 230 supports any number of input and output data types and formats, as known in the art. For example, the host unit 230 may include built-in Bluetooth for hands-free calling and / or audio streaming, Universal Serial Bus (USB) connectivity, voice recognition, rearview camera input, video output for any number and type of displays, and any number of audio outputs. Typically, any number of sensors ( For example One or more sensors 120), displays, receivers, transmitters, etc., can be integrated into the host unit 230, or they can be implemented externally to the host unit 230. In various embodiments, external devices can communicate with the host unit 230 in any technically feasible manner.

[0040] While driving, the vehicle driver will be subject to the primary task ( For exampleVarious stimuli related to (guiding the vehicle) and / or any number of secondary tasks. For example, the driver may see lane markings 240, cyclists 242, police cars 244, and / or pedestrians 246 through the windshield 220. In response, the driver may maneuver the vehicle to follow lane markings 240 while avoiding cyclists 242 and pedestrians 246, and then apply the brake pedal to allow police cars 244 to cross the road ahead of the vehicle. Furthermore, the driver may simultaneously or intermittently engage in conversation 250, listen to music 260, and / or attempt to soothe a crying baby 270; these are examples of secondary tasks. As described, different driving environments can cause the driver of a vehicle to be distracted and unable to focus on secondary tasks, such as entertaining media content.

[0041] Media controller application

[0042] For example More detailed illustrations are provided based on various implementation schemes. Figure 3 The media controller application 140 includes a mental state measurement module 142, a distracted mental state event detector module 144, a media bookmarking / buffering and playback controller module 146, and a media playback module 148, as shown in the figure. The mental state measurement module 142 receives sensor data from one or more sensors 120 and determines mental state measures indicative of cognitive load, emotional load, and / or mental wandering experienced by the user based on the received sensor data. In embodiments, the mental state measurement module 142 may calculate regression or categorical values ​​of the mental state measures. For example, the mental state measurement module 142 may use pupil-based measurement techniques to calculate specific cognitive load values, such as regression values. As another example, the mental state measurement module 142 may use a machine learning classifier to categorize the cognitive load experienced by the user as high, medium, or low, or use some other set of categories for categorization.

[0043] Given a mental state metric determined by the mental state measurement module 142, the distraction-high mental state event detector module 144 identifies time periods when a user may be distracted from entertainment media content, such as periods of high cognitive load and / or emotional load, or periods with high mental wandering. In some embodiments, the distraction-high mental state event detector module 144 determines when a user is distracted by comparing a mental state metric value calculated by the mental state measurement module 142 during media content playback (continuously or periodically) with a threshold indicating high cognitive load, high emotional load, and / or high mental wandering. The distraction-high mental state detector module 144 then instructs the media bookmarking / buffering module 146 to store the identified playback time as a bookmark.

[0044] When the mental state measurement module 142 determines that the mental state measurement meets the aforementioned threshold, the media bookmarking / buffering and playback controller module 146 maintains a set of bookmarks indicating recent and / or historical playback times. In some embodiments, playback times may be stored as time codes (such as timestamps) in the memory 114 of the computing device 110 or elsewhere. Figure 1 Example (Stored in data storage area 152 accessible via network 150). Any suitable metadata, such as metadata about file names and / or URLs associated with media content, can also be stored in bookmarks. For example This metadata can be useful when switching from one media file to another occurs during periods of high cognitive load, high emotional load, and / or high mental wandering. In some implementations, the media bookmarking / buffering module 146 can also store one or more portions of the media content between defined times in a buffer for later playback. The buffer can also be located in the memory 114 of the computing device 110 or elsewhere. For example (In data storage area 152).

[0045] In some implementations, the media bookmarking / buffering and playback controller module 146 may be used in conjunction with the media playback module 148 to: (1) automatically pause and resume playback of media content based on a mental state metric that meets a threshold and a playback time that stops meeting the threshold, the playback time of which may optionally be stored in a bookmark; (2) automatically rewind the media content and start playback from the playback time when the mental state metric meets the threshold (as indicated in the stored bookmark); or (3) process user requests to rewind the media content and start playback from such playback time when the mental state metric meets the threshold (as indicated in the stored bookmark).

[0046] The media playback module 148 controls the multimedia application 145, such as a media player, which receives a command to start playing media from a certain time or a time-shifted media stream from a buffer. In some implementations, playback may also include intelligent time compression, which speeds up the playback of the buffered media content. Returning to the example of live broadcasting, if the user is distracted for 20 seconds, the live audio stream ( For example (News) may be paused and stored in a buffer during periods of distraction. Then, when the user is no longer distracted, the media playback module 148 can read the stored audio from the buffer and play it at a faster speed than normal. That isThe stored audio is played back (compressed). The live audio stream continues to be stored in the buffer, and the previously stored audio is read out and played back at a faster speed until the audio catches up with the current time of the live broadcast, at which point the media playback module 148 switches to unbuffered playback of the live audio stream.

[0047] In some implementations, the media controller application 140 can be integrated into a multimedia player, such as an audio player or a video player. In this case, the media playback module 148 can perform the functions of the multimedia player.

[0048] Figure 4 The following are shown according to various implementation schemes: Figure 3 An example vehicle system 400 with a media controller application 140 is shown. As illustrated, the vehicle system 400 includes a sensing module 420, a host unit 230, a network 150, and an output module 440. The sensing module 420 includes a driver-facing sensor 422, a cabin-side non-driver-facing sensor 424, and a vehicle sensor 426. The host unit 230 includes an entertainment subsystem 412, a navigation subsystem 414, a network module 416, and an advanced driver assistance system (ADAS) 418. The output module 440 includes a display 442 and a speaker 444.

[0049] The sensing module 420 includes various types of sensors, including a driver-oriented sensor 422. For example Cameras, motion sensors, etc.), and 424 non-driver-facing sensors in the cabin. For example Motion sensors, pressure sensors, temperature sensors, etc.) and vehicle sensors ( For example (e.g., outward-facing cameras, accelerometers, etc.). In various implementations, the sensing module 420 provides a combination of sensor data, which describes a more detailed observation of the user's mental state in the context of the situation.

[0050] In various embodiments, vehicle sensor 426 may also include other external sensors. Such external sensors may include optical sensors, acoustic sensors, road vibration sensors, temperature sensors, etc. In some embodiments, the sensing module and / or network module 416 may acquire other external data, such as geographic location data. For example GNNS systems, including Global Positioning System (GPS), GLONASS, Galileo, etc. In some implementations, navigation data and / or geolocation data can be combined to predict changes in mental state metrics based on anticipated driving conditions. For example, anticipated traffic congestion might cause the media controller application 140 to predict an increase in mental state metrics as the vehicle approaches the affected area.

[0051] Network module 416 receives and transmits data via network 150. In some embodiments, network module 416 retrieves sensor data from sensor module 420. In various embodiments, network module 416 may retrieve specific values, such as sensed data 462, connected vehicle data 464, and / or historical data. Figure 5 Previous mental state measurements, calculation results from remote devices, etc.

[0052] In some implementations, network module 416 may transmit data acquired by host unit 230, such as one or more mental state measurements, and / or sensing data 462 acquired by sensor module 420. In this case, one or more devices connected to network 150 may combine data received from network module 416 with data from other vehicles and / or infrastructure before it is consumed by the computing module. For example, one or more devices may accumulate and compile sensing data to correlate driving conditions with mental state metrics. For example, one or more devices may accumulate multiple mental state metric calculations into a set of aggregated measurements of attention or engagement, which can then be used to calibrate a threshold that, when met by a mental state metric, indicates that a user is distracted from entertainment media content. In some implementations, baselineization may also be performed to personalize such a threshold, which defaults to a threshold more appropriate for a particular user. For example, each time a user enters a vehicle, a mental state metric may be calculated, and these values ​​may be aggregated over time to determine the user's baseline and a threshold indicating that particular user is distracted from entertainment media content.

[0053] Figures 1 to 4 This is a flowchart of methodological steps for playing media content based on mental state metrics, according to various implementation schemes. Although relative to... Figure 1 The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of various implementations.

[0054] As shown in the figure, method 500 begins with step 502, where the media controller application 140 receives sensor data. (As described above...) For example The aforementioned method can acquire various sensor data and use it to determine mental state measures representing the cognitive load, emotional load, and / or mental wandering experienced by the user. For example, sensor data may include biometric data related to the user (…). For example Heart rate, respiratory behavior, brain activity, skin conductance, blood oxygenation, pupil size, eye fixation and saccades, skin conductance response, blood pressure level, mean blood glucose concentration, etc.) and / or data related to the user's ongoing activities. For example(Speech recorded via an acoustic sensor). Alternatively, sensor data may include data relating to objects in the environment that are not the user.

[0055] In step 504, while the user is consuming media content, the media controller application 140 calculates a mental state metric based on sensor data. The media controller application 140 may continuously or periodically measure the user's mental state while consuming media content. Figure 1 (Calculates mental state metrics every second).

[0056] In some implementations, the media controller application 140 calculates a mental state metric by analyzing relevant portions of sensor data and estimating user cognitive load, emotional load, mental wandering, or any combination thereof not caused by the media content itself. For example, as described above... For example The media controller application 140 can perform various pupil measurement techniques on the received image data to determine fluctuations in the user's pupils in indicative of cognitive load and / or mental wandering. Specifically, the media controller application 140 can employ various time-series-based or machine learning-based techniques to determine cognitive load and / or mental wandering experiences based on pupil fluctuation data. As another example, the media controller application 140 can determine mental wandering by comparing changes in the user's eye saccades and fixations relative to a baseline, identifying when eye divergence reaches infinity, and / or identifying certain EEG patterns in neural activity sensor data. As yet another example, the media controller application 140 can determine the user's emotional load based on facial expressions, visual cues, voice tone and other audio cues, physiological signals, etc., extracted from sensor data. For example, the media controller application 140 can perform various facial expression estimation techniques on the received image data to determine parameterized arousal and valence values ​​in a specific emotion or measure of emotional load that the user is experiencing. The media controller application 140 can also combine estimates of the user's cognitive load, emotional load, and / or mental wandering. For example, in some implementations, additive combinations or machine learning-based combinations may be used.

[0057] As described, the mental state metric component caused by the media content itself can also be removed. In some embodiments, the media controller application 140 can determine, based on sensor data, that the mental state metric value is caused by the environment, rather than by the media content itself. In other embodiments, the media controller application 140 can analyze the media content or associated metadata to determine the mental state metric value caused by the media content at various times. In this case, the media controller application 140 can subtract the mental state metric value caused by the media content from the current value of the mental state metric to determine the mental state metric value that is not caused by the media content.

[0058] In step 506, the media controller application 140 determines that a mental state metric meets a threshold. As previously stated, the threshold is a value that is met ( Figure 6 When a mental state metric exceeds a certain threshold, it indicates that the user is experiencing high cognitive load, high emotional load, and / or high mental wandering. In some implementations, baselineization can be performed to personalize the default threshold to a more accurate threshold for a specific user. For example, if the mental state metric indicates cognitive load, the media controller application 140 can determine that the value of the mental state metric exceeds a threshold indicating that the user is unable to focus on the entertainment media content. As another example, if the mental state metric indicates emotional load, the media controller application 140 can determine that the value of the mental state metric exceeds a threshold indicating that the user is experiencing a high level of emotion that may distract the user from the entertainment media content. As yet another example, if the mental state metric indicates high mental wandering, the media controller application 140 can determine that the value of the mental state metric exceeds a threshold indicating that the user's mind is wandering too much to focus on the entertainment media content. Furthermore, the mental state metric may include a combination of cognitive load, emotional load, and / or mental wandering, in which case, a combination exceeding a threshold may indicate that the user is unable to focus on the entertainment media content.

[0059] Although this document describes the content primarily in relation to thresholds, in alternative implementations, the media controller application 140 may determine whether playback of media content should be paused, resumed, or otherwise modified based on mental state metrics without using thresholds. For example, suppose the mental state metrics are determined using a machine learning classifier and indicate low, medium, or high cognitive load. In this case, the media controller application 140 may pause the media content when the machine learning classifier outputs a high cognitive load value.

[0060] In step 508, the media controller application 140 pauses playback of the media content. In some embodiments, the media controller application 140 may control the multimedia application 145 to pause playback of the media content. In other embodiments, the media controller application 140 may be included as part of the multimedia application 145 and directly pause playback of the media content.

[0061] In step 510, the media controller application 140 optionally stores a portion of the media content in a buffer while pausing playback of the media content. For example, the media content could be a live broadcast. In this case, the media controller application 140 could store portions of the live broadcast during distraction periods (and after the distraction periods until playback catches up with the current time of the live broadcast) in the buffer for later playback, ensuring that the user does not miss any live broadcasts.

[0062] In step 512, the media controller application 140 determines that the mental state metric no longer meets the threshold. Then, in step 514, the media controller application 140 resumes playback of the media content. In some embodiments, the media controller application 140 may control the multimedia application 145 to resume playback of the media content. In other embodiments, the media controller application 140 may be included as part of the multimedia application 145, in which case the media controller application 140 may directly resume playback of the media content. In alternative embodiments, similar to the discussion above in conjunction with step 506, the media controller application 140 may resume playback of the media content based on a mental state metric without comparing the mental state metric to a threshold.

[0063] In some implementations, playback of media content can be resumed from a buffer where a portion of the media content is stored. Returning to the example of live broadcasting, media controller application 140 can resume playback of a live broadcast from a buffer storing a portion of the live broadcast when the broadcast is paused. In this case, media controller application 140 can also temporally compress the playback of the content stored in the buffer while continuing to store the live broadcast in the buffer until playback is synchronized with the current time of the live broadcast, at which point media controller application 140 switches to unbuffered playback of the live broadcast.

[0064] Figures 1 to 4 This is a flowchart of method steps for playing media content based on mental state metrics, according to various alternative implementation schemes. Although relative to... Figure 5 The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of various implementations.

[0065] As shown in the figure, method 600 begins at step 602, where the media controller application 140 receives sensor data. Similar to the combination above... Figure 5 Step 502 of the described method 500 can acquire various sensor data and use them to determine mental state measures representing the cognitive load, emotional load, and / or mental wandering of the user.

[0066] In step 604, while the user is consuming media content, the media controller application 140 calculates a mental state metric based on sensor data. Similar to the above combination... Figure 5In step 504 of the described method 500, the media controller application 140 can calculate a mental state metric by analyzing relevant portions of sensor data and estimating user cognitive load, emotional load, mental wandering, or any combination thereof, that is not caused by the media content itself. In some implementations, the mental state metric component caused by the media content itself can be removed by analyzing the media content or associated metadata, determining whether environmental conditions are causing a specific value of the mental state metric based on sensor data, etc.

[0067] In step 606, the media controller application 140 determines that the mental state metric meets a threshold. Step 606 is similar to the combination of the above. Figure 3 The method 500 is described in step 506. Then, in step 608, the media controller application 140 stores the playback time at which the mental state metric begins to meet the threshold. In some embodiments, the media controller application 140 may store the playback time and any associated metadata in bookmarks, as described above. Figure 5 As described. In conjunction with the above. For example Contrary to step 508 of method 500, the media controller application 140 does not pause the media content. In an alternative implementation, the media controller application 140 may store playback time based on a mental state metric without comparing the mental state metric to a threshold. For example, similar to the discussion above in conjunction with step 506 of method 500, the media controller application 140 may store playback time when a machine learning classifier categorizes the user's cognitive load as high.

[0068] In step 610, the media controller application 140 optionally stores a portion of the media content in a buffer, starting from a playback time when a mental state metric satisfies a threshold. As described, the media content can be... For example In the case of live broadcasts, the media controller application 140 can store a portion of the live broadcast in a buffer for later playback, ensuring that the user does not miss any live broadcasts.

[0069] In step 612, the media controller application 140 determines that the mental state metric no longer meets the threshold. Then, in step 614, the media controller application 140 stores another playback time when the mental state metric no longer meets the threshold. Similar to step 608, the media controller application 140 may store the other playback time and any associated metadata in a bookmark. In an alternative implementation, the media controller application 140 may store other playback times without comparing the mental state metric to the threshold, similar to the discussion of step 506 in conjunction with method 500 above. For example, the media controller application 140 may store playback times when the machine learning model classifies the user's cognitive load as low or medium.

[0070] In step 616, the media controller application 140 optionally receives a user request to rewind media content. For example, a user may realize they have missed a portion of entertaining media content and request to rewind it. The user can request to rewind media content in any technically feasible manner. For example, the media controller application 140 may provide a graphical user interface including buttons for rewinding media content. As another example, the media controller application 140 may accept verbal or gestural commands to rewind media content. In some implementations, the media controller application 140 may also send a user (…) For example The option to rewind media content to the playback time stored in step 808 is provided via a displayed notification or audio notification.

[0071] In step 618, the media controller application 140 rewinds the media content to the playback time stored in step 608 and plays the media content again. In some embodiments, the media controller application 140 can control the multimedia application 145 to rewind and replay the media content. In other embodiments, the media controller application 140 can be integrated into the multimedia application 145 and can directly rewind and replay the media content.

[0072] In some implementations, rewinding and replaying media content may include playing a portion of the media content stored in a buffer. Returning to the example of live broadcasting, the media controller application 140 may begin playing a stored portion of the live broadcast from the time when a mental state metric meets a threshold. In this case, after the mental state metric stops meeting the threshold, the media controller application 140 may temporally compress the playback of the stored content from the buffer while continuing to store the live broadcast in the buffer until playback is synchronized with the current time of the live broadcast, at which point the media playback module 148 switches to unbuffered playback of the live broadcast.

[0073] Although this document is primarily described with reference to driving as a reference example, the techniques disclosed herein can also be applied to other situations where a user is consuming media content as a secondary task while performing a primary task. For example, the primary task might be work, exercise, housework, rowing, etc. In this case, the media controller application can determine a mental state metric associated with the user and modify the playback of media content based on that metric. For instance, a user exercising outdoors might experience high cognitive and / or emotional load due to passing vehicles. In this case, the media controller application running on a mobile device carried by the user or a wearable device worn by the user can modify the playback of media content based on sensor data ( For example(Heart rate monitor data, EEG data, etc.) to identify high cognitive and / or emotional loads. Then, the media controller application can... ​ The following methods can be used to modify the playback of media content that a user is currently enjoying: rewind and replay a portion of the media content at a later time when the user's cognitive and / or emotional load has decreased, or pause and resume the media content.

[0074] Although this document describes media content primarily with respect to uninterrupted playback, in other embodiments, the media controller application may also consider interruptions in media content playback. For example, a user parking their car may experience high cognitive and / or emotional loads that distract them from entertaining media content, which may be reflected in a mental state metric calculated by the media controller application 140. Additionally, playback of the media content may cease once the vehicle is parked and turned off. In this case, the media controller application 140 may store (1) the playback time associated with the mental state metric meeting a threshold indicating user distraction, and (2) another playback time when media content playback is interrupted due to the vehicle being turned off. Subsequently, when the user reopens the vehicle, the media controller application 140 may resume playback of the media content from the time the mental state metric exceeds the threshold, provided that the current value of the mental state metric does not exceed the threshold. However, if the current value of the mental state metric exceeds the threshold, the media controller application 140 may wait until the mental state metric drops below the threshold before resuming playback of the media content.

[0075] Although this document describes primarily with respect to a media controller application 140 running on a single computing device 110, in other embodiments, modification of media content playback can be achieved through multiple applications running on different computing devices. For example, the techniques disclosed herein can be applied to modify media content playback when media content is "transferred" between different playback devices, such as when media content played via a vehicle entertainment system is transferred to a home entertainment system after the user arrives home. In this case, one or more playback times when a user-associated mental state metric meets and / or stops meeting a threshold, and / or portions of the media content between such playback times, can be stored by the vehicle media controller application at an external location, such as in a data storage area 152 accessible via network 150. Another media controller application in the home entertainment system can then retrieve one or more of the stored playback times and / or portions of the media content and use said one or more playback times and / or portions of the media content to rewind and replay or restore the playback of the media content.

[0076] In summary, the media controller application receives sensor data and modifies the playback of media content based on the user's mental state while performing another task within the environment. In implementations, various sensors acquire sensor data associated with the user and / or the environment and send the sensor data to the media controller application. The media controller application determines a metric indicating the user's associated mental state based on the sensor data. The mental state metric may indicate the cognitive load the user is experiencing, the emotional load the user is experiencing, the user's mental wandering, or any combination thereof. After calculating the mental state metric, the media controller application analyzes the mental state metric to determine whether the playback of media content should be modified to take the user's mental state into account. In some implementations, the playback of media content may be modified by: (1) pausing the playback of media content when the mental state metric meets a threshold, and (2) resuming the playback of media content when the mental state metric no longer meets the threshold. In other implementations, the playback of media content may be modified by: (1) storing the playback times when the mental state metric begins to meet the threshold and stops meeting the threshold, and (2) automatically or in response to user requests rewinding and replaying media content based on the stored playback times.

[0077] Compared to existing technologies, at least one technical advantage of the disclosed technology is that it can rewind and replay, or pause and resume media content based on a determined user's mental state. Compared to conventional technologies that rewind media content in fixed time increments, the disclosed technology can more accurately rewind media content to the point in time when the user became distracted by the entertainment media content. Furthermore, the disclosed technology can automatically pause playback of media content when the user becomes distracted and resume playback when the user is no longer distracted. These technical advantages represent one or more technical improvements superior to existing methods.

[0078] 1. In some implementations, a computer-implemented method for playing media content includes: determining a first mental state measure associated with a user accessing the media content based on sensor data; and modifying the playback of the media content based on the first mental state measure.

[0079] 2. The computer-implemented method according to Clause 1, wherein modifying the playback of the media content comprises: rewinding the media content and replaying it starting from a time point in time when the first mental state satisfies a threshold.

[0080] 3. The computer-implemented method according to Clause 1 or 2, wherein modifying the playback of the media content comprises: pausing the playback of the media content in response to the first mental state metric satisfying a threshold; and resuming the playback of the media content in response to the first mental state metric not satisfying the threshold.

[0081] 4. The computer-implemented method according to any one of clauses 1 to 3, further comprising: storing a portion of the media content starting from a first time when the first mental state metric satisfies the threshold; and playing a temporally compressed version of the portion of the media content when the playback of the media content is resumed.

[0082] 5. A computer-implemented method according to any one of Clauses 1 to 4, wherein determining the first mental state metric comprises: determining a second mental state metric associated with the user accessing the media content; and removing the second mental state metric from the first mental state metric.

[0083] 6. The computer-implemented method according to any one of Clauses 1 to 5, wherein the first mental state measure includes at least one of a cognitive load state measure, an emotional load state measure, or a mental wandering measure.

[0084] 7. The computer-implemented method according to any one of clauses 1 to 6, wherein the first mental state measure is determined using a pupil-based measurement technique.

[0085] 8. The computer-implemented method according to any one of clauses 1 to 7, wherein the first mental state measure is determined using a machine learning-based technique.

[0086] 9. The computer-implemented method according to any one of Clauses 1 to 8, wherein the media content includes either audio content or video content.

[0087] 10. In some embodiments, one or more computer-readable storage media include instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps to play media content: determining a first mental state metric associated with a user accessing the media content based on sensor data; and modifying the playback of the media content based on the first mental state metric.

[0088] 11. One or more computer-readable storage media as described in Clause 10, wherein modifying the playback of the media content comprises: rewinding the media content and replaying it starting from a point in time when a first mental state satisfies a threshold.

[0089] 12. One or more computer-readable storage media as described in Clause 10 or 11, wherein the rewinding and replaying of the media content is performed in response to a user request.

[0090] 13. One or more computer-readable storage media according to any one of Clauses 10 to 12, the step further comprising: determining the threshold based on historical mental state data associated with the user.

[0091] 14. One or more computer-readable storage media according to any one of clauses 10 to 13, wherein modifying the playback of the media content comprises: pausing the playback of the media content in response to the first mental state metric satisfying a threshold; and resuming the playback of the media content in response to the first mental state metric not satisfying the threshold.

[0092] 15. One or more computer-readable storage media according to any one of clauses 10 to 14, the step further comprising: storing a portion of the media content starting from a first time when the first mental state metric satisfies the threshold; and playing a temporally compressed version of the portion of the media content when the playback of the media content is resumed.

[0093] 16. One or more computer-readable storage media according to any one of clauses 10 to 15, wherein determining the first mental state metric comprises: determining a second mental state metric associated with the user accessing the media content; and removing the second mental state metric from the first mental state metric.

[0094] 17. One or more computer-readable storage media according to any one of Clauses 10 to 16, wherein the first mental state measure includes at least one of a cognitive load state measure, an emotional load state measure, or a mental wandering measure.

[0095] 18. One or more computer-readable storage media according to any one of clauses 10 to 17, wherein at least one of pupil measurement-based techniques or machine learning-based techniques is used to determine the first mental state measure.

[0096] 19. In some embodiments, an apparatus includes: a memory that includes instructions; and a processor coupled to the memory, which, upon execution of the instructions, determines a first mental state metric associated with a user accessing media content based on sensor data; and modifies playback of the media content based on the first mental state metric.

[0097] 20. The apparatus of claim 19, wherein the processor, when executing the instructions, further determines a second mental state measure based on environment-related data included in the sensor data; and removes the second mental state measure from the first mental state measure.

[0098] Any combination of any element of any claim and / or any element described in this application, in any form, falls within the intended scope of this invention and protection.

[0099] Various implementation schemes have been described for illustrative purposes and are not intended to be exhaustive or limited to the disclosed schemes. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described schemes.

[0100] Various aspects of this disclosure may be implemented as systems, methods, or computer program products. Therefore, aspects of this disclosure may take the form of entirely hardware implementations, entirely software implementations (including firmware, resident software, microcode, etc.), or implementations combining software and hardware aspects, all of which are generally referred to herein as “modules,” “systems,” or “computers.” Furthermore, any hardware and / or software techniques, processes, functions, components, engines, modules, or systems described in this disclosure may be implemented as circuits or collections of circuits. Additionally, aspects of this disclosure may take the form of computer program products embodied in one or more computer-readable media on which computer-readable program code is embodied.

[0101] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination of the foregoing media. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing media. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store programs for use with or in connection with an instruction execution system, device, or apparatus.

[0102] Various aspects of this disclosure are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When the instructions are executed via the processor of the computer or other programmable data processing apparatus, it enables the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such processors may be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions presented in the blocks may not occur in the order presented in the drawings. For example, depending on the functionality involved, two blocks shown consecutively may be executed substantially simultaneously, or sometimes the blocks may be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs the specified function or action.

[0104] Although the foregoing describes embodiments of this disclosure, other and more embodiments of this disclosure may be conceived without departing from its essential scope, as defined by the appended claims.

Claims

1. A computer-implemented method for playing media content, the method comprising: acquiring sensor data via one or more sensors that perform measurements related to at least one of a user or a user environment; determining, based on the sensor data, a first mental state metric associated with the user during playback of media content; determining a first component of the first mental state metric that is caused by the user engaging with the media content; removing the first component of the first mental state metric from the first mental state metric to produce a second component of the first mental state metric; and modifying the playback of the media content based on the second component of the first mental state metric.

2. The computer-implemented method of claim 1, wherein modifying the playback of the media content comprises: rewinding the media content and replaying from a point in time at which the first mental state metric satisfies a threshold.

3. The computer-implemented method of claim 1, wherein modifying the playback of the media content comprises: suspending the playback of the media content in response to the second component of the first mental state metric satisfying a threshold; and resuming the playback of the media content in response to the second component of the first mental state metric not satisfying the threshold.

4. The computer-implemented method of claim 3, further comprising: storing a portion of the media content that begins at a first time at which the second component of the first mental state metric satisfies the threshold; and playing a time-compressed version of the portion of the media content when resuming the playback of the media content.

5. The computer-implemented method of claim 1, further comprising: determining a second mental state metric associated with the user environment; and modifying the playback of the media content based on the second mental state metric.

6. The computer-implemented method of claim 1, wherein the first mental state metric comprises at least one of a cognitive load state metric, an emotional load state metric, or a mind wandering metric.

7. The computer-implemented method of claim 1, wherein the first mental state metric is determined using a technique based on pupil measurements.

8. The computer-implemented method of claim 1, wherein the first mental state metric is determined using a technique based on machine learning.

9. The computer-implemented method of claim 1, wherein the media content comprises one of audio content or video content.

10. One or more computer-readable storage media comprising instructions that, when executed by one or more processors, cause the one or more processors to play media content by performing the following steps: determining, based on sensor data, a first mental state metric associated with a user during playback of media content; determining a first component of the first mental state metric that is caused by the user engaging with the media content; removing the first component of the first mental state metric from the first mental state metric to produce a second component of the first mental state metric; ​ ​ ​ ​ and modify the playback of the media content based on the second component of the first mental state metric.

11. The one or more computer-readable storage media of claim 10, wherein modifying the playback of the media content comprises: rewind the media content and replay from a point in time at which the second component of the first mental state metric satisfies a threshold.

12. The one or more computer-readable storage media of claim 11, wherein the rewinding and replaying of the media content is performed in response to a user request.

13. The one or more computer-readable storage media of claim 11, the steps further comprising: determine the threshold based on historical mental state data associated with the user.

14. The one or more computer-readable storage media of claim 10, wherein modifying the playback of the media content comprises: pausing the playback of the media content in response to the second component of the first mental state metric satisfying a threshold; and resuming the playback of the media content in response to the second component of the first mental state metric not satisfying the threshold.

15. The one or more computer-readable storage media of claim 14, the steps further comprising: storing a portion of the media content from a first time at which the first mental state metric satisfies the threshold; and playing a time-compressed version of the portion of the media content when resuming the playback of the media content.

16. The one or more computer-readable storage media of claim 10, the steps further comprising: determining a second mental state metric associated with the user environment; and modifying the playback of the media content based on the second mental state metric.

17. The one or more computer-readable storage media of claim 10, wherein the first mental state metric comprises at least one of a cognitive load state metric, an emotional load state metric, or a mind wandering metric.

18. The one or more computer-readable storage media of claim 10, wherein the first mental state metric is determined using at least one of a pupil-based measurement technique or a machine learning-based technique.

19. An apparatus for playing media content, comprising: a memory comprising instructions; and a processor coupled to the memory and, when executing the instructions: determining a first mental state metric associated with a user during playback of media content based on sensor data; determining a first component of the first mental state metric caused by the user engaging with the media content; removing the first component of the first mental state metric from the first mental state metric to produce a second component of the first mental state metric; and modifying the playback of the media content based on the second component of the first mental state metric.

20. The apparatus of claim 19, wherein the processor, when executing the instructions, further: determining a second mental state metric based on data associated with an environment included in the sensor data; and modifying the playback of the media content based on the second mental state metric.

Images