Emotional / cognitive state trigger record

Abstract

This describes how emotional / cognitive state-triggered recording is described. A buffer is used to temporarily store captured video content until a change in the user's emotional or cognitive state is detected. Sensor data indicating a change in the user's emotional or cognitive state triggers the creation of a current content video segment based on the buffer.

Description

[0001] Related application citation

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2017 / 031841, international application date of May 10, 2017, entry into the Chinese national phase date of November 16, 2018, Chinese national application number 201780030728.0, and invention title "Emotional / Cognitive State Trigger Record". Technical Field

[0003] Embodiments of this disclosure relate to the recording of emotional / cognitive states. Background Technology

[0004] Digital media has made it increasingly easy for people to capture meaningful moments in their lives through photos or videos. However, often, the meaningful moment has already begun before the user realizes it's the moment they wanted to capture. Furthermore, users are often so focused on capturing important moments that they spend more time trying to take photos or videos than enjoying them.

[0005] Furthermore, while social media apps make it easy for users to share their emotions with others (e.g., posting emoticons in response to another user's post), the prevalence of social and digital media has done little to improve face-to-face user interactions. Summary of the Invention

[0006] This disclosure describes a technique for triggering recording of emotional / cognitive states. In one example, one or more sensors collect sensor data, while a camera captures video content into a buffer. Based on the sensor data, the user's emotional or cognitive state is determined. Once a change in the user's emotional or cognitive state is detected, a video segment is created based on the video content currently in the buffer. Additional video content can be captured and added to the video segment, for example, until another change in the user's emotional or cognitive state is detected. In this way, video segments are created that correspond to the time period during which the user experiences a change in emotional or cognitive state. For example, when watching a child's sports event, if the child scores a point, making the user (parent) excited, happy, or proud, then a video segment including the minutes before and after the child's score would be created and stored.

[0007] This disclosure also describes an emotion / cognitive state presentation system that enables users to dynamically and automatically share their emotion or cognitive state with another user using an electronic device. For example, two users with a trusting relationship can each use a device configured to present their emotion / cognitive state. Sensor data associated with each user can be collected and used to determine the current emotion or cognitive state of the respective user. The data is then shared and presented so that each user can know the other user's current emotion or cognitive state.

[0008] This summary is provided to introduce some concepts in a simplified manner, which are further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to aid in determining the scope of the claimed subject matter. The term "technology" may refer, for example, to a system, method, computer-readable instructions, module, algorithm, hardware logic, and / or operation, as permitted throughout the specification and the context described above. Attached Figure Description

[0009] The specific embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numerals of the reference numerals identify the drawing in which the reference numerals first appear. In different drawings, the same reference numerals indicate similar or identical items.

[0010] Figure 1 This is a schematic diagram of an example environment in which an emotional / cognitive state trigger recording system can be implemented.

[0011] Figure 2 This is a block diagram illustrating an example component of an emotional / cognitive state trigger recording system.

[0012] Figure 3 This is a flowchart of an example method for performing emotional / cognitive state trigger recording.

[0013] Figure 4 This is a schematic diagram of an example environment in which an emotional / cognitive state presentation system can be implemented.

[0014] Figure 5 This is a block diagram illustrating example components of an emotional / cognitive state representation system.

[0015] Figure 6 This is a flowchart of an example method for performing emotional / cognitive state presentation.

[0016] Figure 7 This is a block diagram illustrating an example system that uses gaze tracking technology to identify objects.

[0017] Figures 8A-8C The illustration shows example details of identifying the object of gaze by tracking an individual's gaze.

[0018] Figures 9A-9F Describe an example technique for obtaining calibration data for gaze tracking.

[0019] Figures 10A-10F Describe example techniques used to process calibration data and other data to identify gaze targets.

[0020] Figure 11 This is a flowchart illustrating an aspect of an example process for determining a gaze target from gaze tracking data.

[0021] Figure 12 This is a schematic diagram of an example computer architecture that can be used to identify objects using gaze tracking technology.

[0022] Figure 13 This is a schematic diagram of an example distributed computing environment that uses gaze tracking technology to identify aspects of objects.

[0023] Figure 14 This is a schematic diagram of another example computing device architecture that can be used to identify objects using gaze tracking technology. Detailed Implementation

[0024] Overview

[0025] This paper describes a technique for triggering recording based on emotional / cognitive states. In the example implementation described herein, a wearable camera continuously records video to a buffer. Any number of sensors of various types can be used to determine the user's emotional or cognitive state. Once a specific emotional or cognitive state is detected, or once the level of an emotional or cognitive state exceeds a threshold, the contents of the buffer and additional recorded video content are saved as a video segment corresponding to a moment meaningful to the user. As used herein, emotional states can include, but are not limited to, happiness, sadness, anger, fear, disappointment, or pride. Similarly, cognitive states can include, but are not limited to, focus, engagement, distraction, boredom, drowsiness, confusion, or frustration.

[0026] This document also describes techniques for presenting emotional / cognitive states. In an example implementation, individuals with devices configured to present emotional / cognitive states can interact with each other, such that each user's emotional / cognitive state is presented to the other user. Any number of sensors of various types can be used to determine a user's emotional or cognitive state. If another user is within proximity and is using a device also configured to present the user's emotional / cognitive state, then each user's device enables the corresponding user's emotional / cognitive state to be presented to the other user. In the example implementation, the other user's emotional / cognitive state is presented as a halo around the user within augmented reality, which is viewed, for example, through a head-mounted display device.

[0027] Figure 1 The illustration shows an example environment 100 in which emotion / cognitive state-triggered recording can be implemented. Example environment 100 includes an emotion / cognitive state-triggered recording system 102, which includes a sensing device 104, a camera 106, a microphone 108, and a recording device 110. In the example implementation, any combination of the sensing device 104, camera 106, microphone 108, and recording device 110 can be implemented as a single device. For example, the sensing device 104, camera 106, microphone 108, and recording device 110 can be implemented as a single wearable device, such as a head-mounted display (HMD) device 112.

[0028] In another device, sensing device 104, camera 106, microphone 108, and / or recording device 110 can be implemented as multiple devices. For example, sensing device 104 can be implemented as one or more devices that can be worn or carried by user 114. For example, the sensor can be a component of HMD device 112, cellular phone 116, and any number of other devices. As another example, camera 106 and microphone 108 can be implemented as components of a device worn or carried by user 114. For example, camera 106 and microphone 108 can be components of HMD device 112 or cellular phone 116. As yet another example, recording device 110 can be implemented as any combination of components of HMD device 112, cellular phone 116, and / or computing device 118. For example, sensing device 104, camera 106, and microphone 108 can be communicatively connected to recording device 110. Any combination of HMD device 112, cellular phone 116, and computing device 118 can communicate with each other via, for example, Bluetooth or other short-range wireless communication protocols or via network 120.

[0029] In at least one example, sensing device 104 can be any device or combination of devices configured to physiologically monitor user 114. Individual sensors of one or more sensing devices 104 may include, but are not limited to: a skin conductance sensor for measuring skin conductance; a skin temperature sensor for measuring the temperature of a surface area of ​​the skin; an electroencephalogram (EEG) device for measuring electrical activity of the brain; an electrocardiogram (ECG or EKG) device for measuring electrical activity of the heart; a camera for tracking eye movements, facial expressions, pupil dilation and / or constriction, etc.; and a sound sensor for measuring speech volume, speech rate, etc. In example implementations, sensor data may include measurements associated with physiological attributes of user 114, which may be indicators of emotional or cognitive states.

[0030] In one example, sensing device 104 is part of a specific device or is built into a specific device. For example, such as Figure 1 As shown, the HMD device 112 may include a camera sensor 104A and a skin conductance sensor 104B associated with the bridge of the nose assembly of the HMD device 112.

[0031] The user of the emotional / cognitive state trigger recording system 102 can activate the system to capture video segments based on the user's emotional / cognitive state detected, for example, via a user interface or hardware switch. Figure 1 In the illustrated example, timeline 122 represents user activity throughout a day in the example. In the illustrated example, the user attends a work meeting between 9:00 AM and 10:00 AM, and attends her son's baseball game between 5:30 PM and 7:00 PM. At approximately 9:10 AM, a colleague announces that all company employees will receive bonuses next week. This announcement evokes feelings of happiness in the user, triggering the emotional / cognitive state trigger recording system 102 to record a video segment 124 based on happiness. Later, in the same meeting, a heated discussion about errors made in processing customer orders evokes feelings of anger in the user. The emotional / cognitive state trigger recording system 102 detects the user's anger and, in response, records a video segment 126 based on anger.

[0032] At 5:30, the user attends her son's baseball game. When her son hits the ball for the first time, the user becomes excited. The emotional / cognitive state triggering recording system 102 detects the user's excitement and, in response, records a video segment 128 based on the excitement. Later in the game, the user's son hits a home run, making the user proud of her son. The emotional / cognitive state triggering recording system 102 detects the user's feeling of pride and, in response, records a video segment 130 based on pride. Still later in the game, the user's son collides with another athlete and falls to the ground, clearly in pain. This scene evokes feelings of worry in the user. The emotional / cognitive state triggering recording system 102 detects the user's worry and, in response, records a video segment 132 based on worry.

[0033] At the end of the day, users can review various video clips recorded throughout the day. In some examples, the video clips also include metadata, which may include, for example, indications of the detected emotional / cognitive state that triggered the recording, emotional / cognitive state indicators that occurred or were periodic during the video clip, and / or overlays of dots, highlights, or other visual indicators of the location the user was viewing while the video was being recorded. In one example, any one or more components of the available metadata can be selectively visible when viewing the recorded video clips.

[0034] Figure 2 The illustration shows selectable components of an emotion / cognitive state triggered recording system 102 that can be implemented on a single device or distributed across multiple devices, such as HMD device 112, cellular phone 116, and computing device 118. An example emotion / cognitive state triggered recording system 102 includes a buffer 202, a sensor data analysis module 204, a recording module 206, a gaze detection module 208, and a video segment repository 210.

[0035] Buffer 202 is configured to store video and audio data when received from camera 106 and microphone 108. In one example, buffer 202 is implemented as a five-minute circular buffer.

[0036] Sensor data analysis module 204 receives sensor data from sensing device 104 and analyzes the received sensor data to determine the user's emotional or cognitive state. In the example implementation, the received sensor data indicates the emotional or cognitive state. In an alternative implementation, a deep neural network (DNN) is used, for example, to analyze the sensor data to determine the user's emotional or cognitive state. Sensor data analysis module 204 forms available data indicating the user's emotional or cognitive state.

[0037] The recording module 206 determines whether to record a video segment based on data indicating the user's emotional or cognitive state. In one example, the recording module 206 can be configured to begin recording a video segment based on a change in the user's emotional or cognitive state. For example, the recording module 206 can be configured to begin recording a video segment when the user's emotional or cognitive state changes from a neutral state to a non-neutral state. In another example, the recording module 206 can be configured to begin recording a video segment when the value representing the user's emotional or cognitive state exceeds a threshold.

[0038] The recording module 206 also determines when to stop recording a specific video segment. For example, the recording module 206 may record the video segment based on a predetermined time period, a change in the user's emotional or cognitive state, or a combination of both. For instance, when the recording module 206 begins recording a video segment, it may allow recording to continue for a predetermined time period (e.g., 10 minutes). Alternatively, the recording module 206 may allow recording to continue until sensor data indicates that the user's emotional or cognitive state has changed or decreased below a threshold. In another example, the predetermined time period may indicate the minimum or maximum length of the video segment to be recorded. In this example, if the predetermined time period indicates a minimum recording time, then the recording module 206 may allow the video segment to be recorded until the predetermined time period is met, or until the user's emotional or cognitive state changes, whichever occurs later. If the predetermined time period indicates a maximum recording time, then the recording module 206 may allow the video segment to be recorded until the predetermined time period is met, or until the user's emotional or cognitive state changes, whichever occurs first.

[0039] The gaze detection module 208 tracks the user's gaze to determine the direction the user is looking. The gaze detection module 208 can be configured to generate a video overlay that includes, for example, colored dots, highlights, or other visual indicators related to the direction the user is looking at while the video is being captured.

[0040] Video segment repository 210 is configured to store video segments recorded based on the user's emotional or cognitive state. For example, when recording module 206 begins recording, video and audio data stored in buffer 202 are copied to a new video segment, which is then stored in video segment repository 210. Additionally, recording module 206 directs additional video and audio data to be recorded to the aforementioned video segment. In the example implementation, video segment repository 210 also stores metadata associated with the video segments, which may include, but is not limited to: indicators of the user's emotional or cognitive state that triggered recording; one or more indicators of the user's emotional or cognitive state while the video segment is being recorded; and overlays of visual indicators (e.g., dots or highlights) to indicate the location of the user's gaze while the video is being recorded.

[0041] Figure 3This diagram illustrates an example method for performing emotional / cognitive state trigger recording. The example process is illustrated as a set of blocks in a logic flowchart, representing a series of operations that can be implemented in hardware, software, or a combination thereof. Blocks are referenced by numbers. In the context of software, a block represents computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processing units (such as hardware microprocessors), perform the recorded operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order of the described operations is not intended to be construed as restrictive, and any number of described blocks can be combined in any order and / or in parallel to implement the process.

[0042] Figure 3 The illustration shows an example method 300 for performing emotion / cognitive state-triggered recording. In box 302, video / audio is recorded to a buffer. For example, when camera 106 and microphone 108 capture video / audio data, the emotion / cognitive state-triggered recording system 102 writes the video / audio data to buffer 202. As described above, buffer 202 can be implemented as a circular buffer, such that buffer 202 continuously stores data from the most recent time period.

[0043] In box 304, sensor data is received. For example, sensor data analysis module 204 receives data from one or more sensing devices 104. As described above, sensor data may include, but is not limited to: data indicating skin conductance, data indicating skin temperature, data indicating brain electrical activity, data indicating heart electrical activity, data indicating eye movement, data indicating facial expression, data indicating pupil dilation and / or constriction, data indicating speech volume, or data indicating speech speed.

[0044] In box 306, the received sensor data is analyzed to determine the user's emotional or cognitive state. For example, the sensor data analysis module determines the user's emotional or cognitive state based on the received sensor data. For example, the sensor data analysis module uses a deep neural network to analyze the received sensor data to determine the user's emotional or cognitive state.

[0045] In box 308, it is determined whether data should be recorded. For example, based on the determined emotional or cognitive state of the user, recording module 206 determines whether to start recording a video segment. For example, as described above, recording module 206 may determine to start recording a video segment based on a change in the user's emotional or cognitive state or based on the determination that a value representing the user's emotional or cognitive state exceeds a threshold.

[0046] If it is determined that data should not be recorded (from the "No" branch of box 308), then processing continues as described above with reference to box 302.

[0047] On the other hand, if it is determined that data should be recorded (from the "Yes" branch of box 308), then in box 310, a video segment is created. For example, recording module 206 creates a video segment by saving the contents of buffer 202 to video segment repository 210. Additional data, such as indicators of the user's current emotional or cognitive state and / or indicators of the current location of the user's gaze, may also be stored in association with the video segment. In the example implementation, the direction of the user's gaze can be determined by gaze detection module 208.

[0048] In block 312, video / audio is recorded to a video segment. For example, instead of recording video / audio to a buffer, or in lieu of recording video / audio to a buffer, recording module 206 causes additional captured video / audio to be recorded to the video segment created as described above with reference to block 310.

[0049] In box 314, it is determined whether recording should be stopped. For example, as described above, recording module 206 can determine that recording should stop after a predetermined time period and / or based on changes in the user's emotional / cognitive state. For example, while recording video, sensor data analysis module 204 can continue to analyze the received sensor data, which can identify changes in the user's emotional / cognitive state (e.g., a return to neutrality).

[0050] If it is determined that recording should be stopped (from the "Yes" branch of box 314), then recording to the video segment stops, and processing continues as described above with reference to box 302.

[0051] On the other hand, if it is determined that recording should not be stopped (from the "No" branch of block 314), then processing continues as described in reference box 312 above, where video / audio data continues to be recorded to the video segment.

[0052] Emotional / cognitive state presentation

[0053] Figure 4 The illustration shows an example environment 400 in which emotional / cognitive state presentation can be implemented. Example environment 400 includes an emotional / cognitive state presentation system 402 that enables emotional / cognitive state data to be shared between devices associated with two or more users. In the illustrated example, a first sensing / presentation device 404 is associated with a first user 406, and a second sensing / presentation device 408 is associated with a second user 410.

[0054] In an example implementation, user 406 chooses to share data indicating her emotional / cognitive state via device 404. Similarly, user 410 chooses to share data indicating his / her emotional / cognitive state via device 408. Sensors associated with device 404 indicate the emotional / cognitive state of user 406, while sensors associated with device 408 indicate the emotional / cognitive state of user 408. Devices 404 and 408 can communicate directly with each other, for example, via Bluetooth or via an emotional / cognitive state presentation system 402 over a network.

[0055] When devices 404 and 408 are determined to be within proximity of each other, data indicating the emotional / cognitive state of user 406 is shared with device 408, and data indicating the emotional / cognitive state of user 410 is shared with device 404. Upon receiving data indicating the emotional / cognitive state of another user, sensing / presentation device 408 presents an indication of the emotional / cognitive state of user 406. In an example implementation, sensing / presentation device 408 provides an augmented reality view element 412 that includes a visual halo 414 surrounding user 406. For example, different colored halos can be used to indicate different emotional or cognitive states. Any number of other techniques can be used to indicate the emotional or cognitive state of another user, including but not limited to audio indicators, text indicators, or visual indicators.

[0056] Example emotion / cognitive state presentation system 402 includes sensing device 414 and presentation device 416. As described in the non-limiting example, devices 404 and 408 are implemented as a single device including sensing device 414 and presentation device 416.

[0057] Similar to the reference above Figure 1 As described, sensing device 414 can be implemented as one or more devices that can be worn or carried by a user. For example, the sensor can be a component of an HMD device (such as device 404 or device 408), or it can be implemented as a component of a cellular phone or any number of other devices.

[0058] In at least one example, sensing device 414 can be any device or combination of devices configured to physiologically monitor a user. Individual sensors of one or more sensing devices 414 may include, but are not limited to: a skin conductance sensor for measuring skin conductance; a skin temperature sensor for measuring the temperature on the surface of the skin; an electroencephalogram (EEG) device for measuring electrical activity of the brain; an electrocardiogram (ECG or EKG) device for measuring electrical activity of the heart; a camera for tracking eye movements, facial expressions, pupil dilation and / or constriction, etc.; and a sound sensor for measuring speech volume, speech rate, etc. In example implementations, sensor data may include measurements associated with the user's physiological attributes, which may be indicators of emotional or cognitive states.

[0059] Presentation device 416 is configured to present to a user an indication of another user's emotional or cognitive state. Figure 4 In the example illustrated, HMD device 408 includes a display for presenting augmented reality, which may include visual indicators of another user's emotional or cognitive state. In an alternative implementation, presentation device 416 may be implemented as a component of any other device, such as a smartphone.

[0060] Figure 5 The diagram illustrates the selection components of an emotion / cognitive state presentation system. (Example:) Figure 5 As shown, the example emotion / cognitive state presentation system 402 includes a device proximity detection module 502, a sensing device 504, a sensor data analysis module 506, emotion / cognitive state sharing parameters 508, a communication interface 510, and an output device 512.

[0061] The device proximity detection module 502 is configured to detect another device within proximity. For example, refer to Figure 4 The device proximity module can determine when devices 404 and 408 are within a predetermined proximity range. In various examples, proximity can be determined based on Global Positioning System (GPS) data, Bluetooth availability, user input, etc.

[0062] Sensing device 504 can be any device or combination of devices configured to monitor a user's physiological state. Individual sensors of one or more sensing devices 504 may include, but are not limited to: a skin conductance sensor for measuring skin conductance; a skin temperature sensor for measuring the temperature of the skin surface; an electroencephalogram (EEG) device for measuring electrical activity of the brain; an electrocardiogram (ECG or EKG) device for measuring electrical activity of the heart; a camera for tracking eye movements, facial expressions, pupil dilation and / or constriction, etc.; and a sound sensor for measuring speech volume, speech rate, etc. In the example implementation, sensor data may include measurements associated with the physiological attributes of user 406 or 410, which may be indicators of emotional or cognitive states.

[0063] Sensor data analysis module 506 receives sensor data from sensing device 504 and analyzes the received sensor data to determine the user's emotional or cognitive state. In the example implementation, the received sensor data indicates the emotional or cognitive state. In an alternative implementation, a deep neural network (DNN) is used, for example, to analyze the sensor data to determine the user's emotional or cognitive state. Sensor data analysis module 506 forms usable data indicating the user's emotional or cognitive state.

[0064] The emotion / cognitive state sharing parameter 508 can define, for example, the level of trust between users sharing emotion / cognitive state data and other users. The emotion / cognitive state sharing parameter 508 can include default and / or user-specified parameters associated with emotion / cognitive state sharing. For example, a user can select specific emotion and / or cognitive states that can be shared, while specifying others as private (not shareable). As another example, a user can select specific users with whom they are willing to share emotion / cognitive state data and / or specific users with whom they are unwilling to share emotion / cognitive state data.

[0065] Communication interface 510 is configured to facilitate the sharing of emotional / cognitive state data between two user devices within close proximity to each other. Communication interface 510 may include logic for verifying the level of trust between the devices (e.g., based on emotional / cognitive state sharing parameter 508 or input from the user), and logic for facilitating the transmission of data between devices using Bluetooth protocol or other network interfaces.

[0066] Output device 512 is configured to present data indicating the emotional or cognitive state of another user. For example, output device 512 may include, but is not limited to, a display device configured to allow a real-world view of objects on a surface to be displayed via hardware, while also providing a rendered display of computer-generated content or scenes (e.g., a halo around an individual).

[0067] Figure 6 This diagram illustrates an example method for performing the presentation of emotional / cognitive states. The example process is illustrated as a set of boxes in a logic flowchart, representing a series of operations that can be implemented in hardware, software, or a combination thereof. These are referenced by numbers. In the context of software, a block represents computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processing units (such as hardware microprocessors), perform the described operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order of the described operations is not intended to be construed as restrictive, and any number of described blocks can be combined in any order and / or in parallel to implement the process.

[0068] Figure 6 The illustration shows an example method 600 for performing emotional / cognitive state presentation. In block 602, the proximity of another device configured to share an emotional / cognitive state is detected. For example, device proximity detection module 502 detects the proximity of another device.

[0069] In block 604, sensor data is received. For example, sensor data analysis module 506 receives data from one or more sensing devices 504. As described above, sensor data may include, but is not limited to: data indicating skin conductance, data indicating skin temperature, data indicating brain electrical activity, data indicating heart electrical activity, data indicating eye movement, data indicating facial expression, data indicating pupil dilation and / or constriction, data indicating speech volume, or data indicating speech speed.

[0070] In box 606, the user's emotional or cognitive state is determined. For example, sensor data analysis module 506 determines the user's emotional or cognitive state based on received sensor data. For example, sensor data analysis module 506 uses a deep neural network to analyze the received sensor data to determine the user's emotional or cognitive state.

[0071] In box 608, data indicating the emotional / cognitive state is shared. For example, an indication of the user's current emotional or cognitive state is sent to another device via communication interface 510. In the example implementation, communication interface 510 verifies the trust relationship between the user and the user on the other device before sending the indication of the user's current emotional or cognitive state. For example, the trust relationship is verified based on the emotional / cognitive state sharing parameter 508.

[0072] In box 610, data indicating the emotional / cognitive state of another user is received. For example, data indicating the emotional / cognitive state of another user is received via communication interface 510.

[0073] In box 612, another user's emotional / cognitive state is presented. For example, output device 512 can provide a real-world view of an object through a hardware display surface, while also providing a rendered display of computer-generated content or scenes (e.g., a halo around an individual).

[0074] Example Architecture

[0075] Figure 7 This is a block diagram illustrating an example system 700 for recording video based on or for presenting emotional or cognitive states. System 700 includes a computing device 702, which can be used to perform at least some of the operations to determine a user's emotional or cognitive state and to present the determined emotional or cognitive state or to record video based on the determined emotional or cognitive state. The computing device 702 may be associated with an entity that acts as a service provider offering services related to the presentation of emotional / cognitive states or video recording. Additionally, the computing device 702 may be associated with a manufacturer, distributor, or both of an electronic device 706. The computing device 702 may include one or more network interfaces (not shown) for communicating with other computing devices via one or more networks 704. The one or more networks 704 may include one or more of the following: the Internet, a wired network, a satellite network, a wide area wireless communication network, a wired local area network, a wireless local area network, or a public switched telephone network (PSTN).

[0076] In a particular embodiment, computing device 702 may communicate with electronic device 706 associated with individual 708 via one or more networks 704. Electronic device 706 may include laptop computing devices, tablet computing devices, mobile communication devices (e.g., mobile phones), wearable computing devices (e.g., watches, glasses, fitness trackers, head-mounted displays, jewelry), portable gaming devices, combinations thereof, etc. Individual 708 may utilize electronic device 706 to record video or share / present data based on detected emotional / cognitive states of individual 708.

[0077] Computing device 702 may include one or more processors, such as processor 710. One or more processors 710 may include at least one hardware processor, such as a microprocessor. In some cases, one or more processors 710 may include a central processing unit (CPU), a graphics processing unit (GPU), or both a CPU and a GPU, or other processing units. Additionally, one or more processors 710 may include local memory, which may store program modules, program data, and / or one or more operating systems.

[0078] Additionally, computing device 702 may include one or more computer-readable storage media, such as computer-readable storage medium 712. Computer-readable storage medium 712 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Such computer-readable storage medium 712 may include, but is not limited to: RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, solid-state storage devices, disk storage devices, RAID storage systems, storage arrays, network-attached storage devices, storage area networks, cloud storage devices, removable storage media, or any other medium that can be used to store desired information and is accessible by the computing device. Depending on the configuration of computing device 702, computer-readable storage medium 712 may be a tangible computer-readable storage medium and may be a non-transitory storage medium.

[0079] Computer-readable storage medium 712 can be used to store any number of functional components executable by one or more processors 710. In many implementations, these functional components include instructions or programs executable by one or more processors 710 and, when executed, implement operational logic for performing operations belonging to computing device 702. As described herein, functional components of computing device 702 that can be executed on one or more processors 710 for implementing various functions and features related to recording audio / video data based on detected emotional / cognitive states include sensor data analysis module 714, recording module 716, and gaze detection module 718, which respectively correspond to Figure 2 Examples shown include sensor data analysis module 204, recording module 206, and gaze detection module 208. One or more of modules 714, 716, and 718 can be used to implement... Figure 1 and Figure 2 Emotional / cognitive state trigger recording system 102.

[0080] As described herein, functional components of a computing device 702, which can be executed on one or more processors 710 for implementing various functions and features related to the presentation of emotional / cognitive states, include a sensor data analysis module 714 and a device proximity detection module 720, respectively corresponding to Figure 5 Examples shown are sensor data analysis module 504 and device proximity detection module 502. One or more of modules 714 and 720 can be used to implement... Figure 4 and Figure 5 The emotional / cognitive state presentation system 402.

[0081] In various embodiments, one or more functional components of the components of the computing device 702 may be implemented as part of the computing device 702 or as part of an integrated circuit accessible to the computing device 702. For example, the sensor data analysis module may be implemented at least in part as an application-specific integrated circuit (ASIC) dedicated to the execution of a deep neural network (DNN).

[0082] The computing device 702 may also include or be coupled to a data repository 728 and a buffer 730, which may include, but are not limited to, RAM, ROM, EEPROM, flash memory, one or more hard disks, solid-state drives, optical storage (e.g., CD, DVD), or other non-transitory storage technologies. The data repository 728 may maintain information used by the computing device 702 to perform operations related to emotional / cognitive state trigger recording or emotional / cognitive state presentation. For example, the data repository 728 may include emotional / cognitive state information 732, gaze tracking information 734 related to determining an individual's gaze, a video segment repository 736, and / or emotional / cognitive state shared parameters 738.

[0083] The emotional / cognitive state information 732 may include, for example, sensor values ​​corresponding to various emotional and cognitive states.

[0084] The gaze tracking information 734 stored in the data repository 728 may include information for determining an individual's gaze. In some cases, the gaze tracking information 734 may include eye position data for one or more individuals. Additionally, the gaze tracking information 734 may include distances to an individual's facial features, reference points, and other information that may be used to determine an individual's gaze. Optionally, when performing gaze calculations, portions of the gaze tracking information 734 may be cached or otherwise temporarily stored. After performing gaze calculations, at least a portion of the gaze tracking information 734 may be discarded to minimize memory resources used for identifying objects using gaze tracking technology.

[0085] The video segment repository 736 can be configured to store video / audio segments recorded based on the detected emotional or cognitive state of a user and associated metadata. For example, the video segment repository 736 can correspond to... Figure 2 The video segment repository 210 is shown in the image.

[0086] The emotion / cognitive state sharing parameter 738 may include default and / or user-specified parameters associated with emotion / cognitive state sharing. For example, a user may select specific emotion and / or cognitive states that can be shared, while specifying others as private (not shareable). As another example, a user may select specific users with whom they are willing to share emotion / cognitive state data and / or specific users with whom they are unwilling to share emotion / cognitive state data.

[0087] Buffer 730 is configured to store video / audio data when an emotional / cognitive state triggers recording activity. For example, when the emotional / cognitive state-triggered recording system 102 is activated by a user (e.g., via a user interface or via a hardware switch), changes detected in the emotional / cognitive state that would trigger the recording of video segments in the video segment repository 736 are expected, and video / audio data is continuously recorded into buffer 730.

[0088] The sensor data analysis module 714 may include computer-readable instructions executable by the processor 710 to receive sensor data and analyze the received sensor data to determine the user's emotional or cognitive state. In an example implementation, the received data indicates the emotional or cognitive state. In an alternative implementation, a deep neural network (DNN) is used, for example, to analyze the sensor data to determine the user's emotional or cognitive state. The sensor data analysis module 714 makes available data indicate the user's emotional or cognitive state.

[0089] The recording module 716 may include computer-readable instructions executable by the processor 710 to determine whether and when to stop recording video segments based on data indicating the user's emotional or cognitive state.

[0090] The gaze detection module 718 may include computer-readable instructions executable by the processor 710 to obtain data that can be used to determine an individual's gaze path. In some cases, the gaze detection module 718 may obtain data from the electronic device 706, which can be used to determine the gaze of an individual (such as individual 708). For example, the gaze detection module 718 may obtain data indicating the position of at least one eye of the individual. In various implementations, the gaze detection module 718 may obtain an image of at least one eye of the individual and analyze the image to determine the position of the individual's eyes. The individual's eye position can be used to determine the individual's gaze path. In a particular implementation, the individual's eye position can be used to determine the direction the individual is looking in. In some cases, the individual's gaze path may be approximated as a conical or prism-shaped field of view of the scene.

[0091] The gaze detection module 718 can also generate video overlays to be stored as metadata along with the video segments in the video segment repository 736, wherein the video overlays include visible points, highlights, or other visual indicators of an individual's gaze path when the video segment is recorded.

[0092] The device proximity detection module 720 may include computer-readable instructions executable by the processor 710 to detect another device within proximity.

[0093] The computing device 702 may also include a communication interface 740, which is configured to support sharing data with other devices, such as sharing data indicating a user’s emotional or cognitive state.

[0094] The electronic device 706 of system 700 may include a processor 742, a computer-readable storage medium 744, a buffer 746, an input / output device 748, and a communication interface 750. The processor 742 may include a hardware processing unit, such as a central processing unit, a graphics processing unit, a DNN chip, or any combination thereof. In one embodiment, the computer-readable storage medium 744 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Such a computer-readable storage medium 744 may include, but is not limited to: RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, solid-state storage devices, disk storage devices, removable storage media, or any other medium that can be used to store desired information and is accessible by the electronic device 706. Depending on the configuration of the electronic device 706, the computer-readable storage medium 744 may be a tangible computer-readable storage medium and may be a non-transitory storage medium. The electronic device 706 may also include one or more network interfaces (not shown) for communicating with other computing devices via one or more networks 704.

[0095] Buffer 746 can be configured to record video / audio to support video recording triggered by emotional / cognitive states. For example... Figure 7 As shown, buffer 746 may be a component of computing device 702 and / or electronic device 706.

[0096] Input / output device 748 may include one or more sensors. In at least one example, input / output device 748 may include a sensor that may include any device or combination of devices configured to sense the condition of individual 708 or the environment surrounding individual 708. Figure 1The sensing device 104 is shown in the figure. The input / output device 748 may include one or more user-facing cameras or other sensors for tracking a user's eye movements or gaze, facial expressions, pupil dilation and / or constriction, gestures, and / or other characteristics. In some examples, the input / output device 748 may include one or more externally facing or environmental cameras for capturing images of the real-world objects and surroundings of the individual 708, such as those shown above. Figure 1 The described camera 106. The input / output device 748 may additionally or alternatively include one or more biometric sensors (e.g., a skin conductance sensor for measuring skin conductance, a heart rate monitor, a skin temperature sensor for measuring the temperature on the surface of the skin, an electroencephalogram (EEG) device for measuring the electrical activity of the brain, an electrocardiogram (ECG or EKG) device for measuring the electrical activity of the heart), one or more other cameras (e.g., a webcam, an infrared camera, a depth camera, etc.), a microphone (e.g., microphone 108), or a sound sensor, a light sensor, an optical scanner, etc., for measuring speech volume, speech rate, etc.

[0097] Input / output device 748 can output data to one or more modules for appropriate processing, such as sensor data analysis module 752, recording module 754, gaze detection module 756, and proximity detection module 758. For example, a biometric sensor can capture sensor data, which can be processed by sensor data analysis module 752 to determine the user's emotional or cognitive state. Sensor data analysis module 752 can then output indicators of the user's emotional or cognitive state to recording module 754, which can then process audio and video data captured by microphones and cameras to produce video segments to be stored in video segment repository 736.

[0098] As another example, a user-facing camera can capture gaze tracking data, which can be processed by a gaze detection module 756 to determine the gaze path of an individual 708. The gaze detection module 756 can then output the gaze path to a recording module 754 or a computing device 702 to generate metadata that will be stored in association with video segments in a video segment repository 736.

[0099] In additional and / or alternative examples, input / output device 748 may include any device or combination of devices configured to detect the position or movement of electronic device 706 and other objects. For example, input / output device 748 may additionally and / or alternatively include depth map sensors, light field sensors, gyroscopes, sonar sensors, infrared sensors, compasses, accelerometers, global positioning system (GPS) sensors, and / or any other devices or components for detecting the position or movement of electronic device 706 and / or other objects. Input / output device 748 may also support the generation of data characterizing interactions with electronic device 706 (such as user gestures). For illustrative purposes, input / output device 748 may support the generation of data defining aspects of the position and movement of one or more objects (e.g., velocity, direction, acceleration), which may include electronic device 706, physical items proximate to electronic device 706, and / or the user.

[0100] In some implementations, at least some input / output devices 748 may be part of or built into electronic device 706. More specifically, electronic device 706 may include a user-facing camera sensor and / or an environmental camera, which are arranged in or integrated with the nose bridge assembly of electronic device 706. As described above, electronic device 706 may include any configuration of one or more input / output devices 748 that are part of or built into electronic device 706. However, in some examples, one or more input / output devices 748 may be removably coupled to electronic device 706, or may be detached from electronic device 706 and communicatively coupled to electronic device 706. In the latter case, data from input / output devices 748 may be transmitted from input / output devices 748 to electronic device 706, for example, via wired and / or wireless networks (such as network 704).

[0101] Additionally, input / output device 748 may include one or more input interfaces, which may include a keyboard, keypad, mouse, microphone, touch sensor, touchscreen, joystick, control buttons, scroll buttons, camera, neural interface, or any other device suitable for generating signals and / or data defining user interaction with electronic device 706. By way of example, and not limitation, input / output device 748 may include displays (e.g., holographic displays, heads-up displays, protectors, touchscreens, liquid crystal displays (LCDs), etc.), speakers, haptic interfaces, etc.

[0102] In at least one example, the display device of electronic device 706 may include a hardware display surface, which may be configured to provide a real-world view of objects through the hardware display surface, while also providing a rendered display of computer-generated content or scenes (e.g., a halo around an individual). The hardware display surface may include one or more components, such as a projector, a screen, or other suitable components for generating the display of objects and / or data. In some configurations, the hardware display surface may be configured to cover at least one eye of a user. In an illustrative example, the hardware display surface may include a screen configured to cover both eyes of a user. The hardware display surface may draw or cause the display of one or more images for generating a view or stereoscopic image of one or more computer-generated virtual objects. For illustrative purposes, objects may be items, data, devices, people, locations, or entities of any type. In at least one example, objects may be associated with features or functions associated with an application. Some configurations may enable electronic device 706 to graphically associate holographic user interfaces and other graphical elements with objects drawn through the hardware display surface or displayed on the hardware display surface of electronic device 706.

[0103] The hardware display surface of electronic device 706 can be configured to allow individual 708 to view objects from different environments. In some configurations, the hardware display surface can display a drawing of a computer-generated virtual object. Furthermore, some configurations of the hardware display surface allow individual 708 to view objects in his or her surrounding environment through selectable segments of the hardware display surface with a controllable level of transparency. For illustrative purposes, the perspective of individual 708 viewing an object through the hardware display surface may be referred to herein as the object's "real-world view" or "real-world view of the physical object." The computer-generated drawing and / or data of the object can be displayed within, around, or near a selected portion of the hardware display surface, enabling individual 708 to view both the computer-generated drawing and the real-world view of the object through the selected portion of the hardware display surface.

[0104] Some configurations described herein provide both "perspective display" and "augmented reality display". For illustrative purposes, a "perspective display" may include a transparent lens on which content is displayed. An "augmented reality display" may include an opaque display configured to display content over a drawn image, which may originate from any source, such as a video feed from a camera used to capture images of the environment. For illustrative purposes, some examples described herein depict the display of content drawn over a displayed image. Furthermore, some examples described herein depict the display of drawn content over a "perspective display", allowing the user to see a real-world view of objects with content. It is understood that examples of the techniques described herein can be applied to "perspective displays", "augmented reality displays", or variations and combinations thereof. For illustrative purposes, devices configured to support "perspective displays", "augmented reality displays", or combinations thereof are referred to herein as devices capable of providing a "mixed environment" or "mixed reality scene".

[0105] In some implementations, at least a portion of the operations performed by the sensor data analysis module 752 may include operations performed by the sensor data analysis module 714, at least a portion of the operations performed by the recording module 754 may include operations performed by the recording module 716, at least a portion of the operations performed by the gaze detection module 756 may include operations performed by the gaze detection module 718, and at least a portion of the operations performed by the proximity detection module 758 may include operations performed by the device proximity detection module 720, or combinations thereof.

[0106] Communication interface 750 is configured to support data sharing with other devices, such as sharing data indicating a user's emotional or cognitive state. In some implementations, at least a portion of the operations performed by communication interface 750 may include operations performed by communication interface 740.

[0107] Now refer to Figures 8A-8C , Figures 9A-9F , Figures 10A-10F and Figure 11 The following sections describe techniques for detecting gaze paths. These techniques are described in the context of head-mounted computing devices with user-facing gaze-tracking cameras. However, the techniques described in these sections can also be applied to other types of computing devices with user-facing cameras for gaze tracking (e.g., computers with webcams, tablets or smartphones with user-facing cameras, game controllers with user-facing cameras, etc.). Figure 8AThis is a rear view of a device 800 (e.g., HMD device 112, 404, 408, or 706) having one or more hardware display surfaces 802 and one or more displays 804 and 804'. In at least one example, sensor 804' is user-facing and can be configured to track the position of at least one of the user's eyes. Furthermore, at least another sensor 804 can be a scene-facing camera (e.g., camera 106) and can be directed toward a real-world object to generate image data of the real-world object. As will be described in detail below, examples can process eye position data, image data, and other data to identify the user's gaze path. As will also be described below, examples described herein can also determine whether the user is viewing a specific portion of the hardware display surface 802, a specific portion of a real-world object, or a specific portion of a drawn object. This information can be useful for determining a gaze target from gaze tracking data, where the gaze target can be identified in the recorded video segment.

[0108] exist Figure 8A In this device 800, two user-facing sensors 804' are included for generating data or signals indicating the position or movement of at least one of the user's eyes. The sensors 804' may be in the form of a camera or other suitable device for tracking the position or movement of at least one of the user's eyes. The device 800 may also include at least one hardware display surface 802 for allowing the user to view one or more objects. The hardware display surface 802 can provide a view of real-world objects and images of objects that can be drawn and displayed on the hardware display surface 802, as described above.

[0109] Figure 8B yes Figure 8A The image shows a side sectional view of device 800. Figure 8B This includes the user's eyes 808 viewing through the hardware display surface 802. The hardware display surface 802 is configured to create transparent portions, allowing the user to view objects through the hardware display surface 802. Figure 8B An example arrangement is shown in which a real-world object 810 is aligned with a transparent portion of a hardware display surface 802, which allows a user to view the real-world object 810 through the hardware display surface 802. The hardware display surface 802 may display one or more drawn objects. The device 800 also includes at least one sensor 804' guided toward at least one of the user's eyes 808.

[0110] Figure 8CThe illustration depicts an example scene or view element 812 that can be observed by a user via a hardware display surface 802. A thick double line 814 illustrates the boundary of the hardware display surface 802. In this illustrative example, the scene or view element 812 includes a real-world object 816, a first drawn object 818, and a second drawn object 820 displayed on the hardware display surface 802. The real-world object 816 is viewed through the hardware display surface 802.

[0111] Device 800 may utilize one or more techniques for calibrating device 800. The following sections are related to... Figures 9A-9F This section describes in combination various aspects of the techniques used to obtain calibration data. The subsequent sections... Figures 10A-10F This section describes an example scenario in which device 800 processes calibration data and other data to identify the gaze target.

[0112] Device 800 can be calibrated in several ways. In one example, device 800 can utilize a display with several graphic elements at predetermined locations. When displaying graphic elements, device 800 can prompt the user to view a specific graphic element and provide input to verify that the user is viewing the specific graphic element. When the user verifies that he or she is viewing the specific graphic element, sensor 804' can generate eye position data defining the position of at least one eye. The eye position data can be stored in a data structure in memory in response to receiving verification from the user.

[0113] Figure 9A The illustration shows an example view element 900 that can be captured by sensor 804' of device 800. From this viewpoint, device 800 can determine one or more values ​​defining the position of at least one eye 808 of a user. In an illustrative example, the values ​​may include a second value (D2) indicating the distance between the user's eyes and a third value (D3), a fourth value (D4), and a fifth value (D7) indicating the distance between at least one eye of the user and a reference point 902. It is understood that by using one or more image processing techniques, one or more aspects of the eye (such as the pupil) can be identified and utilized to determine the eye position.

[0114] Furthermore, reference point 902 can be selected by utilizing one or more suitable technologies. Reference point 902 can be based on user features (e.g., tip of the nose, eyebrow, beauty mark), or reference point 902 can be located at any position. Figure 9A In the example, the point between the user's eyes is used as reference point 902. This example reference point 902 is provided for illustrative purposes and is not to be construed as limiting. It is understood that reference point 902 can be located at any suitable location, and can be based on identifiable features or characteristics of the user or any object.

[0115] As described above, the device 800 can generate a plurality of graphic elements at predetermined positions on the hardware display surface 802. When the graphic elements are displayed on the hardware display surface 802, the device 800 can prompt the user to view the graphic elements and provide input to verify that the user is viewing the graphic elements. Figure 9B The illustrated graphic element 906 is shown as an example view element 904, which can be generated by device 800 to facilitate the calibration process. In this example, device 800 draws the graphic element 906 at the center of the viewing area. Although the graphic element 906 is displayed, device 800 can generate prompts for the user to verify that he or she is viewing the graphic element 906. The prompts and the user's response to the prompts can include gestures, voice commands, or other suitable input types.

[0116] When device 800 verifies that a user is viewing graphic element 906, device 800 can record one or more values ​​indicating the position and / or movement of at least one of the user's eyes 808. For example, as described above and in... Figure 8B and Figure 9A One or more values ​​shown may be stored in a data structure in memory. It is understood that any suitable value or combination of values ​​may be stored or utilized, including but not limited to: a first value (D1) indicating the distance between sensor 804' and at least one of the user's eyes 808; a second value (D2) indicating the distance between the user's eyes; and other values ​​(D3, D4, and D7) indicating the distance between at least one eye 808 and reference point 902. These values ​​are provided for illustrative purposes and are not to be construed as limiting. It is understood that these values, subsets of these values, and other measured values ​​may be used to determine the movement and / or position of one or more of the user's eyes.

[0117] During the display of other graphic elements in various locations, values ​​from other groups can be measured. For example, such as Figure 9C As shown, in the display as Figure 9D The second graphic element 908 shown can measure the values ​​of the second group (D2', D3', D4', and D7'). For example... Figure 9E As shown, in the display as Figure 9F The third graphic element 910 shown can measure the values ​​of the third group (D2”, D3”, D4” and D7”).

[0118] These example measurements and positions of the graphic elements are provided for illustrative purposes. It will be understood that any number of graphic elements can be placed in different locations to obtain measurements that can be used to calibrate the device 800. For example, the device 800 may sequentially display graphic elements at predetermined positions on the view element 904, such as each corner of the view element 904. As will be understood, more or fewer graphic elements may be used during the calibration process.

[0119] Values ​​indicating the position of at least one eye 808 at each predetermined location can be used to generate calibration data. The calibration data can be configured to associate the eye position dataset with data identifying the positions of graphic elements.

[0120] Any known technique for generating calibration data can be used. It is understood that the generation of calibration data may include extrapolation, projection, and / or estimation techniques that project the correlation between the eye position dataset and various portions and / or pixels of the hardware display surface 802. These examples are provided for illustrative purposes and are not to be construed as limiting, and the values ​​and / or calibration data may be obtained in other ways, including receiving such calibration data from one or more remote resources.

[0121] Once calibration data is generated or obtained, such data or other data can be used by device 800 to determine whether a user is viewing a specific gaze target, which may include a portion of hardware display surface 802, a drawn object, a portion of a drawn object, a real-world object, or a portion of a real-world object. Figures 10A-10F An aspect of an example scenario is described, in which a device 800 having at least one sensor 804' is used to track the movement of at least one eye 808 of a user in order to identify the gaze target.

[0122] Now refer to Figure 10A and Figure 10B This example illustrates and describes a scene showing an identifier indicating the object being viewed. In this example, the user is viewing example view element 812. (Refer to the above...) Figure 8C As outlined, the example view element 812 includes views of objects drawn on a hardware display surface 802 (e.g., a first drawn object 818 and a second drawn object 820) and views of a real-world object 816 through the hardware display surface 802. When a user is viewing the view element 812, the sensor 804' can cause one or more measured values ​​to be generated, such as... Figure 10A The values ​​shown are used. In some examples, using any combination of suitable techniques, these values ​​can be compared with calibration data and / or other data to identify the gaze target. In this example, in Figure 10AOne or more values ​​measured in the scene described can be processed using calibration data to determine if the user is viewing a real-world object 816. In this example, Figure 10A One or more of the measured values ​​shown can also be used to determine a predetermined portion of the interface that the user is viewing, such as Figure 10B The first part 1000 of the hardware display surface 802.

[0123] Continuing with this example, in Figure 10C One or more values ​​measured in the scenario described can be processed using calibration data to determine whether the user is viewing a second drawn object 818. In this example, Figure 10C The values ​​of one or more measurements shown can also be used to determine what the user is watching. Figure 10D The second part 1002 of the hardware display surface 802.

[0124] Continuing with this example, in Figure 10E One or more values ​​measured in the scenario described can be processed using calibration data to determine whether the user is viewing a second drawn object 820. In this example, Figure 10E The values ​​of one or more measurements shown can also be processed using calibration data to determine what the user is watching. Figure 10F The third part 1004 of the hardware display surface 802.

[0125] In some examples, device 800 can use a combination of data from resources to determine whether a user is viewing a second drawn object 820 via hardware display surface 802. As outlined above, a camera or other type of sensor 804 is mounted to device 800. Figure 8A The eye can be guided toward the user's field of view. Image data generated from the camera can be analyzed to determine whether an object in the field of view is located in a predetermined position within the image data. If the object is located within a predetermined area of ​​the image, such as the center of the image, the device can determine the gaze target by processing this data using eye position data. This data can be used to supplement other types of data, such as location data from GPS and / or data generated from a compass or accelerometer, to assist device 800 in determining the gaze direction (e.g., left, right, up, or down) and / or the gaze target.

[0126] Figure 11 This is a flowchart illustrating various aspects of an example process 1100 for determining the gaze target. Figure 11 For convenience, the operation is described in the context of device 800. However, the operation can also be applied to other devices, including but not limited to: HMD device 112, HMD device 404, HMD device 408, or electronic device 706.

[0127] Block 1102 illustrates obtaining calibration data. In at least one example, the operating system, application, or another module may obtain the calibration data. The calibration data may be stored in a data structure of a computer-readable storage medium for later access. The calibration data may be generated by device 800 or received from a remote resource. In some examples, sensors of computing device 800 may be positioned to track the position of at least one of a user's eyes. The sensors may cause the generation of one or more values ​​that associate the position of the user's at least one eye with a specific portion or location on a hardware display surface. This example may utilize an initialization process in which device 800 displays one or more graphic elements at a predetermined location. During the display of one or more graphic elements, one or more inputs from the user may indicate that they are viewing one or more graphic elements. In response to the input, device 800 may generate calibration data including values ​​that associate the position of the user's at least one eye with data identifying a specific location or portion of the hardware display surface.

[0128] Block 1104 illustrates obtaining sensor data indicating the position of at least one of a user's eyes. In at least one example, an operating system, application, or another module may obtain sensor data from one or more sensors. The sensor data may be stored in a data structure within the sensor data collection module or elsewhere in a computer-readable storage medium for later access. As outlined above, sensors guided toward at least one of the user's eyes can result in the generation of sensor data (e.g., gaze tracking data) indicating the position of at least one of the user's eyes. The sensor data can be processed to generate data indicating the user's gaze direction. As described below, calibration data can be used to process the data indicating the user's gaze direction to determine whether the user is looking at a gaze target, which may include a drawn object displayed on a hardware display surface.

[0129] Box 1106 illustrates the acquisition of image data of an object. In at least one example, an operating system, application, or another module associated with a computer-readable medium may acquire the sensor data. The image data or other information about the object may be stored in a data structure within the sensor data acquisition module or elsewhere on the computer-readable storage medium for later access. In some examples, a camera or other type of sensor mounted to or otherwise communicating with computing device 800 may be directed toward the user's field of view. The camera or other type of sensor may cause the generation of image data, which may include one or more images of the object in the user's field of view. The image data may be in any suitable format and generated by any suitable sensor, which may include depth map sensors, cameras, etc.

[0130] Box 1108 illustrates the use of image data or sensor data to determine the gaze target. In at least one example, an operating system, application, or another module associated with a computer-readable medium can determine the gaze target. For example, if a user is viewing a real-world view of an object through a hardware display surface 802, sensors guided toward the user's field of view can generate image data of the real-world object. The image data can be analyzed to determine whether the object in the field of view is located at a predetermined position within the image data. For example, if the object is positioned within a predetermined area of ​​the image, such as the center of the image, then the computing device 800 can determine that the object is the gaze target. In another example, sensor data indicating the position of at least one of the user's eyes (e.g., gaze tracking data) can be processed using calibration data and / or image data to determine whether the user is viewing a drawn object displayed on a hardware display surface. This example can be used to determine that a drawn object displayed on a hardware display surface is a gaze target.

[0131] Figure 12 Additional details are shown of an example computer architecture 1200 for a computer (such as HMD device 112, mobile computing device 116, HMD device 404, HMD device 408, computing device 702, and / or electronic device 706) capable of executing the program components described above for recording or sharing data based on detected user emotional or cognitive states. Therefore, Figure 12 The computer architecture 1200 illustrated in the diagram is an architecture used for server computers, mobile phones, PDAs, smartphones, desktop computers, netbooks, tablets, laptops, and / or wearable computers. Computer architecture 1200 is an example architecture that can be used, wholly or partially, to perform various aspects of the software components presented herein.

[0132] Figure 12 The computer architecture 1200 illustrated includes a central processing unit 1202 (“CPU”), a system memory 1204 including random access memory 1206 (“RAM”) and read-only memory (“ROM”) 1208, and a system bus 1210 coupling the memory 1204 to the CPU 1202. The ROM 1208 stores a basic input / output system (“BIOS”) containing basic routines that facilitate the transfer of information between components within the computer architecture 1200, such as during startup. The computer architecture 1200 further includes storage for an operating system 1214, programs, and modules 1216 (e.g., ...). Figure 1 and Figure 2 Emotional / cognitive state trigger recording system 102, Figure 4 and Figure 5 The emotional / cognitive state presentation system 402, Figure 2 Modules 204, 206, and 208, Figure 5 Modules 502, 506, 508, and 510, and Figure 7 Mass storage device 1212 (modules 714, 716, 718, 720, 728, 752, 754, 756 and / or 758). Additionally and / or alternatively, the mass storage device may store sensor data 1218, image data 1220 (e.g., photographs, computer-generated images, object information about real and / or virtual objects in a scene, metadata about any of the foregoing, etc.), calibration data 1222, content data 1224 (e.g., computer-generated images, videos, scenes, etc.), as described herein.

[0133] Mass storage device 1212 is connected to CPU 1202 via a mass storage controller (not shown) connected to bus 1210. Mass storage device 1212 and its associated computer-readable medium provide a non-volatile storage device for computer architecture 1200. Mass storage device 1212, memory 1204, computer-readable storage medium 712, and computer-readable storage medium 738 are examples of computer-readable media according to this disclosure. Although the description of computer-readable media contained herein refers to mass storage devices (such as solid-state drives, hard disks, or CD-ROM drives), those skilled in the art will understand that computer-readable media can be any available computer storage or communication medium that can be accessed by computer architecture 1200.

[0134] Communication media include computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves or other transmission mechanisms) and include any delivery medium. The term "modulated data signal" refers to a signal having one or more of its characteristics that are altered or set in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of communication media.

[0135] By way of example and not limitation, computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). For example, computer storage media includes, but is not limited to: RAM, ROM, erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other solid-state memory technologies, compressed disc read-only memory (“CD-ROM”), digital multifunction disc (“DVD”), high-definition / density digital multifunction / video disc (“HD-DVD”), blu-ray disc or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computer architecture 1200. For the purposes of the claims, the phrases “computer storage media”, “computer-readable storage media”, and variations thereof do not include communication media.

[0136] Depending on the configuration, computer architecture 1200 can operate in a networked environment using a logical connection to a remote computer via network 1226 and / or another network (not shown). Computer architecture 1200 can be connected to network 1226 via network interface unit 1228 connected to bus 1210. It should be understood that network interface unit 1228 can also be used to connect to other types of networks or remote network systems. Computer architecture 1200 may also include an input / output controller 1230 for receiving and processing input from input devices or input interfaces, and for providing output to output devices or output interfaces.

[0137] It should be understood that the software components described herein, when loaded onto and executed, can transform the CPU 1202 and the overall computer architecture 1200 from a general-purpose computing system to a dedicated computing system tailored to facilitate the functions presented herein. The CPU 1202 can be constructed from any number of transistors or other discrete circuit elements, which can individually or uniformly assume any number of states. More specifically, in response to executable instructions within the software modules described herein, the CPU 1202 can operate as an effective state machine. These computer-executable instructions can transform the CPU 1202 by specifying how it transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the CPU 1202. In some examples, processor 710 and / or processor 742 may correspond to the CPU 1202.

[0138] The encoding software modules presented herein can also transform the physical structure of the computer-readable medium presented herein. In different implementations of this specification, the specific transformation of the physical structure can depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable medium, regardless of whether the computer-readable medium is characterized as a primary or secondary storage device. For example, if the computer-readable medium is implemented as a semiconductor-based memory, then the software described herein can encode on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. The software can also transform the physical state of such components to store data thereon.

[0139] As another example, the computer-readable medium described herein can be implemented using magnetic or optical techniques. In such an implementation, when software is coded therein, the software presented herein can transform the physical state of the magnetic or optical medium. These transformations may include altering the magnetic properties of a specific location within a given magnetic medium. These transformations may also include altering the physical characteristics or properties of a specific location within a given optical medium to change the optical properties of those locations. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification, wherein the foregoing examples provided merely facilitate the discussion.

[0140] In light of the above, it should be understood that many types of physical transformations occur within computer architecture 1200 to store and execute the software components presented herein. It should also be understood that computer architecture 1200 may include other types of computing entities, including handheld computers, embedded computer systems, personal digital assistants, and other types of computing entities known to those skilled in the art. It is also contemplated that computer architecture 1700 may not include… Figure 12 All of the components shown may include Figure 12 Other components not explicitly shown, or those that can be utilized with Figure 12 The architecture shown is completely different from the one described above.

[0141] Figure 13 A distributed computing environment 1300 is described, capable of executing the software components described herein for object identification using gaze tracking technology. Therefore, Figure 13 The distributed computing environment 1300 illustrated in the figure can be used to execute any aspect of the software components presented herein to implement various aspects of the techniques described herein.

[0142] According to various embodiments, the distributed computing environment 1300 includes a computing environment 1302 that operates on, communicates with, or is part of a network 1304. In at least one example, at least some of the computing environments 1300 may correspond to HMD device 112, mobile computing device 116, HMD device 404, HMD device 408, computing device 702, and / or electronic device 706. The network 1304 may be or may include the above-described references. Figure 1 and Figure 7 Networks 120 and 704 are described. Network 1304 may also include various access networks. One or more client devices 1306A-1306N (hereinafter uniformly and / or generally referred to as "client 1306") can access the network via network 1304 and / or other connections. Figure 13 (Not shown in the diagram) communicates with computing environment 1302. By way of example, HMD device 112, mobile computing device 116, HMD device 404, HMD device 408, and / or electronic device 706 may correspond to one or more client devices 1306A-1306Q (collectively, "client 1306"), where Q can be any integer greater than or equal to 1 depending on the desired architecture. In an illustrated configuration, client 1306 includes: computing device 1306A (such as a laptop computer, desktop computer, or other computing device); tablet or writing tablet computing device ("tablet computing device") 1306B; mobile computing device 1306C (such as a mobile phone, smartphone, or other mobile computing device); server computer 1306D; wearable device 1306E, and / or other device 1306N. It should be understood that any number of clients 1306 may communicate with computing environment 1302. References herein Figure 12 and Figure 14 Two example computing architectures for client 1306 are illustrated and described. It should be understood that the illustrated client 1306 and the computing architectures illustrated and described herein are illustrative and should not be construed as limiting in any way.

[0143] In the illustrated configuration, computing environment 1302 includes application server 1308, data storage device 1310, and one or more network interfaces 1312. Depending on the implementation, application server 1308 may be provided by one or more server computers that are part of or communicate with network 1304. In some examples, computing environment 1302 may correspond to or represent... Figure 7One or more computing devices 702 in the network, which communicate with and are accessible by one or more computing devices 706 via network 704 and / or 1304.

[0144] In at least one example, application server 1308 may host various services, virtual machines, portals, and / or other resources. In the illustrated configuration, application server 1308 may host one or more virtual machines 1314 for executing applications or other functions. Depending on various implementations, virtual machine 1314 may execute one or more applications and / or software modules for object identification using gaze tracking technology. Application server 1308 also hosts or provides access to one or more portals, linked pages, websites, and / or other information (“web portals”) 1316. Web portal 1316 may be used to communicate with one or more client computers. Application server 1308 may include one or more email services 1318.

[0145] Depending on the implementation, application server 1308 may also include one or more mailbox messaging services 1320. Mailbox service 1318 and / or messaging service 1320 may include email (“email”) service, various personal information management (“PIM”) services (e.g., calendar service, contact management service, collaboration service), instant messaging service, chat service, forum service and / or other communication services.

[0146] Application server 1308 may also include one or more social networking services 1322. Social networking services 1322 may include various social networking services, including but not limited to: services for sharing or publishing status updates, instant messages, links, pictures, videos, and / or other information; services for commenting on or displaying interest in articles, products, blogs, or other resources; and / or other services. In some configurations, social networking services 1322 are provided by, or include, services such as Facebook® Social Networking Service, LinkedIn® Professional Networking Service, MySpace® Social Networking Service, Foursquare® Geographic Networking Service, and Yammier® Office Colleague Networking Service. In other configurations, social networking services 1322 are provided by other services, sites, and / or providers that may or may not be explicitly referred to as social networking providers. For example, some websites allow users to interact via email, chat services, and / or other tools during various activities and / or contexts (such as reading published articles, commenting on goods or services, posting, collaborating, playing games, etc.). Examples of these services include, but are not limited to, Windows Live® and Xbox Live® services from Microsoft Corporation in Redmond, Washington. Other services are possible and anticipated.

[0147] Social networking service 1322 may also include commenting, blogging, and / or microblogging services. Examples of such services include, but are not limited to: Yelp® commenting service, KUDZU® rating service, OfficeTalk® enterprise microblogging service, Twitter® messaging service, Google Buzz® service, and / or other services. It should be understood that, for the sake of brevity, the above list of services is not exhaustive, and a large number of additional and / or alternative social networking services 1322 are not mentioned herein. Thus, the above configuration is illustrative and should not be construed as limiting in any way. Depending on the implementation, social networking service 1322 may host one or more applications and / or software modules for providing the functionality described herein for providing context-aware location-sharing services for computing devices. For example, any of the application servers 1308 may communicate with or facilitate the functionality and features described herein. For example, social networking applications, email clients, messaging clients, browsers running on a telephone, or any other clients 1806 may communicate with social networking service 1322.

[0148] like Figure 13 As shown, application server 1308 can also host other services, applications, portals, and / or other resources (“other resources”) 1324. Other resources 1324 can deploy service-oriented architectures or any other client-server management software. Therefore, it can be understood that computing environment 1302 can provide integration with various mailboxes, messaging, social networking, and / or other services or resources using the gaze-based object identification concepts and techniques described herein.

[0149] As mentioned above, computing environment 1302 may include data storage device 1310. Depending on the implementation, the functionality of data storage device 1310 may be provided by one or more databases operating on or communicating with network 1304. The functionality of data storage device 1310 may also be provided by one or more server computers configured to host data for computing environment 1302. Data storage device 1310 may include, host, or provide one or more physical or virtual containers 1326A-1326N (uniformly and / or generally referred to as "container 1326"). Although Figure 13 Not shown, but container 1326 can also host or store one or more modules for use by a remote computing device (e.g., Figure 1 and Figure 2 Emotional / cognitive state trigger recording system 102, Figure 2 Modules 204, 206 and / or 208, Figure 4 and Figure 5The emotional / cognitive state presentation system 402, Figure 5 Modules 502, 506, 508, 510 and 512, and / or Figure 7 The modules 714, 716, 718, 720, 752, 754, 756, and 758 execute data structures and / or algorithms. Aspects of repository 1326 may be associated with database programs, file systems, and / or any data storage program with secure access features. Aspects of container 1326 may also be implemented using products or services such as ACTIVE DIRECTORY®, DKM®, ONEDRIVE®, DROPBOX®, or GOOGLEDRIVE®.

[0150] The computing environment 1302 can communicate with or be accessed by the network interface 1312. The network interface 1312 may include various types of network hardware and software to support communication between two or more computing entities, including but not limited to client 1306 and application server 1308. It should be understood that the network interface 1312 can also be used to connect to other types of networks and / or computer systems.

[0151] It should be understood that the distributed computing environment 1300 described herein can provide any number of virtual computing resources and / or other distributed computing functions to any aspect of the software components described herein, which can be configured to execute any aspect of the software components described herein. According to various implementations of the concepts and techniques disclosed herein, the distributed computing environment 1300 provides the software functions described herein as a service to client 1306. It should be understood that client 1306 may include real or virtual machines, including but not limited to: server computers, network servers, personal computers, tablet computers, game controllers, smart TVs, mobile computing entities, smartphones, and / or other devices. Thus, various configurations of the concepts and techniques described herein enable any device configured to access the distributed computing environment 1300 to utilize the functions described herein for providing recording and sharing based on a user's emotional or cognitive state. In a specific example, as outlined above, the techniques described herein can be at least partially derived from... Figure 13 It is implemented by combining the application server 1308 with the browser application.

[0152] Figure 14This is an illustrative computing device architecture 1400 for computing devices capable of executing the various software components described herein. In some examples, computing device architecture 1400 can be used to implement aspects of emotional / cognitive state trigger recording or emotional / cognitive state presentation. Computing device architecture 1400 can be applied to computing entities that facilitate mobile computing in part due to enabling factors, wireless connectivity, and / or battery-powered operation. In some configurations, computing devices include, but are not limited to: mobile phones, tablet devices, tablet devices, wearable devices, portable video game devices, etc. Furthermore, aspects of computing device architecture 1400 can be applied to conventional desktop computers, portable computers (e.g., laptop computers, notebook computers, ultra-portable computers, and netbooks), server computers, and other computing devices. As an example and not a limitation, computing device architecture 1400 can be applied to... Figure 1 , Figure 4 , Figure 7 , Figure 12 and Figure 13 Any of the clients shown in the image.

[0153] Figure 14 The computing device architecture 1400 illustrated includes a processor 1402, a memory component 1404, a network connectivity component 1406, a sensor component 1408, an input / output (“I / O”) component 1410, and a power component 1412. In the illustrated configuration, the processor 1402 communicates with the memory component 1404, the network connectivity component 1406, the sensor component 1408, the input / output (“I / O”) component 1410, and the power component 1412. Although not shown... Figure 14 The diagram illustrates the connections between individual components, but these components can interact to implement device functions. In some configurations, components are arranged to communicate via one or more buses (not shown).

[0154] Processor 1402 includes a central processing unit (“CPU”) configured to process data, execute computer-executable instructions for one or more application programs, and communicate with other components of computing device architecture 1400 to perform the various functions described herein. Processor 1402 can be used to execute aspects of the software components presented herein. In some examples, processor 1402 may correspond to the above references. Figure 7 and Figure 12 The processors 710, 742 and / or CPU 1202 are described.

[0155] In some configurations, processor 1402 includes a graphics processing unit (“GPU”) configured to accelerate operations performed by the CPU, including but not limited to operations performed by general scientific and / or engineering computing applications and graphics-intensive computing applications such as high-resolution video (e.g., 1080i, 1080p, and higher resolutions), video games, three-dimensional (“3D”) modeling applications, etc. In some configurations, processor 1402 is configured to communicate with a discrete GPU (not shown). In some examples, processor 1402 may additionally or alternatively include a holographic processing unit (HPU) specifically designed to process and integrate data from multiple sensors of a head-mounted computing device, and to handle tasks such as spatial mapping, gesture recognition, and voice and speech recognition. In any case, the CPU, GPU, and / or HPU can be configured according to a common CPU, GPU, and / or HPU computing model, wherein processing tasks are divided among the CPU, GPU, and / or HPU according to their respective intensities. For example, the sequential part of the application can be executed on the CPU, the computationally intensive part can be accelerated by the GPU, and certain specialized functions (such as spatial mapping, gesture recognition, and voice and speech recognition) can be executed by the HPU.

[0156] In some configurations, processor 1402 is a system-on-a-chip (“SoC”) along with one or more other components described herein below, or processor 1402 is included in the SoC along with one or more other components described herein below. For example, the SoC may include processor 1402, a GPU, one or more network connectivity components 1406, and one or more sensor components 1408. In some configurations, processor 1402 is fabricated in part using package-on-package (“PoP”) integrated circuit packaging technology. Processor 1402 may be a single-core processor or a multi-core processor.

[0157] Processor 1402 can be built based on an ARM architecture, which is licensed from ARM HOLDINGS in Cambridge, UK. Alternatively, processor 1402 can be built based on an x86 architecture, such as that available from INTELCORPORATION in Mountain View, California. In some configurations, processor 1402 is a SNAPDRAGON SoC available from QUALCOMM in San Diego, California; a TEGRA SoC available from NVIDIA in Santa Clara, California; a HUMMINGBIRD SoC available from SAMSUNG in Seoul, South Korea; an Open Multimedia Application Platform (“OMAP”) SoC available from TEXASINSTRUMENTS in Dallas, Texas; a custom version of any of the above SoCs; or a proprietary SoC.

[0158] Memory component 1404 includes random access memory (“RAM”) 1414, read-only memory (“ROM”) 1416, integrated storage memory (“integrated storage device”) 1418, and removable storage memory (“removable storage device”) 1420. In some configurations, RAM 1414 or a portion thereof, ROM 1416 or a portion thereof, and / or some combination of RAM 1414 and ROM 1416 are integrated into processor 1402. In some configurations, ROM 1416 is configured to store firmware, an operating system or a portion thereof (e.g., an operating system kernel), and / or a bootloader to load the operating system kernel from integrated storage device 1418 or removable storage device 1420. In some examples, memory component 1404 may correspond to the above references respectively. Figure 7 and Figure 12 The computer-readable storage medium 712, computer-readable storage medium 738, and / or memory 1204 are described.

[0159] Integrated storage device 1418 may include solid-state memory, hard disk drive, or a combination of solid-state memory and hard disk drive. Integrated storage device 1418 may be soldered or otherwise connected to a logic board, and processor 1402 or other components described herein may also be connected to the logic board. Thus, integrated storage device 1418 is integrated into a computing device. Integrated storage device 1418 may be configured to store an operating system or a portion thereof, application programs, data, or other software components described herein.

[0160] Removable storage device 1420 may include solid-state storage, hard disk drive, or a combination of solid-state storage and hard disk drive. In some configurations, removable storage device 1420 is provided in place of integrated storage device 1418. In other configurations, removable storage device 1420 is provided as an additional optional storage device. In some configurations, removable storage device 1420 is logically combined with integrated storage device 1418 such that the total available storage is available as the total combined storage capacity. In some configurations, the total combined capacity of integrated storage device 1418 and removable storage device 1420 is displayed to the user, rather than the individual storage capacities of integrated storage device 1418 and removable storage device 1420.

[0161] The removable storage device 1420 is configured to be inserted into a removable storage memory slot (not shown) or other mechanism to facilitate connection, through which the removable storage device 1420 can communicate with other components of a computing device (such as processor 1402). The removable storage device 1420 can be implemented in various memory card formats, including but not limited to: PC cards, compressed flash memory cards, memory sticks, secure digital cards (“SD”), miniSD, microSD, universal integrated circuit cards (“UICC”) (e.g., a Subscriber Identity Module (“SIM”) or Universal SIM (“USIM”), proprietary formats, etc.

[0162] It is understood that one or more memory components in memory component 1404 may store an operating system. Depending on various configurations, the operating system includes, but is not limited to: SYMBIAN OS from SYMBIAN LIMITED, WINDOWS MOBILE OS from Microsoft Corporation in Redmond, Washington, WINDOWS PHONE OS from Microsoft Corporation, WINDOWS from Microsoft Corporation, PALM WEBOS from Hewlett-Packard Company in Palo Alto, California, BLACKBERRY OS from Research In Motion Limited in Waterloo, Ontario, Canada, IOS from Apple Inc. in Cupertino, California, and ANDROID OS from Google Inc. in Mountain View, California. Other operating systems are also anticipated.

[0163] Network connectivity component 1406 includes a wireless wide area network component (“WWAN component”) 1422, a wireless local area network component (“WLAN component”) 1424, and a wireless personal area network component (“WPAN component”) 1426. Network connectivity component 1406 facilitates communication to and from network 1427 or another network, which may be a WWAN, WLAN, or WPAN. Although only network 1427 is described, network connectivity component 1406 can facilitate communication with multiple networks (including...). Figure 14 Simultaneous communication with multiple networks (network 1427). For example, network connectivity component 1406 can facilitate simultaneous communication with one or more networks via WWAN, WLAN, or WPAN. In some examples, network 1427 may correspond to, for example, Figure 1 , Figure 7 , Figure 12 and Figure 13 All or part of the network 120, network 704, network 1226 and / or network 1304 shown.

[0164] Network 1427 may be a WWAN or may include a WWAN, such as a mobile telecommunications network that utilizes one or more mobile telecommunications technologies to provide voice and / or data services to computing devices utilizing computing device architecture 1400 via WWAN component 1422. Mobile telecommunications technologies may include, but are not limited to: Global System for Mobile Communications (“GSM”), Code Division Multiple Access (“CDMA”) ONE, CDMA2000, Universal Mobile Telecommunications System (“UMTS”), Long Term Evolution (“LTE”), and Global Microwave Access Interoperability (“WiMAX”). Furthermore, network 1427 may utilize various channel access methods (which may or may not be used by the aforementioned standards), including but not limited to: Time Division Multiple Access (“TDMA”), Frequency Division Multiple Access (“FDMA”), CDMA, Wideband CDMA (“W-CDMA”), Orthogonal Frequency Division Multiplexing (“OFDM”), Space Division Multiple Access (“SDMA”), etc. Data communications can be provided using General Packet Radio Service (“GPRS”), Evolved Global Enhanced Data Rate (“EDGE”), the High-Speed ​​Packet Access (“HSPA”) protocol family (including High-Speed ​​Downlink Packet Access (“HSDPA”), Enhanced Uplink (“EUL”), or alternatively known as High-Speed ​​Uplink Packet Access (“HSUPA”), Evolved HSPA (“HSPA+”)), LTE, and various other current or future wireless data access standards. Network 1427 can be configured to provide voice and / or data communications using any combination of the above technologies. Network 1427 can be configured or adapted to provide voice and / or data communications according to future generation technologies.

[0165] In some configurations, WWAN component 1422 is configured to provide dual multimode connectivity to network 1427. For example, WWAN component 1422 may be configured to provide connectivity to network 1427, where network 1427 provides services via GSM and UMTS technologies or some other combination of technologies. Alternatively, multiple WWAN components 1422 may be used to perform this function and / or provide additional functionality to support other incompatible technologies (i.e., those that cannot be supported by a single WWAN component). WWAN component 1422 may facilitate similar connectivity to multiple networks (e.g., UMTS networks and LTE networks).

[0166] Network 1427 may be a WLAN that operates according to one or more Institute of Electrical and Electronics Engineers (“IEEE”) 802.17 standards, such as IEEE 802.17a, 802.17b, 802.17g, 802.17n, and / or future 802.17 standards (collectively referred to herein as Wi-Fi). Draft 802.17 standards are also anticipated. In some configurations, the WLAN is implemented using one or more wireless Wi-Fi access points. In some configurations, the one or more wireless Wi-Fi access points are another computing device connected to a WWAN used as a Wi-Fi hotspot. WLAN component 1424 is configured to connect to network 1427 via the Wi-Fi access points. This connection may be protected by various encryption technologies, including but not limited to: Wi-Fi Protected Access (“WPA”), WPA2, Wired Equivalent Privacy (“WEP”), etc.

[0167] Network 1427 may be a WPAN, which operates according to the Infrared Data Association (“IrDA”), BLUETOOTH, Wireless Universal Serial Bus (“USB”), Z-wave, ZigBee, or some other short-range wireless technology. In some configurations, WPAN component 1426 is configured to facilitate communication with other devices, such as peripherals, computers, or other computing devices, via the WPAN.

[0168] In at least one example, sensor assembly 1408 includes a magnetometer 1428, an ambient light sensor 1430, a proximity sensor 1432, an accelerometer 1434, a gyroscope 1436, and a global positioning system sensor (“GPS sensor”) 1438. It is contemplated that other sensors (such as, but not limited to, temperature sensors or shock detection sensors, strain sensors, humidity sensors) may also be incorporated into computing device architecture 1400.

[0169] Magnetometer 1428 is configured to measure the strength and direction of a magnetic field. In some configurations, magnetometer 1428 provides measurements to a compass application stored within a memory component of memory component 1404 to provide the user with accurate orientation in a reference frame including basic directions (north, south, east, and west). Similar measurements can be provided to navigation applications that include a compass component. Other uses for the measurements obtained by magnetometer 1428 are anticipated.

[0170] Ambient light sensor 1430 is configured to measure ambient light. In some configurations, ambient light sensor 1430 provides the measurement to an application stored in a memory component 1404 to automatically adjust the brightness of a display (described below) to compensate for low-light and bright-light environments. Other uses of the measurements obtained by ambient light sensor 1430 are anticipated.

[0171] The proximity sensor 1432 is configured to detect the presence of an object or feature within proximity to the computing device without direct contact. In some configurations, the proximity sensor 1432 detects the presence of a user's body (e.g., the user's face) and provides this information to an application stored in one of the memory components of memory component 1404. This application uses the proximity information to enable or disable certain functions of the computing device. For example, a telephone application may automatically disable the touchscreen (described below) in response to receiving proximity information so that the user's face does not inadvertently end the call or enable / disable other functions in the telephone application during a call. Other uses of the proximity detected by the proximity sensor 1432 are contemplated.

[0172] Accelerometer 1434 is configured to measure appropriate acceleration. In some configurations, the output from accelerometer 1434 is used as an input mechanism by an application to control some functions of the application. For example, the application may be a video game in which characters, parts thereof, or objects are moved or otherwise manipulated in response to input received via accelerometer 1434. In some configurations, the output from accelerometer 1434 is provided to the application for switching between landscape and portrait modes, calculating coordinate acceleration, or detecting descent. Other uses for accelerometer 1434 are anticipated.

[0173] The gyroscope 1436 is configured to measure and maintain orientation. In some configurations, the output from the gyroscope 1436 is used by the application as an input mechanism to control some functions of the application. For example, the gyroscope 1436 can be used to accurately identify movement within a 3D environment in video game applications or some other applications. In some configurations, the application utilizes the output from the gyroscope 1436 and the accelerometer 1434 to enhance control over some functions of the application. Other uses for the gyroscope 1436 are anticipated.

[0174] GPS sensor 1438 is configured to receive signals from GPS satellites for use in calculating location. Any application that needs or benefits from location information can use the location calculated by GPS sensor 1438. For example, the location calculated by GPS sensor 1438 can be used with a navigation application to provide directions from that location to a destination or from a destination to that location. Furthermore, GPS sensor 1438 can be used to provide location information to external location-based services, such as the E1717 service. GPS sensor 1438 can utilize one or more network connectivity components 1406 to obtain location information generated via Wi-Fi, WiMAX, and / or cellular triangulation technologies to help GPS sensor 1438 achieve location fixation. GPS sensor 1438 can also be used in assisted GPS (“A-GPS”) systems.

[0175] In at least one example, I / O component 1410 may correspond to the above reference. Figure 12 The described input / output device 740. Additionally and / or alternatively, the I / O components include a display 1440, a touchscreen 1442, a data I / O interface component (“Data I / O”) 1444, an audio I / O interface component (“Audio I / O”) 1446, a video I / O interface component (“Video I / O”) 1448, and a camera 1470. In some configurations, the display 1440 and the touchscreen 1442 are combined. In some configurations, two or more of the data I / O component 1444, the audio I / O component 1446, and the video I / O component 1448 are combined. I / O component 1410 may include a discrete processor configured to support the various interfaces described below, or may include processing functions built into processor 802.

[0176] Display 1440 is an output device configured to present information in a visual form. Specifically, display 1440 can display graphical user interface (“GUI”) elements, text, images, video, notifications, virtual buttons, virtual keyboards, message data, internet content, device status, time, date, calendar data, preferences, map information, location information, and any other information that can be presented visually. In some configurations, display 1440 is a liquid crystal display (“LCD”) utilizing any active or passive matrix technology and any backlight technology (if used). In some configurations, display 1440 is an organic light-emitting diode (“OLED”) display. In some configurations, display 1440 is a holographic display. Other display types are anticipated.

[0177] In at least one example, display 1440 may correspond to the hardware display surface of computing device 112 and / or electronic device 706. As described above, the hardware display surface may be configured to graphically associate holographic user interfaces and other graphical elements with objects seen through the hardware display surface or drawn objects displayed on the hardware display surface.

[0178] Touchscreen 1442 (also referred to herein as a "touchscreen") is an input device configured to detect the presence and location of a touch. Touchscreen 1442 may be a resistive touchscreen, a capacitive touchscreen, a surface acoustic wave touchscreen, an infrared touchscreen, an optical imaging touchscreen, a dispersive signal touchscreen, an acoustic impulse recognition touchscreen, or may utilize any other touchscreen technology. In some configurations, touchscreen 1442 is incorporated as a transparent layer on top of display 1440, allowing a user to interact with objects or other information presented on display 1440 using one or more touches. In other configurations, touchscreen 1442 is a touchpad incorporated on the surface of a computing device excluding display 1440. For example, the computing device may have a touchscreen incorporated on top of display 1440 and a touchpad on a surface opposite display 1440.

[0179] In some configurations, touchscreen 1442 is a single-touch touchscreen. In other configurations, touchscreen 1442 is a multi-touch touchscreen. In some configurations, touchscreen 1442 is configured to detect discrete touches, single-touch gestures, and / or multi-touch gestures. For convenience, these are collectively referred to herein as "gestures." Several gestures will now be described. It should be understood that these gestures are illustrative and are not intended to limit the scope of the appended claims. Furthermore, the described gestures, additional gestures, and / or alternative gestures can be implemented in software for use with touchscreen 1442. In this way, developers can create gestures specific to particular applications.

[0180] In some configurations, the touchscreen 1442 supports tap gestures, whereby a user taps the touchscreen 1442 once on an item displayed on the display 1440. Tap gestures can be used to perform various functions, including but not limited to opening or launching any content tapped by the user. In some configurations, the touchscreen 1442 supports double-tap gestures, whereby a user taps the touchscreen 1442 twice on an item displayed on the display 1440. Double-tap gestures can be used to perform various functions, including but not limited to zooming in or out in stages. In some configurations, the touchscreen 1442 supports tap and hold gestures, whereby a user taps the touchscreen 1442 and holds the contact for at least a predetermined time. Tap and hold gestures can be used to perform various functions, including but not limited to opening context-specific menus.

[0181] In some configurations, the touchscreen 1442 supports panning gestures, where a user places and maintains contact with their finger on the touchscreen 1442 while moving their finger across it. Panning gestures can be used to perform various functions, including but not limited to moving across a screen, image, or menu at a controlled rate. Multi-finger panning gestures are also anticipated. In some configurations, the touchscreen 1442 supports swipe gestures, where a user flicks their finger in the direction they want the screen to move. Swipe gestures can be used to perform various functions, including but not limited to scrolling horizontally or vertically through menus or pages. In some configurations, the touchscreen 1442 supports pinch and stretch gestures, where a user uses two fingers (e.g., thumb and forefinger) to pinch or stretch across the touchscreen 1442. Pinch and stretch gestures can be used to perform various functions, including but not limited to gradually zooming in or out of a website, map, or image.

[0182] Although the gestures described above have been referenced to the use of one or more fingers to perform gestures, other appendages (such as toes) or objects (such as styluses) may be used to interact with the touchscreen 1442. Thus, the gestures described above should be understood as illustrative and not construed as limiting in any way.

[0183] Data I / O interface component 1444 is configured to facilitate data input to and output from a computing device. In some configurations, data I / O interface component 1444 includes a connector configured to provide a wired connection between the computing device and a computer system, for example, for synchronous operation purposes. The connector may be a dedicated connector or a standard connector, such as USB, micro USB, mini USB, etc. In some configurations, the connector is a mating connector for mating the computing device with another device, such as a docking station, audio device (e.g., a digital music player), or video device.

[0184] Audio I / O interface component 1446 is configured to provide audio input and / or output capabilities to a computing device. In some configurations, audio I / O interface component 1446 includes a microphone configured to collect audio signals. In some configurations, audio I / O interface component 1446 includes a headphone jack configured to provide connection to headphones or other external speakers. In some configurations, audio I / O interface component 1446 includes a speaker for outputting audio signals. In some configurations, audio I / O interface component 1446 includes an optical audio cable output.

[0185] Video I / O interface component 1448 is configured to provide video input and / or output capabilities to a computing device. In some configurations, video I / O interface component 1448 includes a video connector configured to receive video as input from another device (e.g., a video media player, such as a DVD or Blu-ray player) or to send video as output to another device (e.g., a monitor, television, or some other external display). In some configurations, video I / O interface component 1448 includes a High Definition Multimedia Interface (“HDMI”), mini HDMI, micro HDMI, DisplayPort, or a dedicated connector for inputting / outputting video content. In some configurations, video I / O interface component 1448 or a portion thereof is combined with audio I / O interface component 1446 or a portion thereof.

[0186] Camera 1470 can be configured to capture still images and / or video. Camera 1470 can utilize a charge-coupled device (“CCD”) or a complementary metal-oxide-semiconductor (“CMOS”) image sensor to capture images. In some configurations, camera 1470 includes a flash to assist in capturing images in low-light environments. Settings for camera 1470 can be implemented as hardware or software buttons. Images and / or video captured by camera 1470 can be additionally or alternatively used to detect non-contact gestures, facial expressions, eye movements, or other movements and / or characteristics of the user.

[0187] Although not illustrated, one or more hardware buttons may also be included in the computing device architecture 1400. Hardware buttons can be used to control some operational aspects of the computing device. Hardware buttons can be dedicated buttons or multi-purpose buttons. Hardware buttons can be mechanical or sensor-based.

[0188] The illustrated power assembly 1412 includes one or more batteries 1472 that can be connected to a battery gauge 1474. Batteries 1472 can be rechargeable or disposable. Rechargeable battery types include, but are not limited to, lithium polymer, lithium-ion, nickel-cadmium, and nickel-metal hydride. Each battery 1472 can be made from one or more battery cells.

[0189] The battery gauge 1474 can be configured to measure battery parameters such as current, voltage, and temperature. In some configurations, the battery gauge 1474 is configured to measure the battery's discharge rate, temperature, age, and other factors to predict remaining lifespan within a certain percentage of error. In some configurations, the battery gauge 1474 provides measurements to an application configured to use the measurements to present useful power management data to a user. Power management data may include one or more of the following: percentage of battery used, percentage of battery remaining, battery state, remaining time, remaining capacity (e.g., in watt-hours), current consumption, and voltage.

[0190] The power component 1412 may also include a power connector, which can be combined with one or more of the aforementioned I / O components 1410. The power component 1412 can interface with an external power system or charging equipment via the power I / O components.

[0191] Example Terms

[0192] A. A system comprising: a camera configured to capture video data; a buffer configured to store a recent window of the video data captured by the camera; a sensor data analysis module configured to determine a user’s emotional or cognitive state based at least in part on sensor data representing the user’s physiological condition; and a recording module configured to: detect changes in the user’s emotional or cognitive state; and in response to detecting changes in the user’s emotional or cognitive state, store a video segment including the contents of the buffer.

[0193] B. The system described in paragraph A, wherein the buffer is configured as a circular buffer.

[0194] C. According to the system described in paragraph A or paragraph B, the user's physiological state includes any combination of one or more of the following: skin conductance response, skin temperature, electrical activity of the brain, electrical activity of the heart, eye movements, facial expressions, pupil dilation, pupil constriction, voice volume, or voice speed.

[0195] D. The system described in any of paragraphs A through C, wherein the recording module is further configured to associate metadata with video segments, wherein the metadata indicates the user’s emotional or cognitive state at the time the video segment is recorded.

[0196] E. The system according to any one of paragraphs A through D further includes a biometric sensor for generating sensor data.

[0197] F. The system according to any one of paragraphs A through E, further comprising: a gaze detection module configured to determine a user's gaze target; wherein the recording module is further configured to associate metadata with a video segment, wherein the metadata indicates the user's gaze target when the video segment is recorded.

[0198] G. The system according to any of paragraphs A through F further includes a microphone configured to capture audio data, wherein the buffer is further configured to store the nearest window of audio data captured by the microphone.

[0199] H. The system described in any of paragraphs A through G, wherein the recording module is further configured to record additional video data to the video segment.

[0200] I. The system according to paragraph H, wherein the recording module is further configured to stop recording additional video data to the video segment based at least in part on another change in the user’s emotional and cognitive state.

[0201] J. The system according to paragraph H or paragraph I, wherein the recording module is further configured to stop recording additional video data to a video segment at least in part based on a predetermined time period.

[0202] K. According to the system described in paragraph J, the predetermined time period includes at least one of the following: the minimum duration of a video segment, or the maximum duration of a video segment.

[0203] L. The system described in any of paragraphs A through K is at least partially present as a head-mounted display device.

[0204] M. A method comprising: capturing video data using a camera; recording the video data to a buffer; receiving sensor data; analyzing the sensor data to detect changes in a user's emotional or cognitive state; and, in response to detecting changes in the user's emotional or cognitive state, creating a video segment that includes the video data currently in the buffer.

[0205] N. The method according to paragraph M, wherein receiving sensor data includes capturing sensor data using a biometric sensor.

[0206] O. The method according to paragraph M or paragraph N further includes associating metadata with video segments, wherein the metadata indicates the user's emotional or cognitive state at the time the video data was captured.

[0207] P. The method according to paragraphs M to O further includes: detecting the user's gaze target when capturing video data; and associating metadata with the video segment, wherein the metadata indicates the user's gaze target when the video data is captured.

[0208] Q. The method described in paragraphs M to P further includes: continuing to capture additional video data after the video segment is created; and adding the additional video data to the video segment.

[0209] R. One or more computer-readable media having computer-executable instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations including: capturing video content into a buffer; receiving sensor data; detecting, at least in part, a change in a user's emotional or cognitive state based on the sensor data; and storing a video segment, at least in part, based on the change in the user's emotional or cognitive state, the video segment including video content in the buffer when the change in the user's emotional or cognitive state was detected.

[0210] S. One or more computer-readable media as described in paragraph R, wherein the video segment further includes additional video content captured after a change in the user's emotional or cognitive state is detected.

[0211] T. According to one or more computer-readable media described in paragraph R or paragraph S, the operation further includes associating metadata with a video segment, wherein the metadata indicates the emotional or cognitive state of a user at the time the video content was captured.

[0212] in conclusion

[0213] Although the technology has been described in language specific to structural features and / or methodological actions, it should be understood that the appended claims are not necessarily limited to the described features or actions. Rather, the features and actions are described as exemplary implementations of this technology.

[0214] The operations of the example process are described in individual boxes and summarized with reference to these boxes. The process is illustrated as a logical flow of boxes, each box of which may represent one or more operations that can be implemented in software, hardware, or a combination thereof. In software, operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable one or more processors to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, modules, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order of the described operations is not intended to be construed as limiting, and any number of described operations may be executed in any order, combined in any order, subdivided into multiple sub-operations, and / or executed in parallel to implement the described process. The described process may be executed by resources associated with one or more devices 112, 116, 118, 404, 408, 702, and 706 (such as one or more internal or external CPUs or GPUs) and / or one or more segments of hardware logic (such as FPGAs, DSPs, or other types of accelerators).

[0215] All the methods and processes described above can be implemented on dedicated computer hardware and can be fully automated via dedicated computer hardware. Some or all of the methods can alternatively be implemented in software code modules executed by one or more general-purpose computers or processors. The code modules can be stored on any type of computer-readable storage medium or other computer storage device.

[0216] Unless otherwise specified, conditional language such as “can,” “able,” “may,” or “possibly” is understood in context to indicate that some examples include certain features, elements, and / or steps, while other examples do not. Therefore, such conditional language is not generally intended to imply that certain features, elements, and / or steps are necessary in any way for one or more examples, or that one or more examples must include logic for determining whether certain features, elements, and / or steps are included in any particular example or whether they should be performed in any particular example, whether or not they are present in the presence or absence of user input or prompts. Unless otherwise specified, connective language such as the phrase “at least one of X, Y, or Z” will be understood to mean that items, terms, etc., may be X, Y, or Z, or combinations thereof.

[0217] Any routine description, element, or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code comprising one or more executable instructions for implementing a specific logical function or element in the routine. Those skilled in the art will understand that alternative implementations are included within the scope of the examples described herein, wherein, depending on the functionality involved, elements or functions may be omitted or executed in an order inconsistent with the shown or discussed order, including substantially synchronously or in reverse order. It should be emphasized that many variations and modifications can be made to the examples described above, and the elements therein should be understood to be among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the appended claims.

Claims

1. A system comprising: a camera configured to capture video data; one or more processors; one or more sensors configured to obtain sensor data; and a memory storing instructions that, when executed by the one or more processors, cause the system to: detect, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, add at least a portion of the video data being captured by the camera to a video segment; detect, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and based at least in part on detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user, cease adding the at least a portion of the video data to the video segment.

2. The system of claim 1, wherein the instructions further cause the system to initiate recording of the video segment in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user.

3. The system of claim 1, wherein: the video segment begins at a first time associated with detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user.

4. The system of claim 1, further comprising a buffer configured to store the video data captured by the camera.

5. The system of claim 4, wherein the buffer is configured as a ring buffer.

6. The system of claim 1, wherein the one or more sensors are configured to obtain the sensor data and the additional sensor data based on one or more of: galvanic skin response, skin temperature, electrical activity of the brain, electrical activity of the heart, eye movement, facial expression, pupil dilation, pupil constriction, voice volume, or voice speed.

7. The system of claim 1, wherein the instructions further cause the system to generate video segment metadata indicative of the second emotional or cognitive state of the user.

8. The system of claim 7, wherein the instructions further cause the system to determine a gaze target of the user, wherein the video segment metadata is indicative of the gaze target of the user.

9. The system of claim 1, implemented at least in part as a head-mounted display device.

10. A method comprising: receiving sensor data from one or more sensors; detecting, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; ​ ​ in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, adding, by one or more processors, at least a portion of video data captured by a camera to a video segment; detecting, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and in response to detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user, ceasing to add the at least a portion of the video data to the video segment.

11. The method of claim 10, further comprising, in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, initiating recording of the video segment.

12. The method of claim 10, wherein: the video segment begins at a first time associated with detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user.

13. The method of claim 10, further comprising obtaining, by the one or more sensors, the sensor data and the additional sensor data based on one or more of: galvanic skin response, skin temperature, electrical activity of the brain, electrical activity of the heart, eye movement, facial expression, pupil dilation, pupil constriction, voice volume, or voice speed.

14. The method of claim 10, further comprising generating video segment metadata indicative of the second emotional or cognitive state of the user.

15. The method of claim 14, further comprising determining a gaze target of the user, wherein the video segment metadata is indicative of the gaze target of the user.

16. A system comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to: receive sensor data from one or more sensors; detect, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, initiate recording or storage of a video segment comprising video data captured by a camera; detect, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and stop the recording or storage of the video segment based at least in part on detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user.

17. The system of claim 16, wherein: ​ the video segment begins at a first time associated with detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user.

18. The system of claim 16, wherein the instructions further cause the system to generate video segment metadata indicative of the second emotional or cognitive state of the user.

19. The system of claim 18, wherein the instructions further cause the system to determine a gaze target of the user, wherein the video segment metadata is indicative of the gaze target of the user.

20. The system of claim 16, implemented at least in part as a head-mounted display device.

Images