Leveraging sensor data for detecting neural states for content generation and control
Through sensor data and cognitive identification models, the problem of difficult to generate fascinating language content in the prior art is solved, and a higher entertainment experience and audience participation is achieved.
Patent Information
- Application Number
- CN201980017606.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-07
- Filing Date
- 2019-01-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-01-08
AI Technical Summary
The prior art is difficult to generate or control linguistic content that can be fascinating, especially in entertainment scenarios that combine linguistic and nonverbal content, and cannot effectively evaluate and adjust content to meet the audience's neurophysiological state, resulting in a bland and tasteless entertainment experience.
By using sensor data, such as EEG, GSR, fEMG, etc., combined with cognitive identification models, we can detect and adjust the evocation and evaluation of content in real time, and dynamically generate or select appropriate content fragments to match the audience's neurophysiological responses, and achieve more High engagement and entertainment experience.
It realizes the adjustment of content based on the audience's real-time neurophysiological status, improves the attractiveness and participation of the entertainment experience, meets the audience's diverse needs, and enhances the commercial value of the content.
Smart Images

Figure CN112352390B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 614,811 filed on January 8, 2018, U.S. Provisional Patent Application Serial No. 62 / 661,556 filed on April 23, 2018, and U.S. Provisional Patent Application Serial No. 62 / 715,766 filed on August 7, 2018, the entire contents of which are incorporated herein by reference in their entirety. Technical Field
[0003] The present disclosure relates to applications, methods, and apparatus for signal processing of sensor data from detection of a user's neural state in applications for generating or controlling language content and other forms of communication content. Summary of the invention
[0004] The present invention content and the following detailed description should be interpreted as supplementary parts of the overall disclosure, which may include redundant topics and / or supplementary topics. The omission in any section does not indicate the priority or relative importance of any element described in the overall application. The differences between these sections may include supplementary disclosures of alternative embodiments, additional details, or alternative descriptions of the same embodiment using different terms, which are obvious based on the corresponding disclosure. The prior application with serial number 62 / 661,556 filed on April 23, 2018 lays the foundation for digitally representing user participation in audio-video content, including but not limited to digital representations of content participation capabilities (CEP) based on sensor data. As more fully described in the earlier application, computer processing develops CEPs for content based on sensor data from at least one sensor, and the at least one sensor is positioned to sense involuntary responses of one or more users when participating in audio-video output. For example, sensor data may include one or more of the following received from corresponding sensors: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and body chemical sensing data and functional near infrared data (fNIR). "User" refers to an audience member, a person who experiences communication content as a consumer for entertainment purposes. The present application builds on this foundation and uses CEP in various applications summarized below.
[0005] CEP is an objective, algorithmic and digital electronic measure of a user's neurophysiological state, which is related to the user's participation in stimulation (e.g., branching content). CEP expresses at least two orthogonal measures, such as arousal and valence. As used herein, "arousal" refers to a state or condition of being physiologically alert, awake, and attentive according to its psychological meaning. High arousal indicates interest and concern, while low arousal indicates boredom and disinterest. "Valence" is also used here in its psychological sense of attractiveness or aesthetics. Positive valence indicates attraction, while negative valence indicates disgust. As used herein, "neurophysiology" refers to a physiological state, a neural state, or both states that indicate or are derived from a person. "Biometrics" refers to a measure of a biological state that encompasses "neurophysiology" and can encompass other information, such as identity information. Certain data (e.g., images of a person's face or other body parts) can indicate both identity and neurophysiological state.
[0006] In one aspect of the disclosure, a CEP or similar measure is used in a method for generating a script. The method may include randomizing, by at least one processor, one or more parameters of a cognitive discrimination model for participants in a role interaction. The method may also include generating, by at least one processor, a script at least in part by modeling the role interaction while varying the parameters between different scripts in a plurality of scripts. The method may also include, for each of the plurality of scripts, estimating, based on the cognitive discrimination model, a measure of effectiveness of one of the plurality of target storylines for neurophysiological responses. The method may include recording the measure of effectiveness in a computer memory.
[0007] In one aspect, the measure of estimating effectiveness may include computing a content engagement power (CEP) based on a cognitive discrimination model. Computing a CEP value may also include determining an arousal value and comparing the stimulus average arousal to an expected average arousal. Additionally, computing the CEP value may also determine an evaluation value. In a related aspect, the measure of estimating effectiveness may also include determining a neural error measure based on comparing the predicted neurophysiological response to a target storyline. The target storyline may be or may include a set of target neural values, each target neural value being uniquely associated with a different interval of a continuous time series.
[0008] In another aspect, a method for generating communication content (which includes verbal content alone or verbal content coordinated with non-verbal content) includes receiving, by at least one processor, neurophysiological data from one or more sensors coupled to at least one user consuming a segment of the communication content, wherein the segment is selected from a library of alternative segments. The method may also include computing, by at least one processor, for each of the segments, a measure of a neurophysiological response of at least one user based on the neurophysiological data and a cognitive discrimination model for evaluating the neurophysiological data. The method may also include recording, by at least one processor, the measure of the neurophysiological response of at least one user for each of the segments in a computer memory associated with an identifier for each of the segments for at least one of the following: making communication content including selected ones of the segments, or selecting participants for a multi-participant communication session. The method may also include generating the communication content at least in part by selecting segments from the library based on the associated measure of the neurophysiological response of at least one user for each segment and combining the segments selected by the selection. In one aspect, the processor may perform the collection, computing, and recording for a plurality of different users each consuming the segments.
[0009] In some applications, such as when the multi-participant communication session includes a computer game or a social chat session, the method can include matching users of the multi-participant communication session at least in part by selecting users from a plurality of different users that have complementary measures of neurophysiological responses to corresponding segments. In some cases, the segments consist essentially of language content, such as written portions of text. In other cases, the segments consist essentially of audio content (including music). Thus, in addition to audio-video content, the method is also useful for generating text documents and audio-only content.
[0010] The operation may include determining a content engagement capability (CEP) value or an equivalent measure as described herein for each of the segments. Determining the CEP value may include determining an arousal value based on neurophysiological data, and comparing the average arousal of the stimulus based on the neurophysiological data to the expected average arousal, and receiving data from a biometric sensor as described elsewhere herein. The method may also include determining an evaluation value based on the neurophysiological data from the biometric sensor. The measure of the neurophysiological response of at least one user for each of the segments may include an indication of at least one evaluation. The selection may also include assigning an error indicator to the associated measure of each segment based on a difference between the associated measure of the neurophysiological response of at least one user and a target value, and minimizing the segment with the assigned error indicator.
[0011] In another aspect, a method for controlling the presentation of communication content (which includes language content alone or language content coordinated with non-language content) includes receiving neurophysiological data from one or more sensors coupled to at least one user while at least one user is consuming a segment of the content on a client device, wherein the content includes a chain of segments. The method also includes calculating, by at least one processor, for each of the segments, a measure of the neurophysiological response of at least one user based on the neurophysiological data and a cognitive discrimination model for assessing the neurophysiological data. The method also includes selecting, by at least one processor, a subsequent segment in the chain based on the measure of the neurophysiological response of at least one user for the segment, and outputting the subsequent segment to the client device. The segments may include content language content, audio content, or audio-video content.
[0012] The operation may include determining a content engagement capability (CEP) value for each of the segments as described elsewhere herein. Determining the CEP value may include determining an arousal value based on neurophysiological data and comparing the average arousal of the stimulus based on the neurophysiological data to the expected average arousal, and / or determining an evaluation value based on the neurophysiological data. Sensors for indicating arousal and evaluation are described elsewhere herein. The method may include normalizing the evaluation or arousal value based on corresponding values collected for a set of specified baseline stimuli. The selection may also include determining an error indicator for the associated measure of each segment based on a difference between an associated measure of a neurophysiological response of at least one user and a target value, such as by minimizing segments having an assigned error indicator.
[0013] In another aspect, a method for collecting neurophysiological data indicative of a neurophysiological response of a user consuming communication content may include detecting, by at least one processor of a mobile device, that a user of the mobile device is consuming communication content independently of the mobile device. The method may also include receiving, by at least one processor, neurophysiological data of the user via one or more sensors in the mobile device while the user is consuming the communication content. The method may also include providing at least one of the neurophysiological data or an indicator of a neural state of the user derived therefrom to at least one of: a second device that outputs the communication content, a remote server, or a computer memory.
[0014] The method may also include identifying, by at least one processor, an identifier for the communication content, and providing the identifier to at least one of a remote server or a computer memory. The method may include computing, by at least one processor, an indicator of a user's neural state based on neurophysiological data and a cognitive discrimination model. The method may also include initiating a communication session with a second device that outputs the communication content before collecting. The collection may be performed via at least one sensor of the mobile device, the sensor including a microphone, a camera, an infrared sensor, a phased array sensor, or a galvanic skin response sensor. In some embodiments, the mobile device may include or be coupled to a wrist-worn smart device. In some embodiments, the mobile device coordinates the switching of a playback session of the communication content from the second device to the third device. In one aspect, the mobile device does not communicate with the second device.
[0015] In one aspect of the method, at least one processor of the mobile device may output a user-perceivable signal to attract at least one of gaze, touch, voice, or gesture at a time synchronized with the playback of the communication content. After outputting the user-perceivable signal, at least one processor of the mobile device may receive at least a portion of the neurophysiological data generated by the gaze, touch, voice, or gesture.
[0016] In a similar application for CEP or similar metrics, a method for rating the effectiveness of personal communications may include monitoring, by at least one processor, digital data representing a user's communications with a recipient. The method may include receiving sensor data from at least one sensor positioned to sense an involuntary response of the recipient when receiving the communication. The method may include determining a content engagement capability (CEP) value based on the sensor data, and recording the CEP value in a computer memory and / or transmitting a representation of the CEP value to the user. In one aspect, determining the CEP value may also include determining an arousal value based on the sensor data, and comparing the average arousal of the stimulus based on the sensor data with the expected average arousal. The sensor data may include one or more of the following: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR).
[0017] In a related aspect, the method may include determining an expected average arousal based on further sensor data that measures similar involuntary responses of recipients when engaging with known audio-visual stimuli. Thus, the method may include playing known audio-visual stimuli that include known non-arousing stimuli and known arousing stimuli. More detailed aspects of determining CEP values, operating one of a plurality of event capabilities for each of one or more users, assigning a weight to each of the event capabilities based on one or more source identifiers of the sensor data, determining an expected average arousal, and determining rating values based on the sensor data may be as described for other applications, as described herein above, or as described in more detail below.
[0018] The foregoing method may be implemented in any suitable programmable computing device by program instructions provided in a non-transitory computer-readable medium, which program instructions, when executed by a computer processor, cause the device to perform the described operations. The processor may be local to the device and the user, remotely located, or may include a combination of local and remote processors. A device may include a computer or a group of connected computers that are used to measure and transmit CEP or similar engagement metrics for content output devices. The content output device may include, for example, a personal computer, a mobile phone, a notebook computer, a television or computer monitor, a projector, a virtual reality device, or an augmented reality device. Other elements of the device may include, for example, an audio output device and a user input device, which participate in the execution of the method. A device may include a virtual or augmented reality device, such as a headset or other display that reacts to the movement of the user's head and other body parts. The device may include a neurophysiological sensor that provides data that is used by a controller to determine a digital representation of the CEP.
[0019] To accomplish the foregoing and related ends, one or more examples include the features fully described below and particularly pointed out in the claims. The following description and drawings set forth certain illustrative aspects in detail and are indicative of only some of the various ways in which the principles of the examples may be employed. Other advantages and novel features will become apparent when the following detailed description is considered in conjunction with the drawings and disclosed examples, and the detailed description includes all such aspects and their equivalents. Background Art
[0020] Language, both written text and spoken, is central to human communication. Many tools have been developed to assist its dissemination. Until the computer age, only humans were involved in the generation of language. Since then, talking computers have become commonplace. Computers are used to generate all types of language, both written and spoken. Computer-generated language is functional, but lacks the ability to capture the imagination and engage an audience like great literature and speech.
[0021] Much verbal content is of commercial importance. For example, for textual content such as books, audio books, dialogues for theatrical performances, scripts, and other uses, verbal content is vital to the entertainment industry. Without engaging verbal content, entertainment becomes bland. If consumers have better options, they will not pay for it or consume it at all. Advertisers will not sponsor it. Despite the use of computers in the generation of verbal content to perform technical tasks such as word processing, the work of making verbal content more engaging maintains the need for human hands. The logic that drives functional responses is not applicable to the generation of engaging content.
[0022] Language content is usually accompanied by non-language content, such as music, graphic art and video. Commercial entertainers are committed to integrating language and non-language content in a more fascinating way to attract as many audiences as possible. The generation and control of language and non-language content are usually interrelated. As used herein, "communication content" includes language content and other entertainment or aesthetic content, which is sorted for presentation or consumption over time. Whether the communication content is pure language or contains other elements, computers cannot generate or control content well, making it more fascinating to the audience. Even if machines may never completely replace human artists, artists and audiences can also benefit by providing artists with more powerful tools to make and control content. "Communication content" is sometimes referred to as "content" in this article.
[0023] It is therefore desirable to develop new methods and other new technologies for generating or controlling verbal content alone or in coordination with non-verbal content to overcome these and other limitations of existing technologies and to help producers provide more engaging entertainment experiences for future audiences. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The features, nature and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, wherein like reference numerals identify like elements throughout the specification and drawings.
[0025] Figure 1Ais a schematic block diagram showing aspects of a system or apparatus coupled to one or more distribution systems for generating scripts, text, or other communication content based on neurophysiological data or controlling the broadcast of communication content, collecting neurophysiological data related to the consumption of communication content, or rating the effectiveness of personal communications.
[0026] Figure 1B-Figure 1C is a diagram showing various aspects of communication content.
[0027] Figure 2 is a schematic block diagram illustrating aspects of a server for generating scripts, text, or other communication content, for controlling the playout of communication content based on neurophysiological data, or for rating the effectiveness of an individual's communications.
[0028] Figure 3 is a schematic block diagram showing various aspects of a client device coupled to one or more distribution systems for generating scripts, text, or other communication content based on neurophysiological data or controlling the broadcast of communication content, collecting neurophysiological data related to consumption of communication content, or rating the effectiveness of personal communications.
[0029] Figure 4 is a schematic diagram showing features of a virtual reality client device coupled to one or more distribution systems, the virtual reality client device being used to generate scripts, text, or other communication content based on neurophysiological data or to control the playout of communication content, to collect neurophysiological data related to consumption of communication content, or to rate the effectiveness of personal communications.
[0030] Figure 5 is a flow chart illustrating the high-level operation of a method for determining a digital representation of a CEP based on sensor data collected during a performance of communication content.
[0031] Figure 6 is a block diagram illustrating high-level aspects of a system for training a machine learning process to determine a digital representation of a CEP based on sensor data collected during a performance of communication content.
[0032] Fig. 7A is a diagram indicating the arrangement of neural states relative to the axes of the two-dimensional neurophysiological space of the cognitive discrimination model.
[0033] Figure 7B is a diagram indicating the arrangement of neural states relative to the axes of the three-dimensional neural system space of the cognitive discrimination model.
[0034] Figure 8 is a flow chart illustrating a process and algorithm for determining content engagement levels based on neurophysiological response data.
[0035] Fig. 9 is a diagram illustrating a system for collecting neurophysiological response data using a mobile device and a cooperating application.
[0036] Fig.10 is the perspective of a user using a mobile application with sensors and accessories for collecting neurophysiological data used in the methods and apparatus described herein.
[0037] Fig.11 is a sequence diagram illustrating aspects of collecting neurophysiological response data using a mobile device with an auxiliary sensor device.
[0038] Fig.12 is a flow chart illustrating aspects of a method for operating an apparatus or system that automatically creates or edits scripts for a theatrical production.
[0039] Fig.13 is a conceptual block diagram illustrating components of an apparatus or system for automatically producing or editing scripts for theatrical productions.
[0040] Fig.14 and Fig.15 is a flow chart illustrating aspects of a method for generating communication content.
[0041] Fig.16 is a conceptual block diagram illustrating components of an apparatus or system for generating communication content.
[0042] Fig.17 and Fig.18 is a flow chart illustrating aspects of a method for controlling presentation of communication content.
[0043] Fig.19 is a conceptual block diagram illustrating components of an apparatus or system for controlling presentation of content.
[0044] Fig. 20 and Fig.21 is a flow chart illustrating aspects of a method for collecting neurophysiological data indicative of a neural response of a user consuming communication content.
[0045] Fig. 22 is a conceptual block diagram illustrating components of an apparatus or system for collecting neurophysiological data indicative of a neural response of a user consuming communication content.
[0046] Fig.23 is a diagram illustrating a system including a mobile device having a sensor to enhance interpersonal communication using neurophysiological talk.
[0047] Fig.24 is a flow chart illustrating aspects of a method for operating a system to enhance interpersonal communication using neurophysiological talk.
[0048] Figure 25-26 It is shown Fig.24 Flowcharts of optional other aspects or operations of the method illustrated in .
[0049] Fig. 27 is a conceptual block diagram illustrating components of an apparatus or system for enhancing interpersonal communication using neurophysiological speech. DETAILED DESCRIPTION
[0050] Various aspects are now described with reference to the accompanying drawings. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it is apparent that various aspects can be practiced without these specific details. In other examples, well-known structures and devices are shown in block diagram form to facilitate description of these aspects.
[0051] refer to Figure 1A , a method for signal processing neurophysiological sensor data to detect neural states in communication enhancement applications can be implemented in a client-server environment 100. Other architectures may also be suitable. In a network architecture, sensor data can be collected and processed locally, and then transmitted to a server, which processes the neurophysiological sensor data from one or more subjects, calculates a digital representation of the user's neural state based on the sensor data, and uses it in a computer memory to control a machine. Communication enhancement contexts for the present technology include generating scripts, texts, or other communication content based on neurophysiological data or for controlling the broadcast of communication content, collecting neurophysiological data related to the consumption of communication content, or rating the effectiveness of personal communications.
[0052] Recorded communications can be either branching or non-branching. By "branching" is meant that the communication content includes alternative segments that can be variously combined using cognitive discrimination models based on assessments of neurophysiological data. A control processor selects at runtime which alternative segments it will include. If the branching content has a non-branching narrative, it will include branches from other dramatic elements. Although the content can be branching, it can have a coherent theme, dramatic purpose, and storyline that encompasses all of the branches. By "non-branching" is meant that the communication content has no alternative segments, such as in a conversation or in a set production. In a production environment, a director may use cognitive discrimination models from a focus group to select one of the alternative shots or scenes based on assessments of neurophysiological data, but these pre-release alternatives are not used in general distribution.
[0053] A data processing server such as a "math" server 110 may receive sensor data from sensors positioned to detect neurophysiological responses of a user during consumption of communication content. The sensors may be coupled to the network via any of client devices 106, 108, 114, 120, 118, which may be used to consume communication content or simply be in the vicinity of a user consuming content output by a different device. The server 100 may process the sensor data to obtain a digital representation indicating a listener's neurophysiological (e.g., emotional) response to the communication content as indicated along one or more measurement axes (e.g., arousal and evaluation) according to time or video frame. In an alternative embodiment, based on real-time biosensor feedback, the content adaptive AI may adaptively adjust the content to increase or maintain user engagement.
[0054] A suitable client-server environment 100 may include various computer servers and client entities that communicate via one or more networks, such as a wide area network (WAN) 102 (e.g., the Internet) and / or a wireless communication network (WCN) 104 (e.g., a cellular telephone network for data communications, such as LTE, 4G, 5G). The computer servers may be implemented in various architectures. For example, the environment 100 may include one or more web / application servers 124 that contain documents and application code compatible with the World Wide Web protocols, including but not limited to, for example, HTML, XML, PHP, and JavaScript documents or executable scripts. The web / application servers 124 may serve applications for outputting communication content and for collecting biometric sensor data from users experiencing the content. In the alternative, the data collection application may be served from a math server 110, a cloud server 122, a blockchain entity 128, or a content data server 126. A user may consume communication content using any of the client devices or terminals described herein; in some embodiments, a user may consume content from an independent source such as a movie projector, a home theater, or a television or radio.
[0055] The environment 100 may include one or more data servers 126 for storing: communication content, such as language content alone or in combination with video, audio-video, audio, and graphics content components of the communication content, for consumption using a client device; software for execution on or in conjunction with a client device, such as sensor control and sensor signal processing applications; and data collected from a user or client device. The data collected from a client device or user may include, for example, biometric sensor data and application data. Sensor data may be collected by a background (non-user-facing) application operating on a client device and transmitted to a data sink, such as a cloud-based data server 122 or a discrete data server 126. Application data refers to application state data, including but not limited to records of user interactions with an application or other application input, output, or internal state. Applications may include software for outputting communication content, collecting and processing biometric sensor data, computing measures or indicators of a user's neurophysiological response, providing the indicator or measure to a specified output, selecting segments of content based on the indicator or measure, and supporting functionality. Applications and data may be served from other types of servers, for example, any server that accesses the distributed blockchain data structure 128, or a peer-to-peer (P2P) server 116, such as may be provided by a set of client devices 118, 120 and operate simultaneously as a micro-server or client.
[0056] As used herein, a "user" is always a consumer of communication content from whom system nodes collect neurophysiological response data for use in determining a digital representation of engagement with the communication content. When actively participating in content via an avatar or other mechanism, a user may also be referred to herein as a "player actor." An audience is not always a user. For example, a spectator may be a passive audience from whom the system does not collect any biometric response data. As used herein, a "node" includes a client or server participating in a computer network.
[0057] The network environment 100 may include various client devices, such as a mobile smartphone client 106 and a laptop client 108 connected to a server via a WCN 104 and a WAN 102, or a mixed reality (e.g., virtual reality or augmented reality) client device 114 connected to a server via a router 112 and a WAN 102. In general, a client device may be or may include a computer used by a user to access communication content provided via a server or from a local storage. The client may also include a biometric sensor that provides neurophysiological data to the apparatus and processes described herein. In one aspect, the "math" server 110 may determine a digital representation of the biometric data for use in real-time or offline applications. Controlling branching or activity of objects in narrative content is an example of a real-time application, for example, as described in U.S. Provisional Patent Application Nos. 62 / 566,257, filed on September 29, 2017, and 62 / 614,811, filed on January 8, 2018, which are incorporated herein by reference.
[0058] Offline applications may include, for example, "green lighting" production proposals, automatic screening of production proposals before green lighting, automatic or semi-automatic packaging of promotional content (such as trailers or video ads), and editing or design (automatic and semi-automatic) of customized content for target users or user groups. In a green light script or story, a client device may receive neurophysiological data using the biometric sensors described herein as a user reads a script or other text on a tablet device. The client or server may record a CEP score or equivalent measure with reference to calibration data from the quantitative emotion data. The creative product may then select or refine a script, proposal, script, etc., taking into account the sum of CEP event capabilities, consistency with emotional plots, or discovery of new, better performing emotional plots. Neurophysiological data may also be used in writers' tools to enable writers to plan conditional text based on biometric emotional feedback and plots of tested story elements.
[0059] In some embodiments, the client device provides both communication content and senses biometric data. The client device can use neurophysiological data locally, or the data can be passed to a cloud server for other communication content sessions, for example, in an over-the-top (OTT) experience on a car or home TV. In the OTT experience, the server uses a high-bandwidth network (5G) to switch content sessions across platforms in real time. Content sessions can be started on a mobile device and can be moved across different platforms (such as virtual reality, home theater, or in-car entertainment systems). In other embodiments, when the client device acts as a sensor and processor for neurophysiological data, different devices provide communication content. Client devices such as smart phones or smart watches can use the following items to collect neurophysiological and other biometric data: a front-facing stereo camera for tracking gaze direction, pupil dilation, and / or facial action units; a microphone for audio speech analysis and / or neuro-linguistic programming word analysis; a phased array sensor (if any) for tracking gaze direction, pupil dilation, and / or facial action units; an infrared sensor for functional near-infrared sensing of skin temperature. Additionally, the client device can connect to an auxiliary device using Bluetooth or other protocols to obtain sensor data. For example, a smartphone can connect to a smart watch or fitness band worn on the wrist or around the head to obtain pulse rate, skin temperature, galvanic skin response, or electroencephalogram (EEG) data to calculate neurophysiological metrics as described herein.
[0060] Figure 1B Various aspects of the communication content 140 are shown. Certain finite "N" segments (S N ) 142, 144, 146, 148 may be video frames, text or speech components, musical phrases (e.g., bars), graphic novel panels, or any other signal or theme arranged along a time-like dimension with a flow direction. The content of the segments may be predetermined (e.g., a movie, a video) or determined on the fly (e.g., a computer game or a talk show) and may be digitized in a computer-readable form. 1 142 to S N The collection of all segments 148 constitutes "communication content", which, as used herein, refers to machine-readable content composed of segments arranged along a time-like dimension 150 having a flow direction from a start 152 to an end 154; the exact time of the end may be unknown or uncertain at an earlier time during the communication content. A discrete collection of segments constitutes a content "title", "experience", or "work". Examples of communication content and application of the methods herein to different forms of communication content are summarized below.
[0061] Reference Figure 1C, a system 160 used with the methods and apparatus described herein may include communication content 162, a computing module 164, a user 176, and neurophysiological data 178. Communication content 162 may include language content 166, which includes text or speech, such as scripts, plays, books, graphic novels, and comic books, as well as other printed content for reading, audio content (such as audio books or podcasts), and may be included in other content, such as video, virtual reality, or video game content that includes speech or text. Although arranged on a page, most text is arranged to be read in a left-to-right, right-to-left, and / or top-to-bottom order along a time-like dimension 150 with a flow direction from start 152 to end 154. Therefore, written materials (text) produced in electronic form generally fall within the definition of communication content 162 used herein. Biometric feedback and measures of neurophysiological responses 178 can be used for production control of written materials or control of presentation of written materials. In presentation control, one or more processors of computing module 164 change the text segment selected and displayed to the user based on neurophysiological feedback via the biometric sensor.
[0062] The computing module 164 includes at least three components: a player 170 that decodes the communication content and presents it to a user 176; a control or computing component 172 that receives data from one or more biometric sensors of a sensing component 174 and computes one or more neurophysiological measures 178 from the sensor data; and a sensing component 174 that responds to neurophysiological changes in the user 176 and generates digital signals for processing. The computing component 172, the sensing component 174, and the player component 170 may be combined in one device or divided between two or more devices.
[0063] The communication content 162 may include non-language content 168, such as music and other audio content, images, and data for other sensory outputs (e.g., tactile, olfactory, or kinetic outputs). Non-language content may be combined with language content, such as music with or without lyrics, and most videos include language elements. When a user listens to music on a client device, a processor may score a segment of the audio content, for example, by receiving and analyzing neurophysiological data from a biometric sensor. The processor may select audio based on the sum of CEP event capabilities, consistency with an emotional episode, or discovery of a new, better performing emotional episode. Typically, the processor may use neurophysiological data in a passive mode (data collection only) or an active mode (content changes based on biofeedback). Neurophysiological data may also be used to avoid personal or cultural triggering of anger, fear, or other undesirable reactions caused by historical data of an individual or cohort. A processor executing a machine learning algorithm may learn "expected" triggers based on historical data.
[0064] In addition to providing neurophysiological data, biometric sensors can also be used for personal identification using facial recognition and other unique biometrics of the user. Thus, a device can use the same sensor for identification and receive neurophysiological data for content analysis, generation, or control. On the one hand, when collecting baseline data, communication content can be selected to be compatible with the available sensor modalities. For example, if eye tracking (also known as gaze detection) is not available, characters or items of interest can be portrayed individually and not with other characters. When eye tracking is used, characters can be displayed together because eye tracking will indicate which character attracted the user's interest or caused the user to avert their eyes.
[0065] In other applications, the processor can use personal patterns of neurophysiological responses to known content as the basis for matching people as prospective friends or co-players. The processor can use personal patterns to help users browse new content (randomly generated) or find content that the user has preferred in the past. The processor can sometimes make suboptimal choices to compensate for high-demand users.
[0066] Figure 2 A data processing server 200 is shown, which is used to generate scripts, texts, or other communication content in computer memory based on biometric sensor data, and the data processing server can operate in environment 100, in a similar network, or as a stand-alone server. Server 200 may include one or more hardware processors 202, 214 (two of the one or more hardware processors are shown). The hardware includes firmware. Each of the one or more processors 202, 214 can be coupled to an input / output port 216 (e.g., a universal serial bus port or other serial or parallel port), coupled to a source 220 of biometric sensor data and viewing history for indicating the user's neural state. The viewing history may include a log level record of changes from a baseline script of a content package or an equivalent record of control decisions made in response to user biometrics and other inputs. The viewing history may also include content watched on TV, Netflix, and other sources. Any source containing a derived storyline can be used as an input to an algorithm for generating a script, text, or other actor, character, or other story element in computer memory based on biometric sensor data. Server 200 can track user actions and biometric responses across multiple content titles for an individual or cohort. Certain types of servers (e.g., cloud servers, server farms, or P2P servers) may include multiple instances of discrete servers 200 that collaborate to perform the functions of a single server.
[0067] The server 200 may include a network interface 218 for sending and receiving applications and data, including but not limited to sensor and application data for generating scripts, text, or other communication content in computer memory based on biometric sensor data. The content may be served from the server 200 to the client device, or may be stored locally by the client device. If stored locally to the client device, the client and server 200 may collaborate to handle the collection of sensor data and transmission to the server 200 for processing.
[0068] Each processor 202, 214 of the server 200 can be operably coupled to at least one memory 204, which stores one or more functional modules 206, 208, 210, 212 of the application to perform the methods described herein. These modules may include, for example, a correlation module 206 that correlates biometric feedback with one or more measures such as arousal or evaluation. The correlation module 206 may include instructions that, when executed by the processor 202 and / or 214, cause the server to use machine learning (ML) or other processes to correlate biometric sensor data with one or more neural (e.g., emotional) states of the user. The event detection module 208 may include functionality for detecting events based on a measure or indicator that one or more biometric sensor inputs exceed a data threshold. These modules may also include, for example, a standardization module 212. The standardization module 212 may include instructions that, when executed by the processor 202 and / or 214, cause the server to use a baseline input to standardize the measure of the evaluation, arousal, or other value. These modules may also include a computation function 210 that, when executed by the processor, causes the server to compute a content engagement capability (CEP) based on sensor data and other outputs from upstream modules. Details of determining CEP are disclosed later in this document. Memory 204 may contain additional instructions, such as an operating system and support modules.
[0069] refer to Figure 3 , content consumption device 300 generates biometric sensor data indicating a user's neurophysiological response to an output generated from a communication content signal. Apparatus 300 may include, for example, a processor 302, such as a processor based on an image processing system provided by Intel TM or AMD TM Designed by ARM with 80x86 architecture central processing unit TMThe processor 302 may be communicatively coupled to auxiliary devices or modules of the 3D environment apparatus 300 using a bus or other coupling means. Optionally, the processor 302 and its coupled auxiliary devices or modules may be housed in or coupled to a housing 301, such as a housing having a form factor of a television, a set-top box, a smart phone, wearable Google glasses, glasses or a visor, or other form factors.
[0070] A user interface device 324 may be coupled to the processor 302 for providing user control input to the media player and data collection process. The process may include outputting video and audio for a display screen or projection display device. In some embodiments, the communication content control process may be or may include an audio-video output for an immersive mixed reality content display process operated by a mixed reality immersive display engine executing on the processor 302.
[0071] User control inputs may include, for example, selections from a graphical user interface or other inputs (e.g., text or directional commands) generated via a touch screen, keyboard, pointing device (e.g., game controller), microphone, motion sensor, camera, or some combination of these or other input devices (represented by box 324). Such user interface devices 324 may be coupled to processor 302 via input / output ports 326 (e.g., universal serial bus (USB) or equivalent port). Control inputs may also be provided via sensors 328 coupled to processor 302. Sensor 328 may be or include, for example, a motion sensor (e.g., an accelerometer), a position sensor, a camera or camera array (e.g., a stereo imaging array), a biometric temperature or pulse sensor, a touch (pressure) sensor, an altimeter, a position sensor (e.g., a global positioning system (GPS) receiver and controller), a proximity sensor, a motion sensor, a smoke or steam detector, a gyroscope position sensor, a radio receiver, a multi-camera tracking sensor / controller, an eye tracking sensor, a microphone or microphone array, an electroencephalogram (EEG) sensor, a galvanic skin response (GSR) sensor, a facial electromyography (fEMG) sensor, an electrocardiogram (EKG) sensor, a video facial action unit (FAU) sensor, a brain-computer interface (BMI) sensor, a video pulse detection (VPD) sensor, a pupil dilation sensor, a body chemistry sensor, a functional magnetic resonance imaging (fMRI) sensor, a photoplethysmography (PPG) sensor, a phased array radar (PAR) sensor, or a functional near infrared data (fNIR) sensor. Any one or more of the eye tracking sensor, FAU sensor, PAR sensor, pupil dilation sensor, or heart rate sensor may be or may include a front (or rear) stereo camera such as used in the iPhone 10 and other smartphones for facial recognition. Likewise, a camera in a smartphone or similar device may be used for ambient light detection, for example, detecting ambient light changes to correlate with changes in pupil dilation.
[0072] One or more sensors 328 may detect biometric data used as an indicator of the user's neurological state, such as one or more of facial expression, skin temperature, pupil dilation, breathing rate, muscle tone, nervous system activity, pulse, EEG data, GSR data, fEMG data, EKG data, FAU data, BMI data, pupil dilation data, chemical detection (e.g., oxytocin) data, fMRI data, PPG data, or fNIR data. In addition, (one or more) sensors 328 may detect the user's context, such as the user's physical environment and the identification position, size, orientation, and motion of objects in the environment; the motion or other state of a user interface display, such as the motion of a virtual reality headset. The sensor may be built into the wearable device, or may be non-wearable, including a display device, or in auxiliary equipment such as a smart phone, smart watch, or implanted medical monitoring device. The sensor may also be placed in a nearby device, such as, for example, an Internet-connected microphone and / or camera array device for hands-free network access, or in an array placed on a physical set.
[0073] Sensor data from one or more sensors 328 may be processed locally by the CPU 302 to control display output, and / or transmitted to the server 200 to be processed by the server in real time or for non-real time processing. As used herein, "real time" refers to processing in response to user input without any arbitrary delay between input and output; that is, reacting as quickly as technically feasible. "Non-real time" or "offline" refers to batch processing or other uses of sensor data that are not used to provide immediate control input to control a display, but may control a display after an arbitrary amount of delay.
[0074] In order to be able to communicate with another node of the computer network (e.g., the communication content server 200), the client 300 may include a network interface 322, such as a wired or wireless Ethernet port. Network communication can be used, for example, to achieve a multi-player experience, including an immersive or non-immersive experience of communication content. The system can also be used for non-directional multi-user applications, such as social networks, group entertainment experiences, teaching environments, video games, etc. Network communication can also be used for data transfer between the client and other nodes of the network, and its purpose includes data processing, content delivery, content control and tracking. The client can use a communication module 306 to manage communication with other network nodes, which handles application-level communication requirements and lower-level communication protocols, preferably without user management.
[0075] The display 320 may be coupled to the processor 302, for example, via a graphics processing unit 318 integrated into the processor 302 or in a separate chip. The display 320 may include, for example, a flat-panel color liquid crystal (LCD) display illuminated by a light emitting diode (LED) or other lamp, a projector driven by an LCD display or by a digital light processing (DLP) unit, a laser projector, or other digital display device. The display device 320 may be incorporated into a virtual reality headset or other immersive display system, or may be a computer monitor, a home theater or television screen, or a projector in a screening room or theater. In a real live theater application, the client of the user and actor may avoid the use of an auxiliary display, or auditory input through a headset or on-site, or tactile impressions through a tactile suit.
[0076] The video output driven by the mixed reality display engine operating on the processor 302, or other applications for coordinating user input with immersive content display and / or generating display, can be provided to the display device 320 and output to the user as a video display. Similarly, the amplifier / speaker or other audio output transducer 316 can be coupled to the processor 302 via the audio processor 312. The audio output associated with the video output and generated by the media player module, the communication content control engine or other applications can be provided to the audio transducer 316 and output to the user as auditory sound. The audio processor 312 can receive analog audio signals from the microphone 314 and convert them into digital signals for processing by the processor 302. The microphone can be used as a sensor for detecting a neural (e.g., emotional) state, and as a device for user input for verbal commands or social verbal responses to non-player characters or other player actors.
[0077] The 3D environment device 300 may also include a random access memory (RAM) 304 that stores program instructions and data for rapid execution or processing by the processor during control of the communication content in response to biometric sensor data collected from the user. When the device 300 is powered off or inactive, the program instructions and data may be stored in a long-term storage, such as a non-volatile magnetic, optical, or electronic memory storage device (not shown). Either or both of the RAM 304 or the storage device may include a non-transitory computer-readable medium that stores program instructions that, when executed by the processor 302, cause the device 300 to perform the methods or operations described herein. The program instructions may be written in any suitable high-level language (e.g., C, C++, C#, JavaScript, PHP, or Java). TM ) and compiled to produce machine language code for execution by a processor.
[0078] Program instructions can be grouped into functional modules 306, 308 to promote coding efficiency and understandability. Communication module 306 can include coordinating biometric sensor data (if metadata) to the communication of the computing server. Sensor control module 308 can include controlling sensor operation and processing raw sensor data for transmission to the computing server. Modules 306, 308, even if they can be identified as partitions or groups in the source code, are not necessarily identifiable as separate code blocks in machine-level coding. Code bundles directed to specific types of functions can be considered to include modules, regardless of whether the machine code on the bundle can be executed independently of other machine codes. These modules can be high-level modules only. Media player module 308 can perform the operations of any method described herein and equivalent methods in whole or in part. Operations can be performed independently, or operations can be performed in collaboration with one or more other network nodes (e.g., server 200).
[0079] The content control methods disclosed herein may be used with virtual reality (VR) or augmented reality (AR) output devices, such as in virtual chats, immersive content, or non-immersive content. Figure 4 is a schematic diagram showing one type of immersive VR stereoscopic display device 400 as an example of a client 300 having a more specific form factor. The client device 300 can be provided in a variety of form factors, where the device 400 provides only one example of a client device 300. The innovative methods, apparatus, and systems described herein are not limited to a single form factor. As used herein, a "communication content signal" includes any digital signal for output of communication content, which can be interactive or non-interactive. In one aspect, the communication content can change in response to a detected neural state of a user computed from biometric sensor data.
[0080] The immersive VR stereoscopic display device 400 may include a tablet support structure made of an opaque lightweight structural material (e.g., a rigid polymer, aluminum, or cardboard) configured to support and allow removable placement of a portable tablet or smart phone device including a high-resolution display screen (e.g., an LCD display). The device 400 is designed to be worn close to the user's face so that a wide field of view can be achieved using a small screen size (such as in a smart phone). The support structure 426 holds a pair of lenses 422 relative to the display screen 412. These lenses can be configured to enable the user to comfortably focus on the display screen 412, which can be held approximately one to three inches from the user's eyes.
[0081] The device 400 may also include a viewing shield (not shown) coupled to the support structure 426 and constructed of a soft, flexible or other suitable opaque material to form a shape that fits the user's face and blocks external light. The shield can be configured to ensure that the only visible light source to the user is the display screen 412, thereby enhancing the immersive effect of using the device 400. A screen divider can be used to separate the screen 412 into independently driven stereoscopic areas, each of which is visible only through a corresponding one of the lenses 422. Therefore, the immersive VR stereoscopic display device 400 can be used to provide a stereoscopic display output, thereby providing the user with a more realistic perception of 3D space.
[0082] The immersive VR stereoscopic display device 400 may also include a bridge (not shown) for positioning over the user's nose to facilitate accurate positioning of the lens 422 relative to the user's eyes. The device 400 may also include an elastic strip or band 424 or other headgear for fitting around the user's head and holding the device 400 to the user's head.
[0083] The immersive VR stereoscopic display device 400 may include additional electronic components of a display and communication unit 402 (e.g., a tablet or smart phone) associated with the user's head 430. When wearing the support 426, the user views the display 412 through a pair of lenses 422. The display 412 can be driven by a central processing unit (CPU) 403 and / or a graphics processing unit (GPU) 410 via an internal bus 416. The components of the display and communication unit 402 may also include, for example, one or more transmit / receive components 418, enabling wireless communication between the CPU and an external server via wireless coupling. The transmit / receive component 418 may operate using any suitable high-bandwidth wireless technology or protocol, including, for example, cellular phone technology, such as a third, fourth, or fifth generation partnership project (3GPP) long term evolution (LTE) (also known as 3G, 4G, or 5G), a global system for mobile communications (GSM) or a universal mobile telecommunications system (UMTS) and / or a wireless local area network (WLAN) technology, for example, using protocols such as the Institute of Electrical and Electronics Engineers (I18E) 802.11. One or more transmit / receive components 418 may enable video data to be streamed from a local or remote video server to the display and communication unit 402, and may uplink sensor and other data to a local or remote video server for use in control or audience response techniques as described herein.
[0084] Components of the display and communication unit 402 may also include one or more sensors 414 coupled to the CPU 403, for example, via a communication bus 416. Such sensors may include, for example, an accelerometer / inclinometer array that provides orientation data indicating the orientation of the display and communication unit 402. Since the display and communication unit 402 is fixed to the user's head 430, the data may also be calibrated to indicate the orientation of the head 430. The one or more sensors 414 may also include, for example, a global positioning system (GPS) sensor that indicates the user's geographic location. The one or more sensors 414 may also include, for example, a camera or image sensor that is positioned to detect the orientation of one or more of the user's eyes, or to capture a video image of the user's physical environment (for VR mixed reality), or both. In some embodiments, a camera, image sensor, or other sensor configured to detect the user's eyes or eye movement may be mounted in the support structure 426 and coupled to the CPU 403 via the bus 416 and a serial bus port (not shown) (e.g., a universal serial bus (USB) or other suitable communication port). One or more sensors 414 may also include, for example, an interferometer positioned in the support structure 426 and configured to indicate a surface profile to the user's eyes. One or more sensors 414 may also include, for example, a microphone, a microphone array, or other audio input transducer for detecting spoken user commands or verbal and non-verbal auditory responses to display output. One or more sensors may include a subvocalization mask using electrodes as described in a paper published by Arnav Kapur, Pattie Maes, and Shreyas Kapur at the 2018 Association for Computing Machinery's ACM Intelligent User Interface conference. The words recognized by the human voice can be used as command input, as an indication of arousal or evaluation, or both. The one or more sensors may include, for example, electrodes or microphones for sensing heart rate, a temperature sensor configured to sense the user's skin or body temperature, an image sensor coupled to an analysis module to detect facial expressions or pupil dilation, a microphone for detecting verbal and non-verbal vocalizations, or other biometric sensors (including those that have been combined with the user's sensor) for collecting biofeedback data (including neural responses indicative of emotions that can be processed via algorithms). Figure 3 any sensor described at 328).
[0085] Components of the display and communication unit 402 may also include, for example, an audio output transducer 420, such as a speaker or piezoelectric transducer in the display and communication unit 402, or an audio output port for headphones or other audio output transducers installed in the headwear 424, etc. The audio output device can provide surround sound, multi-channel audio, so-called "object-oriented audio" or other soundtrack output accompanying the stereoscopic immersive VR video display content. Components of the display and communication unit 402 may also include, for example, a memory device 408 coupled to the CPU 403 via a memory bus. The memory 408 may store, for example, program instructions that, when executed by the processor, cause the device 400 to perform the operations described herein. The memory 408 may also store data (e.g., audio-video data) in a library or buffer it during streaming from a network node.
[0086] Having described examples of suitable clients, servers, and networks for performing signal processing of biometric sensor data to detect neural states in communication enhancement applications, more detailed aspects of suitable signal processing methods will be discussed. Figure 5 An overview of a method 500 for computing content engagement capabilities (CEP) is shown, which may include four related operations performed in any functional order or in parallel. As described herein, these operations may be programmed as executable instructions for a server.
[0087] Correlation operation 510 uses an algorithm to correlate biometric data for a user or group of users with neural indicators. Optionally, the algorithm can be a machine learning algorithm that is configured to process contextual indicator data in addition to the biometric data, which can improve accuracy. The contextual indicator data can include, for example, user orientation, user location, time of day, day of the week, ambient light level, ambient noise level, etc. For example, if the user's background is full of distractions, the saliency of the biofeedback data may be different than in a quiet environment.
[0088] As used herein, a "measure" or "indicator" of a neurophysiological response is a machine-readable symbolic value associated with a segment of the communication content. The indicator or measure can have constituent elements, which can be quantitative or non-quantitative. For example, the indicator or measure can be designed as a multidimensional vector, whose values represent the strength of psychological qualities (such as cognitive load, arousal, and evaluation). Psychological evaluation is the state of attractiveness or desirability of an event, object, or situation; when the subject feels that something is good or attractive, the evaluation is said to be positive, and when the subject feels that the object is repulsive or bad, the evaluation is said to be negative. Arousal is the state of alertness and attention of the subject. The machine learning algorithm may include at least one supervised machine learning (SML) algorithm, for example, one or more of a linear regression algorithm, a neural network algorithm, a support vector algorithm, a naive Bayes algorithm, a linear classification module, or a random forest algorithm.
[0089] The event detection operation 520 analyzes time-related signals from one or more sensors during the output of communication content to the user, and detects events in which the signal exceeds a threshold. The threshold can be a fixed predetermined value, or a variable number such as a rolling average. Examples of GSR data are provided herein below. A discrete measure of the neurophysiological response can be calculated for each event. The neural state cannot be measured directly, so the sensor data indicates emotional modulation. Emotional modulation is the modulation of a biometric waveform attributed to a neural state or a change in a neural state. In one aspect, in order to obtain a baseline correlation between emotional modulation and neural state, a known visual stimulus (e.g., from a focus group test or a personal calibration session) can be displayed to the user to induce a certain type of emotion. Under stimulation, the test module can capture the user's biometric data and compare the stimulus biometric data with the static biometric data to identify emotional modulation in the biometric data waveform.
[0090] CEP measurements and related methods can be used as drivers for branching (configurable) content. The measured error between the target storyline and the group response can help inform the design of branching content, the design and production of future content, distribution and marketing, or any activity that is influenced by the neurophysiological response of a cohort group to content. Additionally, the measured error can be used in a computer-implemented theater management module to control or influence real-time narrative branching or other management of the theater experience. During focus group testing, it may be useful to use a smartphone or tablet because such programmable devices already contain one or more sensors for collecting biometric data. For example, Apple TM iPhone TMIncludes front-facing stereo cameras that can be used, for example, for eye tracking, FAU detection, pupil dilation measurement, heart rate measurement, and ambient light tracking. Participants in the focus group may view the content on a smartphone or similar device, which collects biometric data with the participant's permission through a focus group app running on the device they are viewing.
[0091] A normalization operation 530 performs an arithmetic or other numerical comparison between the test data for the known stimulus and the measured signal for the user and normalizes the measured values for the event. Normalization compensates for variations in the individual responses and provides a more useful output. Once the input sensor events are detected and normalized, an operation 540 determines CEP values for the user or group of users and records these values in a time-dependent record in computer memory.
[0092] Machine learning (also known as AI) can be an effective tool for uncovering correlations between complex phenomena. Figure 6 As shown, a system 600 responsive to sensor data 610 indicating a user's neural state can use a machine learning training process 630 to detect correlations between sensory and narrative stimuli 620 from a live theater experience and biometric data 610. The training process 630 can receive stimulus data 620 temporally correlated with the biometric data 610 from a media player client (e.g., client 300, 402). The data can be associated with a specific user or user group, or can be universal. Two types of input data (user-associated and universal) can be used together. Universal input data can be used to calibrate a baseline of neurophysiological responses to classify baseline neurophysiological responses to scenes or arrangements of movie elements. For example, if most users show similar biometric tell when watching a scene within a narrative context, the scene can be classified together with other scenes that stimulate similar biometric data from the user. Similar scenes can be collected and reviewed by a human creator, who can score the scene on a neural indicator metric 640 using an automated analysis tool. In the alternative, indicator data 640 may be scored by humans and semi-automatic processing without categorizing by similar scenes. Human-scored elements of live theater productions may become training data for machine learning process 630. In some embodiments, human-scored elements of communication content may include users, such as via an online questionnaire. Scoring should take into account cultural demographics and should be informed by experts about how different cultures respond to scene elements.
[0093] The ML training process 630 compares the scores of scenes or other movie elements determined by humans and machines, and uses iterative machine learning methods known in the art to reduce the error between the training data and its own estimates. Creative content analysts can score data from multiple users based on their professional judgment and experience. Individual users can score their own content. For example, a user who is willing to assist in training his personal "director software" to identify his neurological state can score his own emotions while watching content. The problem with this approach is that user ratings may interfere with their normal reactions, thereby misleading the machine learning algorithm. Other training methods include clinical testing of subjects' biometric responses in shorter content clips and then surveying clinical subjects for their neurological conditions. A combination of these and other methods can be used to develop training data for the machine learning process 630.
[0094] As used herein, biometric data provides a "tell" about how users think and feel about their experience of communication content, i.e., whether they engage with the sense of entertainment value in narrative theory. Content engagement capacity is a measure of overall engagement with communication content throughout the user experience, and is monitored and scored during and after the experience is completed. Overall user enjoyment is measured as the difference between expected biometric data modulation capacity (measured during calibration) and average sustained biometric data modulation capacity. Measures of user engagement can be determined by other methods and related to "content engagement capacity" or made part of "content engagement capacity". For example, exiting an interview response or accepting an offer to purchase, subscribe, or follow can be included in or used to adjust the calculation of content engagement capacity. Offer response rates can be used during or after content presentation to provide a more complete measure of user engagement.
[0095] The mood in which the user enters the interaction affects how the "story" is interpreted, so the story experience should be calibrated whenever possible. If the process is not able to calibrate for mood, then that mood can be taken into account in the presented storyline to support more positive evaluative interactions, provided we can measure evaluativeness from the user. Instant systems and methods will work best for healthy and calm individuals, providing an interactive experience for everyone involved.
[0096] Fig. 7A An arrangement 700 of neural states relative to axes of a two-dimensional neural space defined by a horizontal appreciation axis and a vertical arousal axis is shown. The illustrated emotions based on the appreciation / arousal neural model are shown in this arrangement only as examples and not actual or typical measurements. The media player client may measure appreciation using a biometric sensor that measures facial action units, while arousal measurements may be accomplished, for example, through GSR measurements.
[0097] Neural space can be characterized by more than two axes. Figure 7B A three-dimensional cognitive discrimination model 750 of a neural space is illustrated, wherein the third axis is social dominance or confidence. Model 750 shows a VAD (appraisal, arousal, confidence) cognitive discrimination model. 3D model 750 may be useful for complex emotions involving social levels. In another embodiment, the engagement measure from biometric data may be modeled as a three-dimensional vector, which provides cognitive workload, arousal, and appraisal, from which the processor may determine the primary and secondary emotions after calibration. The engagement measure may be generalized to an N-dimensional model space, wherein N is one or greater. In the example described herein, the CEP as described herein is in a two-dimensional space 700 with an appraisal axis and an arousal axis, but the available cognitive discrimination model is not limited thereto. For example, confidence is another psychometric axis that may be added, other axes may be added, and the basic axes other than appraisal and arousal may also be useful. Any cognitive discrimination model that maps biometric data to different cognitive parameters in a vector space may be used together with the method described herein. During emotional calibration, baseline arousal, appraisal, dominance, and other cognitive parameters may be determined based on the individual.
[0098] In the detailed example below, neural state determination from biometric sensors is based on an appreciation / arousal neural model, where appreciation is (positive / negative) and arousal is magnitude. With this model, producers of live theater and other creative productions can verify the intent of the creative work by measuring narrative theory structures such as tension (hope vs. fear) and tension buildup (arousal increases over time). During the presentation of story elements, either live or recorded, an algorithm can use the neural model to dynamically change story elements based on the user's psychology, as described in more detail in U.S. Provisional Patent Application 62 / 614,811 filed on January 8, 2018. The focus of the present disclosure is on determining useful measures of neural states associated with engagement in targeted entertainment (CEP) for real-time and offline applications, as described in more detail below. The inventive concepts described herein are not limited to the neural models described herein, and may be adapted for use with any useful neural model characterized by quantifiable parameters.
[0099] In a test environment, electrodes and other sensors can be manually placed on a subject user in a clinical function. For consumer applications, the placement of sensors should be less intrusive and more convenient. For example, image sensors for visible and infrared wavelengths can be built into display equipment. For another example, a phased array radar transmitter can be made as a miniature device and placed behind the display of a mobile phone or tablet to detect biometric data such as facial action units or pupil dilation. When a user wears jewelry or grabs a controller while using VR equipment, electrodes can be built into headgear, controllers, and other wearable jewelry to measure skin conductivity, pulse, and electrical activity.
[0100] The target storyline based on the content of the communication can be stored in a computer database as a sequence of target values in any useful neural model for assessing engagement with the content, such as a rating / arousal model. Using the example of the rating / arousal model, the server can perform a difference operation to determine the error between the planned / predicted and measured arousal and rating. The error can be used for content control. Once the delta between the predicted and measured values exceeds a threshold, the story management software can select the next or future segment of the communication. For example, if the user's rating is in the "wrong" direction based on the target storyline, the processor can change the content through the following logic: if the absolute value of (predicted rating - measured rating)>0, then change the content. The change in content can be dedicated to several different items that the software knows about the user, or it can be a trial or recommendation from an AI process. Similarly, if the arousal error drops below a predicted threshold (e.g., 50%) (absolute value of error <0.50*prediction), the processor can select a different content segment.
[0101] Figure 8 A method 800 for determining a content rating of communication content including content engagement capability (CEP) is shown. The method may be implemented by encoding as an algorithm executable by a computer processor and may be applied to other methods described herein wherever a CEP calculation is required. CEP is the sum of the event capabilities of the subject content, "CEP". v "Expected capabilities compared to comparable content in this genre" x For different topics, and usually for different users, the same method is used to calculate P v and P x In this way, the sum covers different total times, event capacity P v Covering the time period "t v ", the time period "t v ” is equal to the “n” event capacity period Δt of the subject content v The sum of:
[0102]
[0103] Similarly, the expected capability P x Covering period "t x ", the period "t x ” is equal to the “m” event capacity period Δt of the desired content x The sum of:
[0104]
[0105] For any given event "n" or "m", the capability P v and P x Each of them is the dot product of the capability vector P and the weight vector W of dimension i, as shown below:
[0106]
[0107]
[0108] Typically, the capability vector Various definitions may be made. In any given CEP calculation, the capability vector and the expected baseline of the subject content should be defined consistently with each other, and the weighting vectors should be the same. The capability vector may include only arousal metrics, only rating values, a combination of arousal metrics and rating metrics, or any of the foregoing in combination with other metrics (e.g., confidence metrics). In one embodiment, a matrix consisting of "j" arousal metrics "a" is used. j ” and “k” evaluation metrics “v k The capability vector defined by the combination of The CEP is calculated by adjusting each measure by a calibration offset “C” relative to a known stimulus, where j and k are any non-negative integers, as follows:
[0109]
[0110] in
[0111] C j =S j -S j O j =S j (1-O j ) Equation 6
[0112] The index "j" in Equation 6 represents the index from 1 to j+k, S j represents the scale factor, and O j Represents the offset between the minimum value of the sensor data range and its true minimum value. The weight vector corresponding to the capability vector of Equation 5 can be expressed as:
[0113]
[0114] Each of these weight values scales its corresponding factor in proportion to the relative estimated reliability of the factor.
[0115] Using the calibrated dot product given by Equation 3 and Equation 4 and the time factor given by Equation 1 and Equation 2, the processor can calculate the content engagement capability (CEP) of a single user as follows:
[0116]
[0117] Ratio t x / t v Normalize for inequalities in the sums of different time series and make ratios unitless. User CEP values greater than 1 indicate that the user / player actor / viewer had an engagement experience that exceeded their expectations relative to the genre. User CEP values less than 1 indicate that engagement was less than the user expected for the content genre.
[0118] CEP can also be calculated for content titles, scenes in live theater, and entire live theater productions across an audience of "v" users, based on the ratio of the content event capabilities of "x" users to the expected capabilities of "m" users, which are not necessarily the same, as follows:
[0119]
[0120] The variables v and x are the number of content users and the number of participating benchmark audiences, respectively. The audience expected capacity in the denominator represents the audience's expectations of the content, while the event capacity in the numerator represents the sum of the audience's arousal or evaluation events when experiencing the content. The processor sums the event capacity for each event (n) and user (v) and the expected capacity for each event (m) and user (x). Then, by calculating the ratio of event capacity to expected capacity, and by the ratio xt x / vt v CEP is calculated by normalizing the different time sums and the number of audiences. CEP is a component of the content rating. Other components of the content rating may include the aggregate rating error and the rating error for the rating target (e.g., victory, despair, etc.).
[0121] Equation 5 describes a calibrated capability vector consisting of arousal and evaluative measures derived from biometric sensor data. In the alternative, the processor may define a partially uncalibrated capability vector where the sensor data signal is scaled before conversion to digital values as part of low-level digital signal processing but is not offset for the user, as shown below:
[0122]
[0123] If a partially uncalibrated capability vector is used, then the aggregate calibration offset can be calculated for each factor before computing the content engagement capability (CEP) and given by the dot product given by Equation 3 and Equation 4 The aggregate calibration offset is subtracted from . For example, The aggregate calibration offset of can be given by:
[0124]
[0125] In this case, the capability vector can be calculated by the following equation The calibrated value of:
[0126]
[0127] The calibrated capability vector can be calculated similarly
[0128] Referring again to method 800 where the aforementioned expression may be used ( Figure 8 ), first a calibration process 802 of the sensor data is performed to calibrate the user's response to known stimuli (e.g., a known static stimulus 804, a known arousing stimulus 806, a known positive evaluation stimulus 808, and a known negative evaluation stimulus 810). The known stimuli 806-810 can be tested using a focus group that is culturally and demographically similar to the target audience and maintained in a database for use in calibration. For example, the International Affective Picture System (IAPS) is a picture database used to study emotions and attention in psychological research. To be consistent with the content platform, images or those found in the IAPS or similar knowledge base can be produced in a format consistent with the target platform for use in calibration. For example, pictures of topics that trigger emotions can be produced as video clips. Calibration ensures that the sensor operates as expected and provides consistent data between users. Inconsistent results may indicate that the sensor is malfunctioning or misconfigured, which can be corrected or ignored. The processor can determine one or more calibration coefficients 816 for adjusting signal values for consistency between devices and / or users.
[0129] Calibration can have scaling and offset characteristics. In order to be used as an indicator of arousal, appraisal, or other psychological state, sensor data may need to be calibrated using both scaling factors and offset factors. For example, GSR may theoretically vary between zero and 1, but in practice depends on the fixed and variable conditions of human skin, which vary from individual to individual and over time. In any given scenario, a subject's GSR may be between some GSR greater than 0. min To a GSR less than 1 max The magnitude of the range and its scaling can be measured by exposing a subject to a known stimulus and estimating the magnitude and scaling of the calibration factor. This estimate is made by comparing the results from a scenario with a known stimulus to the expected range for the same type of sensor. In many cases, the reliability of the calibration may be questionable, or calibration data may not be available, and it is necessary to estimate the calibration factor from the field data. In some embodiments, the sensor data can be pre-calibrated using an adaptive machine learning algorithm that adjusts the calibration factor for each data stream as more data is received, and eliminates the higher level processing from the task of making adjustments for calibration.
[0130] Once the sensor is calibrated, the system normalizes the sensor data response data for genre differences at 812 using, for example, Equation 8 or Equation 9. Different genres produce different ratings and arousal scores. For example, the adventure action genre has different pace, story goals, and intensity. Therefore, unless the genre's engagement profile is considered, it is impossible to compare engagement capabilities across genres. Genre standardization scores content relative to content in the same genre, enabling comparisons on an equivalent basis between genres. Standardization 812 can be performed on a test audience or focus group or subject group before a main feature using an expected standardized stimulus 814. For example, an audience can watch one or more trailers of the same genre as the main feature, and event capabilities can be calculated for one or more trailers. In an alternative, archived data of the same user or the same user group can be used to calculate expected capabilities. The expected capabilities are calculated using the same algorithm as used or to be used for the measurement of event capabilities, and the expected capabilities can be adjusted using the same calibration coefficient 816. The processor stores the expected capabilities 818 for later use.
[0131] At 820, the processor receives sensor data during playback of the subject content and computes event capabilities for each metric of interest, such as arousal and one or more evaluative qualities. At 828, the processor sums or aggregates the event capabilities of the content after playback ends or on a running basis during playback. At 830, the processor computes the content rating, including content engagement capabilities (CEP) as previously described. The processor first applies the applicable calibration factor and then computes the CEP by dividing the aggregated event capabilities by the expected capabilities, as described above.
[0132] Optionally, the operation function 820 may include comparing the event capabilities of each detected event or a smaller subset of the detected events with a reference storyline defined for the content at 824. The reference storyline may be, for example, a target storyline defined by a creative producer, a predicted storyline, one or more past stories of the content, or a combination of the foregoing stories. At 826, the processor may save, increment, or otherwise accumulate error vector values describing errors in one or more variables. The error vector may include the difference between the reference storyline and the measured response for each measurement value (e.g., arousal and evaluation values) for a specified scene, time period, or a set of video frames. Error vectors and vector matrices may be used for content evaluation or content control.
[0133] Error measurement can include or increase other indicators for content evaluation. Content participation ability and error measurement can be compared with purchase, subscription or other conversions related to the presented content. The system can also use standard deviation or other statistical measures to measure the consistency of audience response. The system can measure content participation ability, evaluation and arousal for individuals, queues and aggregated audiences. Error vectors and CEP can be used for various real-time and offline tasks. In some embodiments, these measures can be used for content control, for example, as described in U.S. Provisional Patent Application No. 62 / 566,257 filed on September 29, 2017 and No. 62 / 614,811 filed on January 8, 2018, which are incorporated herein by reference.
[0134] Further details of digitally representing user engagement with language content, including but not limited to digitally representing content engagement capabilities (CEP) based on biometric sensor data, can be described in U.S. Patent Application No. 62 / 661,556 filed on April 23, 2018. Digitally representing user engagement in computer memory based on biometric data can find many applications, some of which are further described below. These applications include: generating scripts for actors in interactive or non-interactive performances, generating information useful to script writers and other creative producers, controlling progress through language work, and other uses claimed herein.
[0135] In one aspect of controlling the content of communications driven by biometric input, a user may use a mobile application (eg, "Sage QCI") and a device for signaling biometric data and other information. Fig. 9 A system 900 is shown for collecting and using biometric response data from a person 902 (e.g., a user or player actor) for interactive entertainment using an application or system 904 installed on a mobile device 920 (e.g., a smart phone). One or more biometric sensors may be coupled to the mobile device 920 via a Bluetooth connection 918 or other suitable coupling means. In an alternative, the sensor may be built into the mobile device 920 and communicate with its processor via a bus or serial port. The biometric sensors 906 may include an electroencephalogram (EEG) sensor 908, a galvanic skin response sensor 910, an electrocardiogram sensor 912, an eye tracking and facial expression sensor 914, and an orientation sensor 916. The processor of the mobile device 920 may transmit the raw sensor data to a cloud-based data processing system 924, which generates a measure of content engagement 956 (e.g., CEP) and other processed data 950. The content engagement software 956 may be provided to a story management module or application 958 for controlling communication content as described herein. Other processed data 950 may include, for example, usage analytics data 952 for a content title and trend data 954 aggregated over one or more content titles.
[0136] Other inputs to the data analysis system 924 may include batched raw sensor data 948. The batched data 948 may be collected in non-real time and stored offline, such as in a personal computing device 942, which stores batched biometric data 944 in a local data storage device, which may be uploaded to the data analysis server 924 from time to time via a website 946 or other portal. For example, offline or non-real time data may be useful for developing user profiles or retrospective analysis.
[0137] The data analysis system 924 can perform distributed processing at two update rates (fast packets and slow packets). The mobile device 920 can process raw biometric data in fast mode and send data summaries to the cloud analysis system 924 for further processing only through data packets. In slow mode, raw data files can be uploaded at a slower data rate for post-session processing. The data analysis system 924 can be configured differently. In some embodiments, the server 924 may include an AmazonTM Kinesis front end 926 for receiving, caching and serving raw data input within the analysis system 924. The data processing component 928 can use machine learning and rule-based algorithms to process raw biometric data, as described elsewhere herein. The processed data can be exchanged with long-term storage units 932 and 934. For convenience, a serverless computing platform 936 (e.g., Amazon lambda (λ)) can be used to provide code execution and scaling without the overhead of managing instances, availability and runtime on the server. The provision of processed data 930 from the data analysis system 924 can be managed via an application program interface (API) 938.
[0138] Fig.10 A mobile system 1000 is shown for a user 1002, including a mobile device 1004 having sensors and accessories 1012, 1020 for collecting biometric data used in the methods and apparatus described herein, and a display screen 1006. The mobile system 1000 can be useful for real-time control or non-real-time applications (such as traditional content range focus group testing). The mobile device 1004 can use built-in sensors typically included on consumer devices (phones, tables, etc.), for example, a front-facing stereo camera 1008 (portrait) or 1010 (landscape). The cameras 1008, 1010, which are typically built-in by the manufacturer for facial detection authentication, can also be used to track attention through eye tracking, FAU for tracking CEP evaluation, and pupil dilation measurement to track CEP arousal and heart rate, which can be obtained through a watch accessory 1014 including a pulse detection sensor 1014 or through the mobile device 1004 itself.
[0139] Accessories such as headphones 1020, hats, or VR headsets can be equipped with EEG sensors 1022. The processor of the mobile device can detect arousal through pupil dilation via the 3D cameras 1008, 1010 that also provide eye tracking data. A calibration scheme can be used to distinguish pupil dilation caused by aperture (light changes) from changes to emotional arousal. Both the front and rear cameras of the device 1004 can be used for ambient light detection to correct pupil dilation detection, thereby excluding dilation caused by lighting changes. For example, the relationship between the pupil dilation distance (mm) and the dynamic range of light expected during the performance for expected ambient light conditions can be measured during the calibration sequence. Based on this, the processor can calibrate the relationship between the lighting effect and the effect of emotional or cognitive workload based on the design of the narrative by measuring additional dilation displacements of narrative elements and the results of the calibration signal test.
[0140] Instead of or in addition to the stereo cameras 1008 or 1010, the mobile device 1004 may include a radar sensor 1030 (e.g., a multi-element microchip array radar (MEMAR)) to create and track facial action units and pupil dilation. The radar sensor 1030 may be embedded below and may be seen through the screen 1006 on the mobile device 1004, regardless of whether there is visible light on the subject. The screen 1006 is invisible to the RF spectrum radiated by the imaging radar array, so that radar imaging can be performed through the screen in any degree of light and dark. In one aspect, the MEMA sensor 1030 may include two arrays, each with 6 elements. Two small RF radar chip antennas, each with six elements, constitute the imaging radar. The advantage of the MEMA sensor 1030 over the optical sensors 1008 and 1010 is that no illumination of the face is required, so darkness does not hinder the sensing of facial action units, pupil dilation, and eye tracking. Although only one 6-chip MEMA array 1030 is shown, the mobile device may be equipped with two or more similar arrays for more robust sensing capabilities.
[0141] Certain aspects of the foregoing methods and apparatus may be adapted for use in screenwriting applications for interactive entertainment, including an application interface that allows screenwriters to define variables related to psychological profiles of players and characters. The application may enable screenwriters to create stories by defining variables and creating matching content. For example, writers may track player parameters (such as personality, demographics, socioeconomic status) during scriptwriting, and set variables (determined by the writer) for how the script branches based on the player parameters. In addition, the application may also enable writers to place branches in scripts that depend on the neural state of the player. The application may facilitate the development of branches during readback by presenting the selections as drop-down menus or links similar to a choose your own adventure book. Screenwriters may manage and create branches via a graphical interface and in a scripting environment. The application may assist screenwriters in managing non-player character profiles, such as by making suggestions for dialogues and actions based on player profiles and other non-player characters' actions in scenes, and also by making suggestions through the interaction of players and other non-players.
[0142] A draft of the script can be made by simulating character interactions using a personality model. Based on the available character data profile information, the script writing application can use machine learning and experiments (e.g., player actor experiments) to construct a script for a traditional linear narrative through simulation. Each "played" path through the simulation can be converted into a linear script based on the data collected about how the simulated player actor performs during the simulation. For example, the recorded interactions, dialogues, and other elements depend on all biometric sensor data and player actor / NPC character profile data. The application can compare alternative drafts and determine the most likely to succeed. The suggestion can be based primarily on personal profile data matching and matching across genres, demographics, background stories, character types / roles related to narrative structure. The application can use a database built on character profiles / background stories and a database storing player actor trial data, storylines, biometric data, and other relevant data.
[0143] The application can use machine learning to identify patterns in character reactions based on profile data, emotional responses, and interactions (stored player-actor interactions from simulated trials). Draft scripts are based on simulated competitions, conflicts, and other interactions between computer-controlled non-player characters (NPCs). NPC interactions and dialogues can be informed or generated by random selection from a corpus of stored movie data character profiles, storylines, emotional plots, dialogues, and interactions across multiple stories. Permutations (trials from NPC to NPC) are scored against popular storyline data to return a percentage score of likability based on past data. Trials with storyline similarities of more than 95% or 99% to popular stories can be re-adjusted for human analysis.
[0144] In addition to or in lieu of defining major elements such as characters and storyline, composite content design can also use more refined "atomic elements" such as lighting, color schemes, framing, music, point of view (POV) or scene change moments to increase audience engagement with the work, rather than just selecting pre-shot scenes or the next node to be shown. Using feedback based on emotional narrative allows producers to inform and guide designers and scriptwriters and photographers and colorists and music selectors, etc. to create content that makes the audience more engaged. The key point is not just to create a dynamic story or call different NPCs, but to change the more refined aspects of the work ("atomic elements") based on the emotional narrative determined via relevant sensors. This can be used to form a better version to achieve green lighting or redesign production, and do it in real time if possible.
[0145] Synthetic content designs can be used for pre-visualization (pre-viz) of previews, perhaps using different versions that have already been shot, or using CGI and pre-viz hardware to present different alternatives. Depending on the available computing bandwidth, the CGI rendered content can react in real time as the presentation progresses, allowing the audience's preferred lighting, soundtrack, framing, etc. to be incorporated into the output.
[0146] Combination Figure 9-10 The described systems and apparatus can be used to perform Figure 20-21 and methods described elsewhere. Fig.11 An example of components of a system for collecting neurophysiological response data using a mobile device with auxiliary sensor devices and a sequence 1100 therebetween is shown. In one use case, a user 1102 is watching entertainment on another screen of a player device or module 1104, while a smart phone 1106 or similar mobile device acts as a biometric sensor hub and connection node. The smart phone 1106 can display auxiliary information during the presentation of content by the player module 1104, attracting the occasional attention of the user 1102. As the user gazes at the screen of the smart phone 1106, the smart phone can use its biometric sensors to collect neurophysiological data, for example, the phone 1106 can collect FAU and pupil dilation from a glance. If the user touches the touch screen or fingerprint sensor of the phone, the smart phone 1106 can collect pulses. Alternatively or additionally, the smart phone can be installed or otherwise supported so that the user 1102 is within its sensor range when consuming content. The cloud component 1112 can be or can include such as in combination with Fig. 9 Described cloud server 924 or equivalent. Equivalent servers may include conventional single-node servers, server farms, or distributed server systems (e.g., blockchain).
[0147] Optionally, the smartphone 1106 can work in conjunction with one or more auxiliary sensors, such as a wrist device 1108 (such as a smart watch or fitness band), a head-mounted device 1110 (such as a headband, hat, or visor) for collecting EEG and GSR, or a finger-mounted device (not shown). The auxiliary devices 1106, 1108 can communicate wirelessly with the smartphone 1106 using Bluetooth or other wireless protocols. Although the various auxiliary devices operate independently of each other, their order of operation can be the same, so they are shown on the same row.
[0148] In other use cases, the smartphone 1106 and the player module 1104 may be combined in a single device. For example, a user may consume content displayed on the screen of the smartphone 1106 directly or via an intermediary device such as, for example, a virtual reality headset or smart TV that receives a streaming audio-video feed from the smartphone 1106.
[0149] The sequence 1100 may begin with the user 1102 causing the player module 1104 to initiate playback 1114 of a desired communication content title, experience, or work. The smartphone 1106 may detect the initiation of the session via a handshake or detection of a signal from the player module 1104 and collect an identifier 1116 for the content title, experience, or work and for the user. The smartphone 1106 may activate the auxiliary sensor devices 1108, 1110 and provide identification information to the cloud server 1112, which may register the identifier 1118 as a parameter of the session.
[0150] The player module 1120 can play the first segment of the content, such as a video clip, an audio clip, an audio-video clip, a text page, a graphic image, or other language or non-language content, by outputting to its display system, audio system, and / or other available output systems. At 1121 (optional), if the smart phone does not display the content, it can display auxiliary content to attract the user's attention, as described above. At 1122, the smart phone 1106 can collect sensor data from the sensor, as described above. Similarly, at 1124, the auxiliary devices 1108, 1110 can collect sensor data. The smart phone 1106 can collect and organize the sensor data at 1125, optionally preprocessing the data into a more compact form. The smart phone can then send the raw and processed data with the segment identifier to the cloud server 1112, which calculates 1126 a measure of participation (e.g., CEP) or other neurophysiological indicators or error indicators relative to the narrative plot based on the data, as described elsewhere in this article, and saves the calculation results 1128 in the data storage. The cloud server 1112 may provide the result of the operation, or a subset thereof, to the smartphone 1106. Based on the result, the smartphone (or alternatively, the cloud server) selects 1130 the next segment.
[0151] The algorithm for selecting the next segment can depend on the use case. In the end-consumer use case, the algorithm can select the next segment to correct the error relative to the narrative plot to provide a more pleasant or more intense experience, or other goals of the use case. In the focus group use case, the algorithm can select the segment to clarify the user's reaction to the previous segment or other test goals. After selecting the next segment, the smart phone 1106 can send an identifier for the next segment to the player module 1104. At 1132, if the playback of the communication content is not over, the sequence can loop back to 1120. If the playback is over, the smart phone 1106 can terminate the session with the auxiliary sensor device 1108, 1110 and the cloud server 1112 by sending a termination signal. After receiving the termination signal, the cloud server can generate a report 1136 for its data storage and provide a summary to the smart phone 1106, which can be provided to the user 1102.
[0152] In view of the foregoing, and as an additional example, Fig.12 Aspects of one or more methods 1200 for generating a script or a portion of a script using an automated algorithm are shown. The computer-implemented method 1200 for generating a script may include, at 1210, randomizing, by at least one processor, one or more parameters of a cognitive identification model for a participant in a role interaction. The cognitive identification model may be a model as described for computing CEP, or a combination of the above Figure 7A-7B Other models described in block 1200. The method 1200 may include, at 1220, generating a script by at least one processor at least in part by modeling character interactions while varying parameters between different scripts in a plurality of scripts. The method may include, at 1230, for each of the plurality of scripts, estimating, by at least one processor, a measure of effectiveness based on one of a plurality of target storylines for the neurophysiological response, based on a cognitive discrimination model. The method may include, at 1240, selecting, by at least one processor, a preferred script element based on the measure of effectiveness estimated at block 1230. The method may include, at 1250, compiling, by at least one processor, a script including the preferred script element selected at block 1250. The method 1200 may include, at 1260, recording the script in a computer-readable medium.
[0153] The computing device may be configured to perform method 1900 . Fig.13 Components of an apparatus or system 1300 for generating a script or a portion of a script are shown. The apparatus or system 1300 may include additional or more detailed components for performing functional or procedural operations as described herein. For example, the processor 1310 and the memory 1316 may contain an example of a process for randomizing one or more parameters of a cognitive discrimination model for a simulated participant in a role interaction. As depicted, the apparatus or system 1300 may include functional blocks capable of representing functions implemented by a processor, software, or a combination thereof (e.g., firmware).
[0154] like Fig.13 As shown, the device or system 1300 may include an electronic component 1302 for randomizing one or more parameters of a cognitive discrimination model for a simulated participant in a role interaction. Component 1302 may be or may include a device for said randomization. The device may include a processor 1310, which is coupled to a memory 1316 and coupled to the output of at least one biometric sensor, and the processor executes an algorithm based on program instructions stored in the memory. Such an algorithm may include, for example, developing a result table for role interaction based on a personality interaction model, identifying a set of results for two or more roles entering the interaction, generating a random number, and selecting one of a set of results based on the random number.
[0155] The apparatus 1300 may also include an electronic component 1304 for generating a script at least in part by modeling character interactions while varying parameters between different scripts in a plurality of scripts. Component 1304 may be or may include a device for said generating. The apparatus may include a processor 1310 coupled to a memory 1316, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as randomly selecting a storyline and a character for a script, configuring a simulated personality for a character, and generating a series of outcomes for the character along the selected storyline.
[0156] The apparatus 1300 may also include an electronic component 1306 for estimating a measure of effectiveness based on one or more target storylines for each of the plurality of scripts, based on a cognitive discrimination model for neurophysiological responses. The cognitive discrimination model predicts audience responses by correlating known or estimated neural and cognitive characteristics of the target audience with the likelihood of neurophysiological responses. Component 1306 may be or may include a device for said estimation. The apparatus may include a processor 1310 coupled to a memory 1316, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as establishing an audience profile characterized by an aggregation of cognitive discrimination models, and generating estimated individual responses using an evaluation-arousal model or other suitable model, and averaging the individual responses to generate an estimated aggregate audience response for each draft script.
[0157] The apparatus 1300 may also include an electronic component 1308 for recording an estimated level of effectiveness associated with each draft script. Component 1308 may be or may include a device for such recording. The apparatus may include a processor 1310 coupled to a memory 1316, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as writing the estimated effectiveness to a computer database. The creative producer may use the stored estimates for further content development or editing.
[0158] The device 1300 may optionally include a processor module 1310 having at least one processor. The processor 1310 may be in operative communication with the modules 1302-1308 via a bus 1313 or similar communication coupling. In an alternative, one or more modules may be instantiated as functional modules in a memory of the processor. The processor 1310 may initiate and schedule processes or functions performed by the electronic components 1302-1308.
[0159] In a related aspect, the apparatus 1300 may include a network interface module 1312 or an equivalent I / O port operable to communicate with system components via a computer network. The network interface module may be or may include, for example, an Ethernet port or a serial port (e.g., a Universal Serial Bus (USB) port), a Wi-Fi interface, or a cellular telephone interface. In a further related aspect, the apparatus 1300 may optionally include a module for storing information, such as a memory device 1316. The computer-readable medium or memory module 1316 may be operably coupled to other components of the apparatus 1300 via the bus 1313 or the like. The memory module 1316 may be adapted to store computer-readable instructions and data to affect the processes and behaviors of the modules 1302-1308 and their subcomponents or the processor 1310, or to affect the method 1200. The memory module 1316 may retain instructions for performing functions associated with the modules 1302-1308. Although shown as being external to the memory 1316 , it is to be understood that the modules 1302 - 1308 may reside within the memory 1316 or within on-chip memory of the processor 1310 .
[0160] Reference Fig.14 , a computer-implemented method for generating communication content 1400 may include, at 1410, receiving, by at least one processor, neurophysiological data from one or more sensors coupled to at least one user consuming a segment of the communication content, wherein the segment is selected from a library of alternative segments. The communication content may be encoded for use with a mobile device, such as a portable flat screen device, a digital projector, or a wearable accessory, some or all of which may be used for alternative reality or augmented reality, in each case the mobile device is optionally coupled to an audio output function and optionally coupled to other output functions (e.g., motion, tactile, or olfactory). Playing the communication content may include, for example, saving the communication content in a cache or other memory of the mobile device and processing the content for output by at least one processor of the mobile device. The communication content may include entertainment, social interaction, and informational / educational content or other content.
[0161] Method 1400 may include, at 1420, computing, by at least one processor, for each of the segments, a measure of a neural response of at least one user based on neurophysiological data and a cognitive discrimination model for assessing the neurophysiological data. For example, the computing may include determining a content engagement capability (CEP) value for each of the segments. The neurophysiological data may include any one or more of the data described herein for arousal, evaluation, or other measures. For example, the neurophysiological data may include one or more of the following: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR). In one aspect, determining the CEP value may also include determining an arousal value based on the neurophysiological data, and comparing the average arousal of the stimulus based on the neurophysiological data with the expected average arousal. In one aspect, the measure of the neural response of at least one user for each of the segments includes an indication of at least one appreciation.
[0162] Method 1400 may include, at 1430, recording, by at least one processor, a measure of a neural response of at least one user for each of the segments in a computer memory associated with an identifier for each of the segments for use in at least one of: producing communication content that includes selected ones of the segments, or selecting participants for a multi-participant communication session.
[0163] Fig.15 Other operations 1500 are listed that may be performed as part of method 1400. The elements of operation 1500 may be performed in any operational order, and any one or any number of the elements may be omitted from method 1400.
[0164] Reference Fig.15 , method 1400 may include any one or more of the additional operations 1500 for generating communication content. Method 1400 may include, at 1510, generating communication content at least in part by selecting segments from a library based on an associated measure of a neural response of at least one user for each segment and combining the segments selected by the selection. In an aspect, the selection may also include assigning an error indicator to the associated measure of each segment based on a difference between the associated measure of the neural response of at least one user and a target value. For example, in an aspect, the selection may include minimizing the segments having the assigned error indicator.
[0165] Method 1400 may include, at 1520, performing collection, calculation, and recording for a plurality of different users that each consume the segment. In one aspect, the segment may consist essentially of language content. In another aspect, the segment may consist essentially of audio content.
[0166] In another aspect, method 1400 may include, at 1530, matching users of a multi-participant communication session at least in part by selecting users from a plurality of different users having complementary measures of neural responses to corresponding segments. For example, in one aspect, the multi-participant communication session may include one or more of a computer game or a social chat session.
[0167] The method 1400 may include, at 1540, determining an desirability value based on the neurophysiological data. The method 1400 may include, at 1550, normalizing the desirability value based on corresponding values collected for a specified set of baseline stimuli.
[0168] Fig.16 16 is a conceptual block diagram illustrating components of an apparatus or system 1600 for generating communication content. The apparatus or system 1600 may include additional or more detailed components for performing functional or process operations as described herein. For example, as described above, the processor 1610 and the memory 1616 may include an instance of a process for operating a CEP in real time. As depicted, the apparatus or system 1600 may include functional blocks that may represent functions implemented by a processor, software, or a combination thereof (e.g., firmware).
[0169] like Fig.16 As shown, the device or system 1600 may include an electronic component 1602 for receiving neurophysiological data from one or more sensors coupled to at least one user consuming a segment of communication content, wherein the segment is selected from a library of alternative segments. Component 1602 may be or may include a device for said receiving. The device may include a processor 1610 coupled to a memory 1616 and coupled to the output of at least one sensor (such as a neurophysiological sensor described elsewhere herein), the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as configuring a data port to receive sensor data from a known sensor, configuring a connection to a sensor, receiving digital data at the port, and interpreting the digital data as sensor data.
[0170] The apparatus 1600 may also include an electronic component 1604 for calculating, for each of the segments, a measure of the neural response of at least one user based on the neurophysiological data and the cognitive discrimination model for evaluating the neurophysiological data. The component 1604 may be or may include a device for calculating. The apparatus may include a processor 1610 coupled to a memory 1616, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, for example, such as combining Figure 8 as described.
[0171] The apparatus 1600 may also include an electronic component 1606 for recording the measure of the neural response of at least one user for each of the segments in a computer memory associated with an identifier of each of the segments for at least one of the following: making communication content including selected ones of the segments, or selecting participants for a multi-participant communication session. Component 1606 may be or include means for said recording. The apparatus may include a processor 1610 coupled to a memory 1616, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as encoding the measure of the neural response of at least one user and storing the encoded measure in a computer memory.
[0172] The device 1600 may optionally include a processor module 1610 having at least one processor. The processor 1610 may be in operative communication with the modules 1602-1606 via a bus 1613 or similar communication coupling. In an alternative, one or more modules may be instantiated as functional modules in a memory of the processor. The processor 1610 may initiate and schedule processes or functions performed by the electronic components 1602-1606.
[0173] In related aspects, the device 1600 may include a network interface module 1612 or an equivalent I / O port operable to communicate with system components via a computer network. The network interface module may be or may include, for example, an Ethernet port or a serial port (e.g., a universal serial bus (USB) port), a Wi-Fi interface, or a cellular phone interface. In further related aspects, the device 1600 may optionally include a module for storing information, such as a memory device 1616. The computer-readable medium or memory module 1616 may be operably coupled to other components of the device 1600 via a bus 1613, etc. The memory module 1616 may be adapted to store computer-readable instructions and data to affect the processes and behaviors of the modules 1602-1606 and their subcomponents or the processor 1610, the method 1400 and one or more additional operations 1500 disclosed herein, or any method performed by a media player as described herein. The memory module 1616 may retain instructions for performing functions associated with the modules 1602-1606. Although shown as being external to the memory 1616 , it is to be understood that the modules 1602 - 1606 may reside within the memory 1616 or within on-chip memory of the processor 1610 .
[0174] The device 1600 may include or may be connected to one or more sensors, which may be of any suitable type. Various examples of suitable biometric sensors are described above. In alternative embodiments, the processor 1610 may include a networked microprocessor from a device operating on a computer network. In addition, the device 1600 may be connected to an output device described herein via an I / O module 1612 or other output port.
[0175] Reference Fig.17 A computer-implemented method 1700 for controlling presentation of communication content may include, at 1710, receiving, by at least one processor, neurophysiological data from one or more sensors coupled to at least one user while the user is consuming segments of the content on a client device, wherein the content includes a segment chain. The content may be encoded for use with a mobile device, such as a portable flat screen device, a digital projector, or a wearable accessory, some or all of which may be used for alternative reality or augmented reality, in each case the mobile device is optionally coupled to an audio output function and optionally coupled to other output functions (e.g., motion, tactile, or olfactory). Playing the content may include, for example, saving the content in a cache or other memory of the mobile device and processing the content for output by at least one processor of the mobile device. The content may include communication, entertainment, social interaction, and informational / educational content or other content. The segments may include, for example, content selected from the group consisting of language content, audio content, and audio-video content.
[0176] Method 1700 may include, at 1720, calculating, by at least one processor, for each of the segments, a measure of the neural response of at least one user based on the neurophysiological data and a cognitive discrimination model for assessing the neurophysiological data. For example, the calculation may include determining a content engagement capability (CEP) value for each of the segments. The neurophysiological data may include any one or more of the data described herein for arousal, evaluation, or other measures. For example, the neurophysiological data may include one or more of the following: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR). In one aspect, determining the CEP value may also include determining an arousal value based on the neurophysiological data, and comparing the average arousal of the stimulus based on the neurophysiological data with the expected average arousal. In one aspect, the measure of the neural response of at least one user for each of the segments includes an indication of at least one appreciation.
[0177] The method 1700 may include, at 1730, selecting, by at least one processor, a subsequent segment in the chain based on a measure of a neural response of at least one user for the segment. In one aspect, the selecting further includes determining an error indicator for an associated measure of each segment based on a difference between an associated measure of a neural response of at least one user and a target value. In another aspect, the selecting includes minimizing the segment having an assigned error indicator.
[0178] The method may include, at 1740, outputting the subsequent segment to a client device.
[0179] Fig.18 Additional operations 1800 are listed that may be performed as part of method 1700. The elements of operation 1800 may be performed in any operational order, and any one or any number of the elements may be omitted from method 1700.
[0180] Reference Fig.18 , method 1700 may include any one or more of the additional operations 1500 for controlling the presentation of content. Method 1700 may include determining a rating value based on neurophysiological data at 1810. In one aspect, the neurophysiological data may be one or more of: electroencephalogram (EEG) data, facial electromyogram (fEMG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR).
[0181] Method 1700 may include, at 1820, normalizing the ratings values based on corresponding values collected for a set of specified baseline stimuli. In one aspect, the segment may consist essentially of language content. In another aspect, the segment may consist essentially of audio content.
[0182] Fig.19 1900 is a conceptual block diagram illustrating components of an apparatus or system 1900 for controlling the presentation of content. The apparatus or system 1900 may include additional or more detailed components for performing the functional or process operations as described herein. For example, the processor 1910 and the memory 1916 may include an instance of a process for operating a CEP in real time as described above. As depicted, the apparatus or system 1900 may include functional blocks capable of representing functions implemented by a processor, software, or a combination thereof (e.g., firmware).
[0183] like Fig.19 As shown, the device or system 1900 may include an electronic component 1902 for receiving neurophysiological data from one or more sensors coupled to at least one user while the user is consuming a segment of content on a client device, wherein the content includes a segment chain. Component 1902 may be or may include a device for said receiving. The device may include a processor 1910 coupled to a memory 1916 and to the output of at least one sensor (such as a neurophysiological sensor described elsewhere herein), the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as configuring a data port to receive sensor data from a known sensor, configuring a connection to the sensor, receiving digital data at the port, and interpreting the digital data as sensor data.
[0184] The apparatus 1900 may also include an electronic component 1904 for calculating, for each of the segments, a measure of the neural response of at least one user based on the neurophysiological data and the cognitive discrimination model for evaluating the neurophysiological data. The component 1904 may be or may include a device for calculating. The apparatus may include a processor 1910 coupled to a memory 1916, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, for example, such as combining Figure 8 as described.
[0185] The apparatus 1900 may also include an electronic component 1906 for selecting a subsequent segment in the chain based on a measure of the neural response of at least one user for the segment. The component 1906 may be or may include a means for said selecting. The apparatus may include a processor 1910 coupled to a memory 1916, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as selecting a subsequent segment in the chain having a CEP value different from (e.g., higher or lower than) the measure of the neural response of at least one user.
[0186] The apparatus 1900 may also include an electronic component 1908 for outputting the subsequent segment to a client device. Component 1908 may be or may include a device for said output. The apparatus may include a processor 1910 coupled to a memory 1916, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as sending the subsequent segment to a client device for presentation to a user, where the presentation may be done via visual, audio, text, video, or other forms perceptible or recognizable by the user.
[0187] The device 1900 may optionally include a processor module 1910 having at least one processor. The processor 1910 may be in operative communication with the modules 1902-1908 via a bus 1913 or similar communication coupling. In an alternative, one or more modules may be instantiated as functional modules in a memory of the processor. The processor 1910 may initiate and schedule processes or functions performed by the electronic components 1902-1908.
[0188] In a related aspect, the device 1900 may include a network interface module 1912 or an equivalent I / O port operable to communicate with system components via a computer network. The network interface module may be or may include, for example, an Ethernet port or a serial port (e.g., a universal serial bus (USB) port), a Wi-Fi interface, or a cellular phone interface. In a further related aspect, the device 1900 may optionally include a module for storing information, such as, for example, a memory device 1916. The computer-readable medium or memory module 1916 may be operably coupled to other components of the device 1900 via a bus 1913 or the like. The memory module 1916 may be adapted to store computer-readable instructions and data to affect the processes and behaviors of the modules 1902-1908 and their subcomponents or the processor 1910, the method 1700, and one or more additional operations 1800 disclosed herein, or any method performed by a client device as described herein. The memory module 1916 may retain instructions for performing functions associated with the modules 1902-1908. Although shown as being external to the memory 1916 , it is to be understood that the modules 1902 - 1908 may reside within the memory 1916 or within on-chip memory of the processor 1910 .
[0189] Device 1900 may include or may be connected to one or more sensors, which may be of any suitable type. Various examples of suitable sensors are described above. In alternative embodiments, processor 1910 may include a networked microprocessor from a device operating on a computer network. In addition, device 1900 may be connected to an output device as described herein via an I / O module 1912 or other output port.
[0190] Reference Fig. 20 , you can use the combination Figure 9-11The described systems and apparatus are used to perform a computer-implemented method 2000 for collecting neurophysiological data indicating a neural response of a user consuming communication content. The method 2000 may include, at 2010, detecting, by at least one processor of a mobile device, that a user of the mobile device is consuming communication content independently of the mobile device. The communication content may be encoded for use with a mobile device, such as a portable flat screen device, a digital projector, or a wearable accessory, some or all of which may be used for alternative reality or augmented reality, in each case the mobile device is optionally coupled to an audio output function and optionally coupled to other output functions (e.g., motion, tactile, or olfactory). In one embodiment, the mobile device includes a wrist-worn smart device. In other aspects, the mobile device may include one or more of other smart devices or wearable devices, such as headwear, finger-worn devices, goggles, etc. Playing the communication content may include, for example, saving the communication content in a cache or other memory of the mobile device and processing the content for output by at least one processor of the mobile device. The communication content may include entertainment, social interaction, and information / educational content or other content.
[0191] Method 2000 may include, at 2020, receiving neurophysiological data of the user via one or more sensors in a mobile device by at least one processor while the user is consuming communication content. For example, the reception may be performed via at least one sensor of the mobile device, the sensor including a microphone, a camera, an infrared sensor or a phased array sensor, or a galvanic skin response sensor. In one aspect, one or more sensors may be used to obtain neurophysiological data that can be used to determine the content engagement capability (CEP) value of the communication content. The neurophysiological data may include any one or more data described herein for arousal, evaluation, or other measures. For example, the neurophysiological data may include one or more of the following: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR).
[0192] The method 2000 may include, at 2030, providing at least one of the neurophysiological data or an indicator of a neural state of a user derived therefrom to at least one of: a second device outputting the communication content, a remote server, or a computer memory. In one aspect, the mobile device coordinates a handover of a playback session of the communication content from the second device to a third device. In one aspect, the mobile device is not in communication with the second device. In one aspect, at least one processor of the mobile device outputs a user-perceivable signal to attract at least one of gaze, touch, voice, or gesture at a time synchronized with the playback of the communication content.
[0193] Fig.21 Additional operations 2100 are listed that may be performed as part of method 2000. The elements of operation 2100 may be performed in any operational order, and any one or any number of the elements may be omitted from method 2000.
[0194] Reference Fig.21 , the method 2000 may include any one or more of the additional operations 2100 for collecting neurophysiological data indicative of a neural response of a user consuming the communication content. The method 2000 may include, at 2110, identifying, by at least one processor, an identifier for the communication content and providing the identifier to at least one of a remote server or a computer memory.
[0195] Method 2000 may include, at 2120, computing, by at least one processor, an indicator of a user's neural state based on neurophysiological data and a cognitive discrimination model. For example, the computing may include determining a content engagement capability (CEP) value for each of the segments. The neurophysiological data may include any one or more of the data described herein for arousal, evaluation, or other measures. For example, the neurophysiological data may include one or more of the following: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR). In one aspect, determining the CEP value may also include determining an arousal value based on the neurophysiological data, and comparing the average arousal of the stimulus based on the neurophysiological data with the expected average arousal. In one aspect, the measure of the neural response of at least one user for each of the segments includes an indication of at least one evaluation. In one aspect, the content may consist essentially of language content. In another aspect, the content may consist essentially of audio content.
[0196] In another aspect, method 2000 may include, prior to collecting, initiating a communication session with a second device that outputs the communication content, at 2130. For example, in one aspect, the communication session may include one or more of a computer game or a social chat session.
[0197] Fig. 22 2200 is a conceptual block diagram illustrating components of an apparatus or system 2200 for generating communication content. The apparatus or system 2200 may include additional or more detailed components for performing functional or process operations as described herein. For example, the processor 2210 and the memory 2216 may include an instance of a process for operating a CEP in real time as described above. As depicted, the apparatus or system 2200 may include functional blocks capable of representing functions implemented by a processor, software, or a combination thereof (e.g., firmware).
[0198] like Fig. 22 As shown, the device or system 2200 may include an electronic component 2202 for detecting that a user of a mobile device is consuming communication content independently of the mobile device. Component 2202 may be or may include a device for said detection. The device may include a processor 2210, which is coupled to a memory 2216 and coupled to the output of at least one sensor (such as a neurophysiological sensor described elsewhere herein), and the processor executes an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as configuring a data port to receive communication content data from a known device other than a mobile device, configuring a connection with the known device, receiving known device data at the port, and interpreting the known device data and the communication content data.
[0199] The apparatus 2200 may also include an electronic component 2204 for receiving neurophysiological data of a user via one or more sensors in a mobile device while the user is consuming communication content. Component 2204 may be or may include a device for said receiving. The apparatus may include a processor 2210 coupled to a memory 2216, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as configuring a data port to receive sensor data from a known sensor, configuring a connection to a sensor, receiving digital data at the port, and interpreting the digital data as sensor data.
[0200] The apparatus 2200 may also include an electronic component 2206 for providing at least one of the neurophysiological data or an indicator of the neural state of the user derived therefrom to at least one of the following: a second device that outputs the communication content, a remote server, or a computer memory. Component 2206 may be or may include means for said providing. The apparatus may include a processor 2210 coupled to a memory 2216, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as encoding a measure of the neural response of at least one user and storing the encoded measure in a computer memory, or as combined with Figure 8 as described.
[0201] The device 2200 may optionally include a processor module 2210 having at least one processor. The processor 2210 may be in operative communication with the modules 2202-2206 via a bus 2213 or similar communication coupling. In an alternative, one or more modules may be instantiated as functional modules in a memory of the processor. The processor 2210 may initiate and schedule processes or functions performed by the electronic components 2202-2206.
[0202] In related aspects, the device 2200 may include a network interface module 2212 or an equivalent I / O port operable to communicate with system components via a computer network. The network interface module may be or may include, for example, an Ethernet port or a serial port (e.g., a universal serial bus (USB) port), a Wi-Fi interface, or a cellular phone interface. In further related aspects, the device 2200 may optionally include a module for storing information, such as, for example, a memory device 2216. The computer-readable medium or memory module 2216 may be operably coupled to other components of the device 2200 via a bus 2213, etc. The memory module 2216 may be adapted to store computer-readable instructions and data to affect the processes and behaviors of the modules 2202-2206 and their subcomponents or the processor 2210, the method 2000, and one or more additional operations 2100 disclosed herein, or any method performed by the media player described herein. The memory module 2216 may retain instructions for performing functions associated with the modules 2202-2206. Although shown as being external to the memory 2216 , it will be appreciated that the modules 2202 - 2206 may reside within the memory 2216 or within on-chip memory of the processor 2210 .
[0203] The device 2200 may include or may be connected to one or more sensors, which may be of any suitable type. Various examples of suitable sensors are described above. In alternative embodiments, the processor 2210 may include a networked microprocessor from a device operating on a computer network. In addition, the device 2200 may be connected to an output device as described herein via an I / O module 2212 or other output port.
[0204] The methods and apparatus described herein may be adapted to improve person-to-person communications in virtual or real environments. Fig.23 A system 2300 is shown, which includes a first node 2300 with a first person 2302, which communicates with a second node 2320 with a second person 2312 via an electronic communication network 2350. The system 2300 can communicate using a CEP model, where CEP values are presented and measured together with the conversation. For example, two people 2302, 2312 can talk, and one or more participating clients 2306, 2316 present data 2340 about emotional impact next to a photo or video 2342, 2344 of each participant. The neurophysiological responses of the participants 2302, 2312 are sensed using corresponding biometric sensors 2308, 2318, and are described elsewhere herein. Each client 2306, 2316 can convert sensor signals from biometric sensors 2308, 2318 into biometric data, and send the biometric data to an analysis server 2330 via corresponding communication components 2307, 2317 and communication network 2350. The server 2330 may generate one or more measures of assessment, arousal, dominance, CEP, or any other suitable measure of neurophysiological response in real time or near real time, and provide the one or more measures to the clients 2306 , 2316 via the network 2350 .
[0205] Each client 2306, 2316 can output the measure as a graphical display 2340 or other useful format (e.g., audible output) via an output device 2304, 2314 (e.g., a display screen). The display 2340 or other output can report the neural state measure for a conversation sequence statement or statement group. For example, the display 2340 may include an indication of arousal 2346, 2350 or evaluation 2348, 2352. The system 2300 can provide an alert at any time when arousal increases rapidly, and report the evaluation associated with the increase. The meaning of the alert can then be identified by a person. The system 2300 is particularly useful for human-to-human communication between player actors within a virtual immersive experience, and may also find application in other contexts.
[0206] In view of the foregoing, and as an additional example, Figure 24-26Various aspects of one or more methods 2400 for rating the effectiveness of personal communications are shown. The method 2400 can be performed by an immersive mixed reality output device or a non-immersive flat screen device, a projector, or other output device including a programmable computer, by one or more computers in communication with the output device, or by a combination of an output device and one or more computers in communication with the output device.
[0207] Reference Fig.24 , a computer-implemented method for rating the effectiveness of personal communications may include, at 2410, monitoring, by at least one processor, digital data representing a user's communications with a recipient. The digital data may be encoded for an output device, such as a portable or non-portable flat screen device, a digital projector, or a wearable accessory for alternative or augmented reality, in each case coupled to an audio output function and optionally to other output functions (e.g., motion, tactile, or olfactory). Playing the digital data may include, for example, storing the digital data in a cache or other memory of the output device and processing the data for output by at least one processor of the output device.
[0208] Method 2400 may include, at 2420, receiving sensor data from at least one sensor positioned to sense an involuntary response of a recipient when receiving a communication. The sensor data may include any one or more of the data described herein for arousal, appreciation, or other metrics.
[0209] The method 2400 may include, at 2430, determining a content engagement capability (CEP) value for the communication based on the sensor data using an algorithm as described above. In an alternative, the method may determine a different measure of the neurophysiological response. The method may include, at 2440, recording the CEP value or other neural measure associated with the communication in a computer memory. In an alternative, the method may include indicating the CEP value or other neural measure to a user and / or recipient.
[0210] Figure 25-26 Additional operations 2500, 2600 are listed that may be performed as part of method 2400. Elements of operations 2500, 2600 may be performed in any operational order, and any one or any number of them may be omitted from method 2400.
[0211] Reference Fig.25, method 2400 may include any one or more of the additional operations 2500 for determining a CEP value. Method 2400 may include, at 2510, determining the CEP value at least in part by determining an arousal value based on sensor data, and comparing an average arousal of the stimulus based on the sensor data to an expected average arousal. The sensor data for arousal may include one or more of: electroencephalogram (EEG) data, galvanic skin response (GSR) data, facial electromyogram (fEMG) data, electrocardiogram (EKG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, video pulse detection (VPD) data, pupil dilation data, functional magnetic resonance imaging (fMRI) data, and functional near infrared data (fNIR). Method 2400 may include, at 2520, determining an expected average arousal based on further sensor data that measures a similar involuntary response of a recipient when engaging with a known audio-visual stimulus.
[0212] In another aspect, method 2400 may include, at 2530, playing a known audio-video stimulus comprising a known non-arousing stimulus and a known arousing stimulus. Method 2400 may include, at 2540, determining a CEP value at least in part by detecting one or more stimulus events based on sensor data exceeding a threshold for a period of time. Method 2400 may include, at 2550, computing one of a plurality of event capabilities for each of one or more users and for each of the stimulus events, and aggregating the event capabilities. Method 2400 may include, at 2560, assigning a weight to each of the event capabilities based on one or more source identifiers for the sensor data.
[0213] Reference Fig.26 , the method 2400 may include any one or more of the additional operations 2600 for determining a CEP value. The method 2400 may include, at 2610, determining an expected average arousal at least in part by detecting one or more stimulation events based on further sensor data exceeding a threshold for a period of time, and computing one of a plurality of expected capabilities of known audio-visual stimulations for one or more users and for each of the stimulation events. The method 2400 may include, at 2620, computing a CEP capability at least in part by computing a ratio of a sum of event capabilities to an aggregate of expected capabilities.
[0214] In related aspects, method 2400 may include, at 2630, determining a ratings value based on sensor data. The sensor data for the ratings may include one or more of: electroencephalogram (EEG) data, facial electromyogram (fEMG) data, video facial action unit (FAU) data, brain-computer interface (BMI) data, functional magnetic resonance imaging (fMRI) data, functional near infrared data (fNIR), and positron emission tomography (PET). Method 2400 may include, at 2640, normalizing the ratings value based on similar values collected for known audio-visual stimuli. Method 2400 may include, at 2650, determining a ratings error measure based on comparing the ratings value to a target rating for communication. Method 2400 may include, at 2660, setting a target ratings value based on input from a user.
[0215] Fig. 27 2700 is a conceptual block diagram illustrating components of an apparatus or system 2700 for rating the effectiveness of personal communications and related functions. The apparatus or system 2700 may include additional or more detailed components for performing the functional or process operations as described herein. For example, the processor 2710 and the memory 2716 may contain an example of a process for computing a CEP in real time as described above. As depicted, the apparatus or system 2700 may include functional blocks that can represent functions implemented by a processor, software, or a combination thereof (e.g., firmware).
[0216] like Fig. 27 As shown, the device or system 2700 may include an electronic component 2702 for monitoring digital data representing a communication between a user and a recipient by at least one processor. Component 2702 may be or may include a device for said monitoring. The device may include a processor 2710 coupled to a memory 2716 and coupled to the output of at least one biometric sensor, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include, for example, detecting the context of a communication, including detecting the communication for eliciting a target neurophysiological response, and creating an association between the communication and the target response.
[0217] The apparatus 2700 may also include an electronic component 2704 for receiving sensor data from at least one sensor, the sensor being positioned to sense an involuntary response of a recipient when receiving a communication. Component 2704 may be or may include a means for said receiving. The apparatus may include a processor 2710 coupled to a memory 2716, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as configuring a data port to receive sensor data from a known sensor, configuring a connection to a sensor, receiving digital data at the port, and interpreting the digital data as sensor data.
[0218] The apparatus 2700 may also include an electronic component 2706 for determining a content engagement capability (CEP) value for the communication based on the sensor data. Component 2706 may be or may include a means for such determination. The apparatus may include a processor 2710 coupled to a memory 2716, the processor executing an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as, for example, combining Figure 8 as described.
[0219] The apparatus 2700 may also include an electronic component 2708 for at least one of: recording a CEP value associated with the communication in a computer memory or indicating the CEP value to a user. Component 2708 may be or may include a means for said recording or indicating. The apparatus may include a processor 2710 coupled to a memory 2716 that executes an algorithm based on program instructions stored in the memory. Such an algorithm may include a series of more detailed operations, such as encoding the CEP value and storing the encoded value in a computer memory, or sending the encoded value to an output device for presentation to a user.
[0220] The device 2700 may optionally include a processor module 2710 having at least one processor. The processor 2710 may be in operative communication with the modules 2702-2708 via a bus 2713 or similar communication coupling. In an alternative, one or more modules may be instantiated as functional modules in a memory of the processor. The processor 2710 may initiate and schedule processes or functions performed by the electronic components 2702-2708.
[0221] In a related aspect, the device 2700 may include a network interface module 2712 or an equivalent I / O port operable to communicate with system components via a computer network. The network interface module may be or may include, for example, an Ethernet port or a serial port (e.g., a universal serial bus (USB) port), a Wi-Fi interface, or a cellular phone interface. In another related aspect, the device 2700 may optionally include a module for storing information, such as, for example, a memory device 2716. The computer-readable medium or memory module 2716 may be operably coupled to other components of the device 2700 via a bus 2713, etc. The memory module 2716 may be suitable for storing computer-readable instructions and data to affect the processes and behaviors of the modules 2702-2708 and their subcomponents or the processor 2710, the method 2400, and one or more additional operations 2500-2600 disclosed herein, or any method performed by the media player described herein. The memory module 2716 may retain instructions for performing functions associated with the modules 2702-2708. Although shown as being external to the memory 2716 , it will be appreciated that the modules 2702 - 2708 may reside within the memory 2716 or within on-chip memory of the processor 2710 .
[0222] The device 2700 may include or may be connected to one or more biometric sensors (not shown), which may be of any suitable type. Various examples of suitable biometric sensors are described above. In alternative embodiments, the processor 2710 may include a networked microprocessor from a device operating on a computer network. In addition, the device 2700 may be connected to an output device as described herein via an I / O module 2712 or other output port.
[0223] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps has been generally described above in terms of its functions. Whether this function is implemented as hardware or software depends on the application and design constraints imposed on the entire system. The technician can implement the described function in a varying manner for each application, but this implementation decision should not be interpreted as causing a departure from the scope of the present disclosure.
[0224] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component or module can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a server and a server can be a component or module. One or more components or modules can reside in a process and / or thread of execution, and a component or module can be located on one computer and / or distributed between two or more computers.
[0225] Various aspects will be presented in terms of systems that may include several components, modules, etc. It should be understood and appreciated that various systems may include additional components, modules, etc., and / or may not include all of the components, modules, etc. discussed in conjunction with the accompanying drawings. Combinations of these methods may also be used. Various aspects disclosed herein may be performed on an electrical device including a device utilizing touch screen display technology, a heads-up user interface, a wearable interface, and / or a mouse and keyboard type interface. Examples of such devices include VR output devices (e.g., VR headsets), AR output devices (e.g., AR headsets), computers (desktop and mobile), televisions, digital projectors, smart phones, personal digital assistants (PDAs), and other electronic devices, both wired and wireless.
[0226] In addition, the various illustrative logical blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD) or complex PLD (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0227] The operational aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, 18PROM memory, registers, hard disk, removable disk, CD-ROM, digital versatile disk (DVD), Blu-ray TMOr any other form of storage medium known in the art. The exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. In an alternative, the storage medium can be integrated with the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a client device or a server. In an alternative, the processor and the storage medium can reside in a client device or a server as discrete components.
[0228] In addition, standard programming and / or engineering techniques may be used to create software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed aspects to implement one or more versions as a method, apparatus, or article of manufacture. Non-transitory computer-readable media may include, but are not limited to, magnetic storage devices (such as hard disks, floppy disks, magnetic strips, or other formats), optical disks (such as compact disks (CDs), DVDs, Blu-ray discs, etc.), and other optical disks. TM or other formats), smart cards, and flash memory devices (e.g., cards, sticks, or other formats). Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the disclosed aspects.
[0229] The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the principles and novel features disclosed herein.
[0230] In view of the exemplary systems described above, methods that can be implemented according to the disclosed subject matter have been described with reference to several flow charts. Although the methodology is shown and described as a series of blocks for the purpose of simplifying the description, it should be understood and appreciated that the claimed subject matter is not limited by the order of the blocks, as some blocks can be performed in different orders and / or simultaneously with other blocks depicted and described herein. In addition, not all of the blocks shown are required to implement the methods described herein. In addition, it should be further appreciated that the methods disclosed herein can be stored on an article of manufacture to facilitate the transfer and transfer of such methods to a computer.
Claims
1. A method for generating a script for a drama production using an automatic algorithm, the method comprising: randomizing, by at least one processor, one or more cognitive parameters of a cognitive discrimination model for a participant in a role interaction, wherein the cognitive discrimination model maps biometric data to different cognitive parameters in a vector space, wherein the cognitive parameters include at least two of evaluation, arousal, confidence, and dominance; generating, by the at least one processor, the plurality of scripts at least in part by modeling the character interactions while varying the one or more cognitive parameters between different ones of a plurality of scripts for the theatrical production, wherein each of the scripts comprises a representation of the theatrical production; generating, by the at least one processor, for a plurality of segments in the plurality of scripts, an estimated effectiveness measure for the corresponding segment based on the cognitive discrimination model when a target user generates one or more target neurophysiological responses in accordance with a target storyline, wherein the target storyline includes a set of target neural values of the cognitive discrimination model, each target neural value being uniquely associated with a different interval of a continuous time series; selecting, by the at least one processor, one or more segments of the plurality of segments as one or more next segments, the selecting being based on the estimated effectiveness measures of the segments; compiling, by the at least one processor, a script based on the one or more next segments; as well as The at least one processor provides the target user with a compiled script or electronic content generated based on the generated script.
2. The method of claim 1 , wherein estimating the effectiveness measure comprises: Based on the cognitive identification model, the content participation capability (CEP) is calculated as the sum of the event capabilities of the segments of the script. v "Expected capabilities with comparable content types" x ", wherein the cognitive discrimination model includes the CEP, wherein the expectation capability represents the target user's expectation of the segment of the script, and the event capability represents the sum of the arousal or evaluation events of the target user when experiencing the segment of the script. 3 . The method of claim 2 , wherein calculating the CEP value further comprises determining an arousal value, and comparing the stimulus average arousal with an expected average arousal. The method according to claim 2 , wherein calculating the CEP value further comprises determining a rating value.
5. The method of claim 1, wherein estimating the effectiveness measure further comprises determining a neural error measure based on comparing the predicted neurophysiological responses to the one or more target storylines.
6. An apparatus for generating a script for a theatrical production using an automatic algorithm, comprising a processor coupled to a memory, the memory storing program instructions which, when executed by the processor, cause the apparatus to: randomizing one or more cognitive parameters of a cognitive discrimination model for a participant in a role interaction, wherein the cognitive discrimination model maps biometric data to different cognitive parameters in a vector space, wherein the cognitive parameters include at least two of evaluation, arousal, confidence, and dominance; generating the plurality of scripts at least in part by modeling the character interactions while varying the one or more cognitive parameters between different ones of a plurality of scripts for the theatrical production, wherein each of the scripts comprises a representation of the theatrical production; generating, for a plurality of segments in the plurality of scripts, an estimated effectiveness measure for the corresponding segment based on the cognitive discrimination model when a target user produces one or more target neurophysiological responses in accordance with a target storyline, wherein the target storyline includes a set of target neural values of the cognitive discrimination model, each target neural value being uniquely associated with a different interval of a continuous time series; selecting one or more segments of the plurality of segments as one or more next segments, the selecting being based on the estimated effectiveness measures of the segments; compiling a script based on the one or more next segments; as well as The compiled script or electronic content generated based on the generated script is provided to the target user.
Citation Information
Patent Citations
Entertainment system using bio-response
US20080161109A1
Method and apparatus for monitoring emotional compatibility in online dating
US20140221866A1
Cited By
Terminal device, processing system, and processing method for displaying content to specific audiences
TWI936602B
