Method, system and medium for measuring, calibrating and training psychological immersion - Patents.com

Abstract

Methods, systems, and media for measuring and training psychological immersion using music. A subject's state immersion measure is calculated by obtaining biometric data from the subject and processing the biometric data to calculate the state immersion measure. A personalized music library may be generated based on the music state immersion. Human subjects may be trained to develop increased trait immersion using music.

Description

[Technical field]

[0001] At least some example embodiments relate to systems for psychological measurement, and in particular, to systems for measuring and training psychological immersion using music. [Background technology]

[0002] The psychological state of "immersion" can be defined as a period of nearly "total" immersion that consumes all of one's attentional and representational resources (perceptual, enactive, imaginative, and ideational). [See non-patent document 1.] Moments of high immersion can result in strong nostalgic impressions in memory, known as "flashbulb memories", which are generated after exceptional events that last for a long period of time and retain many details that are easily forgotten in "normal" everyday memory. States of immersion are closely related to self-transcendence, awe, and spiritual experiences. [See non-patent document 2.] Preoccupation has also been considered a trait variable such that some individuals are more likely than others to become preoccupied with a stimulus. Trait preoccupation is closely related to the psychological constructs of imaginative engagement and openness to experience. [See non-patent literature 3.] Musical preoccupation is considered both a state and a trait variable.

[0003] Music state immersion is strongly correlated with music preference but is independent of the emotional valence of the music (i.e., whether the music conveys a positive or negative mood). [See non-patent literature 4.] State immersion can be measured through simple self-report questions [see Non-Patent Document 4, also Non-Patent Document 5] or via physiological responses [see Non-Patent Document 6, also Non-Patent Document 7]. Studies including neuroimaging show that state immersion results in unique patterns of brain activation similar to psychedelic states that differ from those expected under analytical observation of the same stimuli [see Non-Patent Document 6]. In particular, state immersion is accompanied by reduced activation in the default mode network (DMN; suggesting less self-referential thinking). Other features may include increased activation of the frontoparietal attention network (suggesting increased attention) and the human mirror neuron system (hMNS; suggesting empathy). Neuroelectrical correlates of these network changes have also been documented and are characterized by increased power in high-frequency oscillatory activity (theta band) and decreased power in high-frequency oscillatory activity (alpha, beta). These findings suggest that it may be possible to use hemodynamic or neuroelectrical measurements to assess changes in state immersion as well as trait immersion that may be realized through training.

[0004] Music trait absorption has been defined as the ability of music to induce and modulate emotions [Non-Patent Document 8] and as the possibility of using music to enter states of self-transcendence (loss of self-consciousness, "awe" emotions, mythical experiences, disorientation in time and space) [Non-Patent Document 9]. Music trait absorption can be measured by psychometric scales [see Non-Patent Document 8] and biomarkers.

[0005] It is therefore desirable to measure state absorption, for example, to enable the creation of personalized music libraries and / or to train human subjects to increase trait absorption using music. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Tellegen A, Atkinson G. "Openness to absorbing and self-altering experiences." Journal of Abnormal Psychology, 1974, Vol. 83, No. 3, p. 268-77. PMID: 4844914. [Non-Patent Document 2] Lifshitz, M., van Elk, M., and Luhrmann, T.M., 2019, "Absorption and spiritual experience: A review of evidence and potential mechanisms", Conscious and cognition, 73, 102760. [Non-Patent Document 3] Roche, S. M., and McConkey, K. M., 1990, "Absorption: Nature, assessment, and correlates", Journal of personality and social psychology, Vol. 59, No. 1, p. 91. [Non-Patent Document 4] Hall, SE, Schubert, E., and Wilson, SJ, 2016, "The role of trait and state absorption in the enjoyment of music." PloS one, vol. 11, no. 11, e0164029. [Non-Patent Document 5] Hall, E. B., Zweck, F., and Sinn, P., 2017, "Microsaccade-rate indicates absorption by music listening." Consciousness and cognition, vol. 55, p. 59-78. [Non-Patent Document 6] Van Elk, M., Arciniegas Gomez, M. A., van der Zwaag, W., Van Schie, HT., and Sauter, D., 2019, "The neural correlates of the awe experience: Reduced default mode network activity during feelings of awe." Human brain mapping, Vol. 40, No. 12, p. 3561-3574 [Non-Patent Document 7] Hu, X., Yu, J., Song, M., Yu, C., Wang, F., Sun, P., ...and Zhang, D., 2017, "EEG correlates of ten positive emotions." Frontiers in human neuroscience, vol. 11, pp. 26 [Non-Patent Document 8] Sandstrom, GM, and Russo, FA, 2013, "Absorption in music: Development of a scale to identify individuals with strong emotional responses to music." Psychology of Music, vol. 41, no. 2, pp. 216-228 [Non-Patent Document 9] Cardona, G., Ferreri, L., Lorenzo-Seva, U., Russo, FA, Rodriguez-Fornells, A. (under review), "The forgotten role of absorption in music reward", Annals of the New York Academy of Sciences Summary of the Invention

[0007] This disclosure describes example devices, methods, systems, and non-transitory media for measuring psychological immersion through biometric measurements, music preference discovery using measured immersion levels, and training of psychological immersion using music.

[0008] In accordance with some aspects, the present disclosure is directed to a method for computing a state immersion measure for a human subject, comprising acquiring biometric data from the human subject, the biometric data including one or more of functional near-infrared spectroscopy (fNIRS) data, electroencephalography (EEG) data, eye tracking data, and photoplethysmography (PPG) data, to compute the state immersion measure.

[0009] According to some aspects, the present disclosure is directed to a method for generating a personalized music library. A plurality of music segments are presented to a human subject. A state immersion measure of the subject is determined during the presentation of each music segment. A music trait immersion measure is calculated for each music segment based on the state immersion measure of the human subject during the presentation of each music segment. One or more music segments of the plurality of music segments are selected for inclusion in the personalized music library based on the music trait immersion measure of the one or more music segments for the human subject. Then, through machine learning methods, similar music segments are discovered based on feature similarity with the segments included in the personalized music library, allowing for a larger library of music that reflects the subject's preferences.

[0010] According to some aspects, the present disclosure is directed to a method for generating a personalized music library. A music library is obtained that includes a plurality of music segments that are likely to induce high trait immersion in a human subject. The human subject is instructed to perform one or more interactive exercises. One or more of the plurality of music segments are presented to the subject while the subject is performing the interactive exercise.

[0011] Embodiments will now be described by way of example with reference to the accompanying drawings, in which like reference numerals may be used to denote like features, and in which: [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an example system for emotional music recommendation according to an example embodiment described herein. [Diagram 2] 1 is a schematic diagram of the operation of a biometric classifier according to an example embodiment described herein. [Diagram 3]FIG. 2 is a frontal top view of a human subject's face showing skull positioning of an fNIRS sensor for biometric determination of state immersion according to an exemplary embodiment described herein. [Figure 4] 1 is a schematic diagram illustrating identification of a flashbulb memory event according to an exemplary embodiment described herein. [Diagram 5] FIG. 1 is a schematic diagram illustrating a machine learning classification module for state immersion according to an example embodiment described herein. [Figure 6] FIG. 1 is a schematic diagram illustrating discovery of music preferences through immersion calibration according to an exemplary embodiment described herein. [Figure 7] FIG. 1 is a schematic diagram illustrating music preference updates using a music recommendation system according to an exemplary embodiment described herein. [Figure 8] FIG. 1 is a schematic diagram illustrating generation of a personalized music library based on music trait immersion, according to an exemplary embodiment described herein. [Figure 9] FIG. 1 is a schematic diagram illustrating an immersion calibration process including a human subject and a user device according to an exemplary embodiment described herein. [Figure 10] FIG. 1 is a schematic diagram illustrating an immersion training process according to an exemplary embodiment described herein. [Figure 11] FIG. 2 is a schematic diagram of a content-based candidate musical segment generation process according to an example embodiment described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Exemplary embodiments are now described with respect to methods, systems, and non-transitory media for measuring psychological immersion through biometric measurements, using the measured immersion levels to discover music preferences, and training psychological immersion using music.

[0014] In some examples, systems and methods are provided for measuring state immersion in real time through biometric input, which provides a powerful means of objectively measuring candidate moments of awe and transcendence while a human subject is engaged in a multisensory environment, which may have applications in a variety of therapeutic practices, most notably in the context of psychedelic-assisted therapy.

[0015] In some examples, systems and methods are provided for calibrating or generating music libraries based on state immersion of a human subject using psychological and biometric measures. Also provided is a framework for training trait immersion to enhance a subject's ability to immerse themselves in music. Both frameworks leverage learning algorithms and state-of-the-art biosensing technology for implementation. Through the state immersion calibration method, an intuitive framework for determining music preferences without requiring manual input from a human subject is provided, which may have applications in the context of music entertainment services, music-based therapeutic practices, and reminiscence therapy (especially for non-communicative patients). In some examples, the trait immersion training module provides a means for inducing transcendent states through music, which may have applications in the context of various therapeutic practices, most notably psychedelic-assisted therapy and music therapy.

[0016] FIG. 1 shows an immersion system 100 comprising a processor system 102 for executing computer program instructions, a memory system 104 for storing executable instructions and data, and a communication system 106 for communicating data with other devices or components.

[0017] The immersion system 100 may be implemented on one or more computer systems. It may be embodied by a single computer, multiple computers, a virtual machine, a distributed computing or cloud computing platform, or any other platform capable of performing the method steps described herein. In some embodiments, the immersion system 100 may encompass one or more electronic devices (user devices 190) used by a human subject, while in other embodiments, the immersion system 100 communicates with such devices directly or indirectly (e.g., via a communication network 170) using a communication system 106.

[0018] The processor system 102 may be embodied as any processing resource capable of executing computer program instructions, such as one or more processors on a computer or computing platform. The memory system 104 may be embodied as any data storage resource, such as one or more disk drives, random access memory, or volatile or non-volatile memory on one or more computing platforms. The communication system 106 may be embodied as one or more communication links or interfaces, including wired or wireless communication interfaces, such as Ethernet, Wifi, or Bluetooth interfaces. In some embodiments, one or more of the user devices 190 may be implemented on the same platform as the immersion system 100. In such embodiments, the communication system 106 may comprise an internal communication bus or other intra-platform data transfer system.

[0019] The memory system 104 may have several types of computer programs stored thereon in the form of executable instructions. A set of executable instructions 110 for performing the method steps described herein may be stored in the memory system 104. There may also be one or more models (not shown), such as machine learning models, content filtering models, and / or collaborative filtering models, to perform the functions described herein, such as identifying audio segments intended to induce a high state of immersion in a listener, identifying music segments likely to match the preferences of a human subject, and / or personalizing an immersion training plan. These models may be deployed on the immersion system 100 after being trained as described further below.

[0020] The memory system 104 may have several types of data 180 stored thereon. The data 180 may include biometric data, such as fNIRS data 212, EEG data 222, video data 232, and / or other biometric data as described below, in raw and / or pre-processed formats. The data 180 may also include a music library 810 comprising a plurality of musical segments 186 and music feature data corresponding to each of the plurality of musical segments 186. The musical segments 186 may include digital audio data stored as individual audio clips or may be extracted from audio clips stored in the music library 810, such as epochs of fixed duration extracted from songs of variable duration. The music feature data is shown here as library MIR data 182. It may include Music Information Retrieval (MIR) metadata associated with each musical segment 186 indicating MIR features of the musical segment 186 with corresponding values. The music feature data may also include non-MIR data or metadata in some embodiments.

[0021] The user device 190 may be an electronic device operated by a human subject of the immersion system 100 (e.g., an end user or a third party user such as a therapist or caregiver), such as a computer or smartphone that communicates with the immersion system 100 via the communications network 170. The immersion system 100 may support multiple types of user devices 190. Some user devices 190 include a user interface component, such as a touch screen 194 for displaying visual data and receiving user input, and an audio output 192, such as a wired or wireless interface to speakers and / or headphones. Communication with the immersion system 100 is enabled via a communications system 196, which may communicate via the communications network 170.

[0022] 2-11 illustrate various functional subsystems or modules of the immersion system 100 and / or the user device 190. The various functional operations are performed by the immersion system 100 by using the processor system 102 to execute executable instructions 110 stored in the memory system 104.

[0023] FIG. 2 illustrates an exemplary biometric classifier 200 implemented by the immersion system 100 for making biometric measurements of state immersion of a human subject. A variety of biomarkers that can be collected outside of a laboratory reveal whole-body changes indicative of state immersion, allowing ongoing biometric assessment of immersion in the real world. Promising tools for such measurements include hemodynamic changes via functional near-infrared spectroscopy (fNIRS), neuroelectrical changes via electroencephalography (EEG), pupil dilation, and microsaccade velocity via video-based eye tracking, as well as heart rate via photoplethysmography (PPG). By combining these metrics and leveraging machine learning, tracking efficacy may be optimized for a given individual human subject.

[0024] The biometric classifier 200 implements a process for constructing one or more software-based classifiers for immersion and / or flashbulb memory potentials using various biometric devices. The software-based classifiers may include the EEG module 210, the fNIRS module 220, the eye tracking module 230, and / or one or more additional software-based classifiers for classifying or predicting state immersion based on biometric data. These software classifiers can be implemented locally (e.g., on the user device 910) or in a cloud-based computing environment (e.g., on the immersion system 100). In some embodiments, either or both of two options may be used for biometric classification of state immersion. The first is a formal algorithmic approach that cleans and converts the raw biometric data into a measure that maps directly to state immersion. This option may be best used with the high probability metrics listed below: EEG, fNIRS, and eye tracking. The second leverages machine learning processing to build a software classifier by correlating ground truth psychometric data with various biometric features, potentially allowing for a more flexible system and the use of lower probability kill measures. The machine learning option also allows for personalized classifiers that are capable of accounting for variance in individual physiology and trait immersion levels. In both options, a quadratic formula can be used to calculate moments of flashbulb memory potentials or "FBM potentials" that may indicate the occurrence of a flashbulb memory event (FBM event). In some instances, FBM potentials are indicated by high immersion levels sustained for periods of 20 seconds or longer. These epoch lengths and the definition of "high immersion" levels can be defined depending on the embodiment. Both options may present advantages in measuring immersion over existing approaches, such as preventing the need for state immersion surveys, thereby effectively allowing for time-series immersion data.

[0025] Because the amount of research that exists on biometric measurements of immersion is limited, the brain-based and elimination biometric measures described herein have been identified with reference to the literature on combined experiences such as "awe" mystical experiences and psychedelics. The biometric measures used by the biometric classifier 200 may include high potential measures that are highly correlated with high state immersion in the literature, and / or elimination measures that may show little direct relationship to high state immersion in the literature.

[0026] The fNIRS module 220 may be used to process raw fNIRS data 222 to calculate a state immersion measure 240 of a human subject. Using fNIRS, the fNIRS module 220 may track immersion through decreased activity in the default mode network (DMN) and increased activity in the frontoparietal attention network and human mirror neuron system (hMNS). This may be further quantified as the ratio of activity between the former and latter two networks (i.e., the frontoparietal attention network and the hMNS). Support for this correlation is found in [Non-Patent Document 6]. It is recognized that Non-Patent Document 6 suggests that the neural mechanisms associated with awe may also support other types of self-transcendent experiences, such as the peak moment in the psilocybin effect of inducing ego disintegration. [See Carhart-Harris, RL, Erritzoe, D., Williams, T., Stone, JM, Reed, LJ, Colasanti, A., ... and Nutt, DJ, 2012 "Neural correlates of the psychedelic state as determined by fMRI studies with psilocybin", Proceedings of the National Academy of Sciences, Vol. 109, No. 6, pp. 2138-2143].

[0027] The fNIRS module 220 first processes the fNIRS data 222 using a cleaning and feature extraction process 224 that, at a high level, operates as follows: First, channel pruning is performed to remove channels that are saturated or not receiving enough light. This can be done after data collection by using an automated function in a matlab toolbox called Homer3. Second, the intensity data can be converted to optical density (OD). Third, a low-pass or band-pass filter is applied. The low-pass cutoff may be set to 0.1 Hz. The purpose of this filter is to suppress physiological noise and high frequency instrument noise. Optionally, if short channels are present, the data is submitted to a general linear model (GLM) using a low-pass filter with a cutoff at 5 Hz. This may suppress high frequency instrument noise but preserve physiological noise (i.e., heart beat). Physiological noise may be retained during processing by the GLM to act as a quality assurance, with the presence of a heart beat indicating good optode-scalp coupling. A regression may then be performed to remove the short channel physiological noise.

[0028] The data is then divided into epochs, such as 10 second windows with a 5 second overlap between windows in the channel of interest (i.e., an fNIRS measure of activity in the medial prefrontal cortex of human subjects).

[0029] The feature extraction process 224 then detects motion and applies motion correction. At this stage, if the data does not contain short channels, some embodiments may use a method aimed at removing systemic physiological noise that overlaps with the bandwidth of interest, for example using a low pass filter to remove the heart rate signal. Epochs of fNIRS data 244 before and after motion correction and / or systemic physiological noise removal may be generated as output of the fNIRS module 220 in some examples.

[0030] The oxygenation calculation process 226 of the feature extraction process 224 then converts the optical density into oxygenation (HbO) and deoxygenation (HbR). An average is then calculated over an epoch to assess the oxygenation level at 10 second intervals. The oxygenation level may be used by the algorithmic classification and FBM calculation process 228 to calculate a state immersion measure 240, which is generated as an output of the fNIRS module 220. The algorithmic classification and FBM calculation process 228 may also identify FBM events, as described below with reference to FIG. 4.

[0031] FIG. 3 is a frontal top view of the face of a human subject 300 showing the skull location 304 of an fNIRS sensor for biometric determination of state immersion in some embodiments of the fNIRS module 220.

[0032] Open source software code supporting fNIRS analysis may be used in some embodiments of the fNIRS module 220, such as the HomER software system described by [Huppert, TJ, Diamond, SG, Franceschini, MA, and Boas, DA, 2009, "HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain", Applied Optics, Vol. 48, No. 10, D280-D298].

[0033] The EEG module 210 may be used to process the raw EEG data 212 to calculate a state immersion measure 240 of a human subject. Using EEG band power analysis, the EEG module 210 may track state immersion through an increase in theta wave power and a decrease in high frequency band power (alpha, beta waves), which may be further quantified as a ratio, for example, theta vs. (alpha + beta).

[0034] The relationship between EEG and state immersion is based at least in part on findings in the literature: increased theta band activity may be associated with calming of the DMN and positive emotions such as amusement, interest, and pleasure. This finding implies that anterior medial electrodes may be most relevant for collecting EEG data 212. [See Scheeringa, R., Bastiaansen, MC, Petersson, KM, Oostenveld, R., Norris, DG, and Hagoort, P., 2008, "Frontal theta EEG activity correlates negatively with the default mode network in resting state", International journal of psychophysiology, Vol. 67, No. 3, p. 242-251] See Van Elk M (2014) "An EEG study on the effects of induced spiritual experiences on somatosensory processing and sensory suppression", Journal for the Cognitive Science of Religion, Vol. 2, No. 2, p. 121. Also see [Non-Patent Document 7].

[0035] Decreased alpha and beta band activity is also observed under the influence of psychedelics and during emotional listening to song snippets. Desynchronization (power reduction) across all frequency bands above alpha, especially for frontal electrodes, may indicate high state absorption. [Muthukumaraswamy, S.D., Carhart-Harris, R.L., Moran, R.J., Brookes, M.J., Williams, T.M., Errtizoe, D., ...and Nutt, D.J., 2013. "Broadband cortical desynchronization underlies the human psychedelic state", Journal of Neuroscience, Vol. 33, No. 38, 15171-15183] See also [McGarry, LM, Pineda, JA, and Russo, FA, 2015, "The role of the extended MNS in emotional and nonemotional judgments of human song," Cognitive, Affective, & Behavioral Neuroscience, Vol. 15, No. 1, pp. 32-44].

[0036] The EEG module 210 first processes the EEG data 212 using a cleaning and feature extraction process 214 that, at a high level, operates as follows: First, a bandpass filter is applied with a highpass cutoff at 1 Hz and a lowpass cutoff at 55 Hz. Power line interference at 60 Hz is also suppressed, which the bandpass filter may not be able to filter out. Second, a baseline value is subtracted from each electrode, which may be calculated as an average over all electrode measurements or for the mastoid electrodes if available.

[0037] Third, the data is divided into epochs, such as 10 second windows with 5 seconds of overlap between windows. Fourth, bad data is removed, for example, by applying the Clean_rawdata function in EEGLab, which performs automatic bad data removal. EEG data epochs 242 may be provided as an output of the EEG module 210 before and after bad data removal.

[0038] The ratio calculation module 216 then calculates the ratio of theta wave power to (alpha wave power + beta wave power) for each epoch. For each epoch, the ratio calculation module 216 calculates the normalized power in each of the following frequency bands: theta (4-7 Hz), alpha (8-12 Hz), and beta (13-20 Hz). The normalization process may be to divide the power in the band of interest across the entire frequency range of interest (i.e., 4-20 Hz) rather than the entire available frequency range (i.e., 1-100 Hz). This calculation may help indicate the relative amount of power driven by each band of interest.

[0039] The algorithmic classification and FBM computation process 218 then calculates a state preoccupation measure 240, for example, as equal to the ratio of theta power to (alpha power + beta power) for each epoch.

[0040] In some embodiments, open source software code supporting EEG spectrotemporal analysis may be used, such as the EEGLAB software described in [Delorme, A., and Makeig, S., 2004, "EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis", Journal of neuroscience methods, Vol. 134, No. 1, pp. 9-21].

[0041] An eye tracking module 230 may be used to process raw video data 232 showing the faces of human subjects to calculate a state immersion measure 240. Using video-based eye tracking, the eye tracking module 230 may track state immersion through pupil dilation and microsaccade velocity.

[0042] Pupil dilation is a widely used measure of listening effort [see Winn, M. B., Edwards, J. R., and Litovsky, R. Y., 2015, "The impact of auditory spectral resolution on listening effort revealed by pupil dilation," Ear and hearing, vol. 36, no. 4, e153] and its effects on autonomic arousal and chills. Pupil dilation can be measured using a smartphone camera and video-based motion tracking tracking software [see Liao, H. I., Yoneya, M., Kashino, M., and Furukawa, S., 2018]. Awe studies including electromyography (EMG) data revealed facial patterns similar to freezing. Therefore, tracking gaze and facial feature movements may allow the computation of immersion states. [Chirico A., Cipresso P., Yaden DB, Biassoni F., Riva G., Gaggioli A., 2017, "Effectiveness of immersive videos in inducing awe", an experimental study. Sci. Rep., 7:1218. 10.1038 / s41598-017]. Another study revealed zygomaticus major activity consistent with smiling in subjects experiencing a transcendent state.[Clayton RB, Raney AA, Oliver MB, Neumann D., Janicke-Bowles SH, Dale KR, 2019, "Feeling transcendent? Measuring psychophysiological responses to self-transcendent media content", Media Psychol, 1, p1-26]. It may therefore be useful to analyze video data to detect semi-smiles with minimal head motion, i.e., to measure the degree of subject's facial motion in addition to zygomaticus major muscle activity.

[0043] The eye tracking module 230 first processes the video data 232 using a cleaning and feature extraction process 234, which, at a high level, operates as follows: First, the video data 232 is acquired. In some embodiments, the video data 232 consists of a sequence of video frames, each containing a full frontal image of the subject's face.

[0044] Second, the raw video data 232 is processed to identify one or more pupils of the subject in the video frames. In some embodiments, the video data 232 is subjected to PupilEXT, an open source analysis framework for pupil image extraction.

[0045] Third, the pupil video data is divided into epochs, such as 10 second windows with a 5 second overlap between windows. The video data epochs 246 may be generated as an output of the eye tracking module 230 before or after pupil extraction.

[0046] The pupil dilation and microsaccade computation process 236 processes the pupil data epochs to determine the degree of pupil dilation and the number of microsaccades in each epoch. Each microsaccade is determined using a thresholding method after [Engbert, R., and Mergenthaler, K., 2006, "Microsaccades are triggered by low retinal image slip," Proceedings of the National Academy of Sciences, Vol. 103, No. 18, p. 7192-7197]. The algorithm classification and FBM computation process 218 then calculates state immersion measures 240 corresponding to, for example, the degree of pupil dilation and the speed of the microsaccades.

[0047] In some embodiments, open source software code may be used to support pupil dilation and microsaccade analysis, such as the PupilEXT software described in [Zandi, B., Lode, M., Herzog, A., Sakas, G., and Khanh, T. Q., 2021, "PupilEXT: Flexible open-source platform for high-resolution pupillom-etry in vision research," Frontiers in neuroscience, 603. The software and documentation may be obtained from [https: / / www.frontiersin.org / articles / 10.3389 / fnins.2021.676220 / full] or [https: / / github.com / openPupil / Open-PupilEXT].

[0048] In some embodiments, head movement and / or zygomaticus major muscle activity may also be detected in the video data and used to calculate state immersion measure 240 as described above using a further video-based software classifier (not shown). First, to assess the degree of movement, the raw video data 232 may be subjected to optical flow analysis using, for example, a flow analyzer as described in [Barbosa, AV, Yehia, HC, and Vatikiotis-Bateson, E., 2008, "Linguistically valid movement behavior measured non-invasively", In AVSP, p173-177]. The open source flow analyzer software calculates pixel displacements between successive frames in the video. All pixel displacements are represented as vectors representing magnitude and 2D direction. These vectors are then summed into a single vector representing the magnitude of movement over time.

[0049] Second, the data is divided into epochs, such as 10 second windows with a 5 second overlap between windows. A state immersion measure 240 may be generated based on epochs that have small amounts of facial movement.

[0050] In some embodiments, smile detection may be used, as described above, using raw video data 232. Open source code to support video-based smile tracking is described in [Eyben, F., Weninger, F., Gross, F., and Schuller, B., "Recent developments in opensmile, the munich open-source multimedia feature extractor," In Proceedings of the 21st ACM international conference on Multimedia, p835-838, October 2013].

[0051] As mentioned above, in some embodiments, additional peripheral biometric or physiological measures may be used to calculate the state immersion measure 240. It is recognized that each physiological measure may yield multiple features, for example, photoplethysmography (PPG) data may be processed to generate respiration, heart rate, and heart rate variability data. Heart rate variability data may be further processed to generate repetition rate data (through autocorrelation) and / or high frequency and / or low frequency signal components. Peripheral biometric measures that may be correlated with state immersion are discussed in the NIH publication [https: / / www.ncbi.nlm.nih.gov / pmc / articles / PMC7701337 / #B4].

[0052] There are several physiological processes that correlate with experiences of awe, transcendence and immersion. Unlike brain measurements, these physiological processes are best understood as the results rather than the causes of immersion. Galvanic skin response (GSR) data may be processed to generate a state immersion scale 240 based on findings suggested by [Clayton RB, Raney AA, Oliver MB, Neumann D., Janicke-Bowles SH, Dale KR, 2019, "Feeling transcendent? Measuring psychophysiological responses to self-transcendent media content", Media Psychol, 1, p1-26, 10.1080 / 15213269.2019.1700135]. See also [Shiota MN, Campos B., Oveis C., Hertenstein MJ, Simon-Thomas E., Keltner D., 2017, "Beyond happiness: building a science of discrete positive emotions", Am. Psychol., Vol. 72, No. 617, 10.1037 / a0040456].

[0053] Respiration rate can be extracted from the PPG data. A slower respiration rate may indicate higher immersion. [Ahani A., Wahbeh H., Nezamfar H., Miller M., Erdogmus D., Oken B., 2014, "Quantitative change of EEG and respiration signals during mindfulness meditation", J. Neuroeng. Rehabil, Vol. 11, p1-11, 10.1186 / 1743-0003-11-87] See also [Wielgosz J., Schuyler BS, Lutz A., Davidson RJ, 2016, "Long-term mindfulness training is associated with reliable differences in resting respiration rate", Sci. Rep., Vol. 6, p1-6.10.1038 / srep27533].

[0054] Heart rate is another measure of autonomic arousal that can be obtained using photoplethysmography embedded in wearables or directly via a smartphone camera. A drop in heart rate can indicate the onset of a period of paying attention to something and therefore high absorption. [See Kitson, A., Chirico A., Gaggioli A., and Riecke, B. E., 2020, "A review on re-search and evaluation methods for investigating self-transcendence," Frontiers in psychology, 11.] Heart rate variability (HRV) has also been used as a measure of transcendence (see, for example, Clayton et al., 2019). However, HRV may be less useful than other measures due to individual differences and the requirement of a long period for baseline measurements. [See Clayton RB, Raney AA, Oliver MB, Neumann D., Janicke-Bowles SH, Dale KR, 2019, "Feeling transcendent? Measuring psychophysiological responses to self-transcendent media content", Media Psychol, 1, p1-26, 10.1080 / 15213269.2019.1700135] In various embodiments, any combination of one or more of the above biometric measures may be used to generate state immersion measure 240. In some examples, data epochs may be evaluated for immersion, and then the highest immersion epochs may be identified as high immersion epochs for purposes of other processing performed by immersion system 100. For example, epochs of high immersion (e.g., above the 90th percentile for a given session) may be used to identify flashbulb memory events, as described further below with reference to FIG.

[0055] FIG. 4 is a schematic diagram illustrating identification of flashbulb memory (FBM) events based on state immersion measure 240. In some embodiments, FBM events are identified based on whether state immersion measure 240 is in the highest scale range (e.g., above an immersion threshold, such as a threshold defined by the 90th percentile of epochs in a session) for an epoch of 20 seconds or more (e.g., two consecutive 10 second windows). Identifying an epoch as indicative of an FBM event is indicated in FIG. 4 as "FBM event=1," and an epoch that is not identified as an FBM event is indicated as "FBM event=0." Thus, for example, epoch 1 402 and epoch 3 406 in FIG. 4 are not FBM epochs, but epoch 2 404 is an FBM event based on the per epoch state immersion measure 240 generated by biometric classifier 200.

[0056] 5 is a schematic diagram illustrating a machine learning classification module 500 for state immersion. The biometric classifier 200 may be used to process biometric data (e.g., 212, 222, 232) from one or more biometric devices 506 (such as video cameras, EEG electrode arrays, and fNIRS sensors) while stimuli are presented (504) to a human subject during data collection 502. Biometric data epochs (e.g., 242, 244, 246) may be output as time series data along with a state immersion measure 240.

[0057] State immersion scale data is also collected in response to a prompt (e.g., a prompt presented every epoch 508). The state immersion scale data 510 may include self-reported state immersion information reported by the subject and / or state immersion information observed by an observer of the subject (e.g., a therapist). The state immersion measure 240 may be generated or validated, in some embodiments, at least in part, based on the state immersion scale data 510.

[0058] The biometric data epochs (e.g., 242, 244, 246) and state immersion measures 240 may be stored in a user data database 520 in association with a user ID 512 that identifies the subject. At block 522, a clustering process (e.g., using a filtering model or a trained machine learning model) may be applied to the user data database 520 to generate clusters of biometric data epochs (e.g., 242, 244, 246) and state immersion measures 240 for users having similar relationships between biometric data and state immersion measures 240. At 524, the user data of the current subject's cluster is separated. From this separated set of user data, at 526, biometric features that are most strongly correlated with state immersion are identified. At 528, these identified features are used as ground truth information to train a classification model, shown as an immersion classifier 530, which processes the subject's biometric data to generate a predicted state immersion measure 240.

[0059] In some embodiments, the immersion classifier 530 is not trained with reference to a particular user, but instead is a non-personalized immersion classifier 530. In such embodiments, steps 522 and 524 may be omitted and the immersion classifier 530 may be trained using all user data instead of user data from a particular cluster. If data is collected from a diverse population set, it is possible to train a general non-personalized classifier, thereby eliminating the need for a data collection process for all users.

[0060] Figure 6 is a schematic diagram showing a process for discovering music preferences through immersion calibration 600. Figure 8 is a schematic diagram showing the generation of a personalized music library 820 based on music state immersion. Figures 6 and 8 show the operation of two common sets of modules and will be described together.

[0061] The process 600 of FIG. 6 starts with an optional stage (starting with standard start 604) in which either the listener or a caregiver (e.g., an observer such as a therapist) narrows down a large catalog of music based on hypothetical feature preferences encoded as personal profile data 602 (e.g., preferred genres, decade of birth, significant geographic locations, significant lyric themes). This optional stage occurs when the immersion system 100 provides this type of onboarding and if the music library providing the music segments has been tagged with metadata indicating these attributes. This optional stage corresponds to the survey process 804 of FIG. 8. If this optional stage continues, a selection of music that matches the subject's preferences is compiled for the next stage in the process; if not, a random selection of test samples is selected with high variance of MIR feature characteristics, providing a wide range of musical aesthetics to best validate the breadth of the listener's musical tastes. Once these selections have been made (shown in FIG. 8 as selection 806), the immersion calibration process 900 can begin. When the standard start 604 process is available, the personal profile data 602 is stored in a database of personal data 619. This stored data is correlated with the data contained in the vectors stored in the embedded preference vector database 616. When training the models used for the collaborative filtering process 606, the personal profile data 602 stored in the personal profile database 619 is used along with the decoded vectors output from the decoder network 218.

[0062] The immersion calibration process 900 is designed to assess the listener's musical preferences through a listening test measured by traditional state immersion scales and / or classified biometric scales (as described above with reference to Figures 2-5). After the immersion calibration process 900, the best (i.e., highest music trait immersion) selection of music segments identified by the immersion calibration process 900 is used to create a matrix 610 of immersion levels and content features (e.g., state immersion scales 240 by MIR features of music segments) that represent the subject's immersion preferences. This matrix 610 is then used in a content-based candidate generation process to generate a personalized music library 820 for the subject, calibrated based on immersion levels, as shown in Figure 8. It is also possible to re-evaluate the samples and continually update the personalized music library 820 to reflect changes in the listener's state immersion behavior over time. For example, if it is possible to utilize this test at the start of each listening session, this calibration process 900 can be performed periodically to best capture the subject's changing preferences. Updating the subject's preferences can be accomplished by processing the matrix 610 with an encoder network 612 configured to generate a user preference vector 614 that encodes the subject's music preferences based on the contents of the matrix 610. The user preference vector 614 may be stored in an embedded preference vector database 616 along with user preference vectors 614 for other subjects. A decoder network 618 can be used to decode the user preference vector 614 stored in the embedded preference vector database 616 and provide collaborative filtering input to the collaborative filtering process 606.

[0063] In some examples, the matrix 610 used to generate the user preference vector 614 is generated using all music segments and corresponding immersion levels, not just the best choices. By including all music segments and corresponding immersion levels in the matrix 610, a more comprehensive and accurate preference vector 614 may be generated. However, when generating a personalized music library 820, some embodiments may generate the matrix 610 using only the best choices.

[0064] The best choices resulting from the immersion calibration process 900, selected to form the personalized music library 820, can also be used for the immersion training process 1000, as shown in FIG. 8 and described in further detail with reference to FIG. 10. The immersion training process 1000 leverages interactive activities and machine learning optimization to increase the level of the trait immersion in the subject. This immersion training process 1000 can be performed multiple times to best increase these trait levels over time. This training is performed before and during listening sessions for enhanced listening results.

[0065] FIG. 8 also illustrates how the results of the immersion calibration process 900 are used to supplement the personalized music library 820 with music segments other than those that directly induce high state immersion in the subject. During this process, the system utilizes objective and / or subjective data capture to evaluate the listener's music preference / state immersion scale 240 for various music segments played in succession. This data is captured, the selections with the highest immersion levels are isolated, and their MIR features (encoded into a matrix 610 by the content feature matrix generation process 808) are analyzed or accessed via the database 810 (as the segments may be analyzed prior to the testing process). A content-based filtering process 1100 using the MIR features (e.g., using cosine similarity, Euclidean distance, or unsupervised clustering machine learning) is then utilized to find other tracks (denoted as the calibrated library 814) such as segments with high immersion levels, and the personalized music library 820 is further populated accordingly.

[0066] The personal profile data collected in the research process 804 may also be used to reduce the size of the library available for the content-based filtering process 1100 and improve the efficiency of that process.

[0067] State music absorption has been empirically determined to be a strong indicator of music preference (see non-patent literature 2). Importantly, this correlation between absorption and preference is independent of the emotion conveyed by the music (see non-patent literature 2). This may explain why music that conveys negative emotions can be deeply immersive and enjoyable. It has been suggested that enjoyment may result from a dissociation between activated negative emotions and associated unpleasant physical effects (see Non-Patent Document 6), which may be supported by the tendency of preferred music to activate reward circuits (see Non-Patent Document 7 and [Salimpoor, VN, van den Bosch, I., Kovacevic, N., McIntosh, AR, Dagher, A., and Zatorre, RJ, 2013, "Interactions between the nucleus accumbens and auditory cortices predict music reward value", Science, Vol. 340, No. 6129, p. 216-219]). Inducing negative emotions without unpleasant physical effects while activating reward circuits may increase the likelihood of reconstructing negative autobiographical memories that may arise in the context of psychedelic-assisted therapy (see non-patent literature 9).

[0068] FIG. 7 is a schematic diagram illustrating a music preference update process 700 using a music recommendation system. In some embodiments, the subject's music preference data encoded as a user preference vector 614 may be used to inform other music recommendation systems, which in turn may inform or update the subject. For example, a conventional or standard music recommendation system 702, such as a system that generates a preference profile based on a listener's selections, may use the user preference vector 614 as a baseline user preference profile for the subject. Adjustments to the user preference profile made by the standard music recommendation system 702 may be propagated to the user preference vector 614 as updates. In some examples, an emotional music recommendation system 704 may be similarly used to receive the user preference vector 614 as a baseline preference profile and update it during operation. The emotional music recommendation system 704 may include a system that evaluates the user's musical preferences based on emotions or affective states induced by different musical stimuli. In some examples, the emotional music recommendation system 704 may employ excitement or anxiety as an exit function to stop presenting music that is considered upsetting to the listener. However, in some embodiments, the state immersion measure 240 calculated by the immersion system 100 may be used as a feedback mechanism used to assess anxiety or arousal.

[0069] FIG. 9 is a schematic diagram illustrating an immersion calibration process 900, comprising a human subject 920 and a user device 910. Conventional music preference surveys rely either on lengthy onboarding processes or on implicit data such as song "skip" or playback control (which do not necessarily directly indicate the listener's preferences). In contrast, the process illustrated in FIG. 9 provides direct insight into the subject's 920 music preferences, allowing for the curation of music that has the potential to induce states of enjoyment and transcendence, providing value for therapeutic processes (e.g., psychedelic-assisted therapy) and entertainment use cases (e.g., music streaming platforms) that rely on these states.

[0070] A musical segment (shown as "Clip 1" 912) is presented to the subject 920 via a user device 910 with an audio output 192, such as headphones or speakers. The user is allowed to select a self-reported level of psychological immersion (e.g., using a slider 914) using a touch screen 194. In some embodiments, a standard immersion scale may be used. In FIG. 9, the slider 914 moves between a maximum immersion position of "(7) Able to be fully immersed without distractions" 916 and a minimum immersion position of "(1) Always distracted by irrelevant impressions or events" 918.

[0071] In some embodiments, this calibration process 900 allows for passive calibration of music preferences (via biometrics), which provides a powerful tool for reminiscence therapy processes when patients are incommunicative and unable to self-assess their music preferences through traditional survey and self-assessment modalities. Given that preference-based music curation is a key component for reminiscence therapy, this process can provide significant advantages for products that leverage music for this form of therapy. The above methods for measuring naturally occurring immersion markers using biometrics can provide a means of objectively determining the music associated with a patient's deepest memories, preventing the need to rely on used accounts. This technology allows for a dramatic leap in the standard of care within reminiscence therapy practices for people living with dementia and brain injury.

[0072] FIG. 10 is a schematic diagram showing an immersion training process 1000 according to an exemplary embodiment described herein. High levels of musical trait immersion are generally seen as a personality trait that develops naturally through biological predisposition, life experiences, musical training, etc. Individuals who live with high levels of musical trait immersion find that they experience a connection with music that has a potentially transformative impact on their lives. Particularly in therapeutic use cases (and even recreational use cases), trait immersion or a person's "openness to transformative music experiences" can have significant implications for treatment outcomes. Lower levels of trait immersion in some individuals may minimize the effectiveness of music-based treatment modalities (see Non-Patent Document 8).

[0073] 10 shows a process 1000 for training an individual to achieve higher levels on the trait immersion scale, potentially impacting the subject's ability to have an impactful listening experience and improved health outcomes following the session. Building on an existing music trait immersion inventory, process 1000 provides a two-way audio experience to help the subject increase their trait immersion while leveraging machine learning to optimize the training process for both the individual and the network of listeners.

[0074] Using the personalized music library 820 resulting from the immersion calibration process 900 and other processes of Figures 6-9, the training process 1000 selects the best performing tracks in the library (those with the highest immersion levels for the listener) and various interactive exercises (see below) are completed with the intent of improving trait-based immersion quality in the listener. These interactive listening activities can feature both complete multi-track audio or isolated music stems. Although a personalized music library 820 (resulting from the calibration process 600) gives the body of content a higher chance of success, it is possible to run the training process without the calibration process (e.g., by asking subjects to complete a survey in step 1002 such as STOMP described in [Rentfrow, PJ, & Gosling, SD (2003). The do re mi's of everyday life: the structure and personality correlates of music preferences. Journal of personality and social psychology, Vol. 84, No. 6, p. 1236]) and run trait immersion training on the queried music based on the results of those surveys.

[0075] The immersion training process 1000 is complemented by a personalized machine learning loop that not only learns which activities work best for each individual, but also learns at a network-wide level using models that can receive demographic and other data to best predict the best immersion training activities for new and returning users. The loop for this system is as follows:

[0076] First, music segment selections are identified using the calibration process 600 or through an existing music preference survey process 1002. The context / state is provided to a context band-it model 1014 and the user performs a state-based immersion assessment 1006 (via self-assessed state immersion questions or biometric classifier 200).

[0077] The context band-it model 1014 predicts the training module (i.e., interactive exercise 1016) with the most likely reward (as calculated by reward calculation 1010) that the subject will complete the selected exercise 1008. Thus, for example, the context band-it model 1014 may predict that movement meditation 1022 will result in the highest reward, and then movement meditation 1022 is selected as the selected exercise 1008.

[0078] The user performs another state-based immersion assessment 1012 (or alternatively is assessed via the biometric classifier 200) and a reward value is calculated 1010 based on the success of the selected exercise 1008. This reward is fed to the context bandit models 1014 (one model per user) which are trained accordingly.

[0079] The interactive exercises 1016 used in the musical trait immersion training 1000 were inspired by the Immersion in Music Scale (AIMS) by Friedrich Schmidt, MD, et al. (2013). Items from this scale that received an item-total correlation of 0.5 or more were selected and grouped thematically into six categories. These categories were then mapped to the closest corresponding mode of meditation (e.g., concentration meditation 1020 or affinity meditation 1026). For each category, an activity has been developed that is inspired by a corresponding question that allows for a specific type of music-based meditation. A machine learning model is then used to cycle between and learn the most effective exercises for the user given the contextual input and the rewards calculated in 1010 based on the trait-based immersion rating 1012 at the end of each exercise. This is designed to optimize the training process 1000 for the user and to quickly and effectively improve its trait-based immersion level. This machine learning loop can be used not only to optimize this training process 1000, but also for any similar cognitive training method with optional multiple modules.

[0080] The interactive nature of the immersion system 100 also allows for novel forms of meditation. When practiced regularly, these activities are expected to enhance the user's capacity for music-based immersion. The six types of music-based mediation are: concentration meditation 1020, movement meditation 1022, transcendental meditation 1024, affinity meditation 1026, progressive relaxation meditation 1028, and visualization meditation 1030.

[0081] The first meditation, Concentration Meditation 1020, is developed in more detail below to provide an illustrative example, however, the same general process is followed for each meditation, which begins with the selection of a musical segment that is likely to promote state immersion for a given subject.

[0082] The concentration meditation 1020 begins with the selection of an immersion track. A learning layer allows the system to determine which exercise works best for a given subject. Following the initial probe, a new candidate music segment is played. The subject is asked to judge it on a 7-point scale rating the music state immersion (from non-patent document 4), as described above with reference to FIG. 9. This music state immersion scale is administered before and after the music mediation. This sequence (rating-meditation-rating) is repeated three or more times. The average pre-post change in state immersion is used to update the system regarding the effectiveness of music-based meditation for a given user.

[0083] The concentration meditation 1020 session begins by asking the subject to block out all distractions by imagining that "nothing exists in the world except for themselves and the music that follows." They are then asked to focus on their breath and synchronize it with the music that follows. The music begins with a percussion layer (e.g., a metronome bass drum with a strong beat at the beginning of each four-beat cycle) that reinforces the beat and metric structure of the track. The beat is further reinforced through the use of a dynamic metronome presented on the user device 910 (e.g., a smartphone or tablet).

[0084] Instructions to the subject (presented through sound and text) support synchronization with each metric cycle (e.g., 4 beats). A new layer of the track (e.g., melodic harmony) is added following the completion of the first 8 bars of music. The starter percussion layer is faded out after 16 bars, while the subject is given instructions to maintain the breathing rate with the support of the visual metronome. Once all layers of the track have been added (e.g., after 32 bars), the visual metronome fades out and the subject is asked to maintain that breathing rate through the completion of the track without the assistance of the visual metronome.

[0085] The same basic process of using a learning layer to evaluate the effectiveness of a mediation for a given subject is used for five other forms of musical meditation. Each of these approaches is briefly outlined below.

[0086] In the movement meditation 1022, subjects are instructed to move their hands as if they were playing music. They are asked to tap on the phone at the rate of the beat, allowing each hit to coincide with the beat onset (which requires beat anticipation). Subjects are given a score on their level of sensorimotor synchrony (determined by mean absolute asynchrony).

[0087] In Transcendental Meditation 1024, the subject is asked to imagine that he or she is on a small boat in the ocean being carried by swells, and that the entire ocean appears to be rising, but the depths of the ocean are still. The subject is asked to choose a mantra and generate it in beat (e.g., "hmmm"). Possible instructions while generating the mantra include letting the music calm the mind, and letting the music achieve deep stillness.

[0088] In the Loving-kindness meditation 1026, subjects are asked to feel connected to others through music, allowing the subject to receive and send back love from others. The instructions ask the subject to identify with the music and imagine it as another person, allowing them to feel connected to the other person.

[0089] In progressive relaxation meditation 1028, the subject is asked to listen to music and slowly tighten and relax different parts of the body (instructions guide this process). The subject is then asked to allow the music to guide awareness of the different body parts, gradually relaxing each area.

[0090] In the visualization meditation 1030, the subject is asked to imagine that he is hearing the music so vividly that he is listening to it "live." The subject is asked to imagine the colors and images that the music is evoking. The sense of immersion is enhanced by using audio spatialization methods.

[0091] FIG. 11 is a schematic diagram illustrating the content-based music segment candidate generation process 1100 described above with reference to FIG. 8. During the content-based filtering process, a weighted feature-based recommendation system 1102 is used to create a personalized library using the matrix 610 of immersion ratings and MIR features. The exemplary system augments the matrix 610 with other available tracks with their corresponding MIR features (stored in the database 810), and a supervised classification algorithm (e.g., k-nearest neighbors) is used to score the augmented set. Using this system, the resulting personalized music library 820 is a selection of music segments in the available library 810 that have similar aesthetics to the best segment identified during the absorption calibration process. This process provides a scalable means for creating a personalized music library 820 from the matrix 610 of immersion data correlated with music features.

[0092] Although the present disclosure may be described, at least in part, in terms of methods and devices, those skilled in the art will appreciate that the present disclosure also covers various components for performing at least some of the aspects and features of the described methods by hardware components, software, or any combination of the two. Thus, the technical solutions of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer or processor readable medium, including, for example, a DVD, a CD-ROM, a USB flash disk, a removable hard disk, or other storage medium. The software product comprises instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute the examples of the methods or systems disclosed herein.

[0093] Those skilled in the art will also appreciate that the output of the above methods and devices, such as the personalized music library 820, may be stored as music data (such as audio files or playlist data) on a storage medium, such as a non-volatile or non-transitory computer or processor readable medium, including a DVD, CD-ROM, USB flash disk, removable hard disk, or other storage medium. The music may also be stored on other digital or analog storage media suitable for use in audio applications or audio playback or broadcast devices, such as cassette tapes, vinyl records, or any other storage medium for digital or analog music data.

[0094] In the described methods or block diagrams, the boxes may represent events, steps, functions, processes, modules, messages, and / or state-based operations, etc. Although some of the above examples are described as occurring in a particular order, it will be understood by those skilled in the art that some of the steps or processes may be performed in a different order, provided that the results of a changed order of any given step do not prevent or impair the occurrence of a subsequent step. Furthermore, some of the messages or steps described above may be removed or combined in other embodiments, and some of the messages or steps described above may be separated into several sub-messages or sub-steps in other embodiments. Furthermore, some or all of the steps may be repeated as necessary. Elements described as methods or steps apply equally to systems or subcomponents, and vice versa. References to words such as "send" or "receive" may be interchanged depending on the perspective of a particular device.

[0095] The above embodiments are illustrative and not limiting: exemplary embodiments described as methods apply equally to systems and vice versa.

[0096] Modifications may be made to some exemplary embodiments, which may include any combinations and subcombinations of the above. The various embodiments presented above are merely examples and are not meant to limit the scope of the present disclosure in any way. Variations of the innovations described herein will be apparent to those skilled in the art, and such variations are within the intended scope of the present disclosure. In particular, features from one or more of the above embodiments may be selected to create alternative embodiments consisting of subcombinations of features that may not be explicitly described above. In addition, features from one or more of the above embodiments may be selected and combined to create alternative embodiments consisting of combinations of features that may not be explicitly described above. Features suitable for such combinations and subcombinations will be readily apparent to those skilled in the art upon reviewing the present disclosure as a whole. The subject matter described herein is intended to cover and encompass all suitable modifications in technology.

Claims

1. A method for calculating a state immersion scale for human subjects, A step of obtaining biometric data from the aforementioned human subject, wherein the biometric data is Functional near-infrared spectroscopy (fNIRS) data and, Electroencephalogram (EEG) data and, Eye-tracking data and, A process including one or more of the following: photoplethysmography (PPG) data, A method comprising the step of processing the biometric data to calculate the state immersion scale.

2. The aforementioned biometric data includes fNIRS data, The process of processing the fNIRS data is as follows: A step to detect a decrease in default mode network (DMN) activity, A process for detecting increases in human mirror system (hMNS) activity and frontoparietal attention network (FPN) activity, A step of calculating the state immersion scale based on the decrease in DMN activity, A step of calculating the ratio of activity between the DMN activity and the combination of the hMNS activity and the FPN activity, A step of calculating the state immersion scale based on the ratio of the activities, The method according to claim 1, comprising the steps of: calculating the state immersion scale based on the increase in the hMNS activity and the FPN activity; and calculating by

3. The aforementioned biometric data includes fNIRS data, The process of processing the fNIRS data is as follows: A step of generating pruned fNIRS intensity data by removing fNIRS data channels that are saturated or not receiving sufficient light, The process involves converting the pruned fNIRS intensity data into fNIRS optical density (OD) data, A step of generating filtered fNIRS data by applying a filter, including a low-pass filter or a band-pass filter, to the fNIRS OD data to suppress physiological noise and high-frequency instrument noise, In response to determining the presence of short channels, the filtered fNIRS data is processed using a general linear model (GLM) to generate GLM fNIRS data; The process involves generating regression fNIRS data by performing regression on the aforementioned GLM fNIRS data and suppressing short-channel physiological noise, The process of processing the aforementioned regression fNIRS data to generate multiple fNIRS epochs, A step of processing the plurality of fNIRS epochs to detect the movement of fNIRS within one or more epochs, A motion correction application step, which involves applying motion correction to one or more epochs in response to detecting motion within one or more epochs, thereby generating a plurality of motion-corrected epochs; A step of processing the motion-corrected plurality of epochs and generating a plurality of oxygenation (HbO) and deoxygenation (HbR) data epochs based on the optical density of the motion-corrected plurality of epochs, A step of generating multiple oxygenation epochs by applying an averaging function to the multiple oxygenation (HbO) and deoxygenation (HbR) data epochs, wherein each oxygenation epoch has its own oxygenation scale, A step of processing each oxygenation epoch and determining the DMN activity and hMNS activity for the oxygenation epoch based on the oxygenation scale of the oxygenation epoch, The method according to claim 2, comprising the step of calculating the state immersion scale for each oxygenation epoch based on the ratio of the DMN activity to the hMNS activity.

4. The aforementioned biometric data includes EEG data, The process for processing the aforementioned EEG data is: A step of detecting a decrease in the high-frequency bandwidth power of the EEG data, wherein the high-frequency bandwidth power includes high-frequency power including alpha waves and beta waves, A step of detecting an increase in the theta wave power of the EEG data, The aforementioned state immersion scale, A step of calculating the ratio between the theta wave power and the high-frequency bandwidth power, The method according to claim 2, comprising the steps of: calculating the state immersion scale based on the ratio; and calculating by

5. The process for processing the aforementioned EEG data is: An EEG data filtering step is performed to filter the aforementioned EEG data and generate filtered EEG data. A step of generating differential EEG data by subtracting a standard from the filtered EEG data, The process of processing the differential EEG data to generate multiple EEG epochs, A process of cleaning the multiple EEGs of defective data to generate multiple clean EEG epochs, Each Clean EEG epoch, A step of calculating a normalized theta wave power value based on the power in the 4-7 Hz frequency band of the aforementioned EEG epoch, A step of calculating a normalized alpha wave power value based on the power in the 8-12 Hz frequency band of the aforementioned EEG epoch, A step of calculating a normalized beta wave power value based on the power in the 13-20 Hz frequency band of the aforementioned EEG epoch, A ratio calculation step of calculating a ratio based on the normalized theta wave power value, the normalized alpha wave power value, and the normalized beta wave power value of the clean EEG epoch, The method according to claim 4, comprising the step of calculating the state immersion scale of the clean EEG epoch based on the ratio.

6. The EEG data filtering process described above is: A step of applying a bandpass filter having a high-pass cutoff at 1 Hz and a low-pass cutoff at 55 Hz, This includes a step to suppress power line interference at 60 Hz, Each of the aforementioned multiple EEG epochs includes a 10-second window with a 5-second overlap between each window and the subsequent window. The 4-7 Hz frequency band, the 8-12 Hz frequency band, and the 13-20 Hz frequency band of each of the aforementioned EEG epochs are, The process of determining the power in the 4-20 Hz frequency band of the aforementioned EEG epoch, The power in the 4-20 Hz frequency band is calculated by dividing it into the 4-7 Hz frequency band, the 8-12 Hz frequency band, and the 13-20 Hz frequency band. The ratio calculation step includes dividing the normalized theta wave power value by the sum of the normalized alpha wave power value and the normalized beta wave power value of the clean EEG epoch, The method according to claim 5, further comprising the step of identifying one or more clean EEG epochs having a state immersion scale that exceeds a threshold as high immersion epochs.

7. The aforementioned biometric data includes eye-tracking data, The eye-tracking data includes video data showing the pupil of the human subject's eye, The process of processing the aforementioned eye-tracking data is: The process of detecting the pupil dilation scale, The method according to claim 1, comprising the step of calculating the state immersion scale based on the extended scale.

8. The aforementioned biometric data includes PPG data, The process for processing the PPG data is as follows: A step of determining the heart rate based on the PPG data, A step of generating respiratory data based on the PPG data, The method according to claim 1, comprising the step of calculating the state immersion scale based on the heart rate and the respiratory data.

9. Based on the aforementioned state immersion scale, flash valve memory (FBM) potential events are identified. A step of detecting a period during which the state immersion scale exceeds the FBM threshold for at least a predetermined length of time, The method according to any one of claims 1 to 7, further comprising the steps of detecting the generation or recall of a flashbulb memory by a human based on the identification of the FBM potential event, and identifying by

10. The process of processing the biometric data and calculating the state immersion scale is as follows: A process of collecting self-reported state immersion data and training with biometric data from one or more training subjects, The process involves using the self-reported state immersion data as semantic labels for supervised learning to train a machine learning model and predict the state immersion data for one or more training subjects. The method according to any one of claims 1 to 7, comprising the step of using the trained machine learning model to generate the state immersion scale for the human subject based on the biometric data of the human subject.

11. A method for generating a personalized music library, A music segment presentation process in which multiple music segments are presented to a human subject, A state immersion scale determination step for determining the state immersion scale of the human subject during the presentation of each music segment, A step of calculating a music trait immersion scale for each music segment based on the state immersion scale of the human subject during the presentation of each music segment, A method comprising the step of selecting one or more music segments from the plurality of music segments to include in the personalized music library based on the music trait immersion scale of the one or more music segments for the human subject.

12. The method according to claim 11, wherein one or more music segments are selected to be included in the personalized music library based on having a high musical trait immersion scale.

13. A step of processing one or more music segments selected to be included in the personalized music library and generating MIR feature data of the music segments, The process of obtaining multiple additional music segments, The method according to claim 11, further comprising: an additional music segment selection step of selecting one or more of the additional music segments to include in the personalized music library based on the similarity of the MIR feature data of the one or more additional music segments to the MIR feature data of the one or more music segments.

14. The aforementioned additional music segment selection step is: The process of obtaining pre-generated MIR feature data of the additional music segment, The method according to claim 13, comprising the step of processing the MIR feature data of one or more music segments and the pre-generated MIR feature data of an additional music segment to determine the similarity between the MIR feature data of one or more music segments and the pre-generated MIR feature data of an additional music segment.

15. The aforementioned additional music segment selection step is: A step of processing the additional music segments to generate the MIR feature data of the additional music segments, The method according to claim 13, comprising the step of processing the MIR feature data of one or more music segments and the MIR feature data of an additional music segment to determine the similarity between the MIR feature data of one or more music segments and the MIR feature data of an additional music segment.

Images