System and method

Abstract

According to a first aspect of the disclosure, there is provided a computer system for encoding a predicted emotional reaction of a subject to sensory media to be sensorially experienced by the subject, comprising: an emotion prediction unit configured receive input data corresponding to sensory media to be sensorially experienced by a subject, to execute a first computer programme configured to determine a predicted emotional response of a subject to the input sensory media based on the input data and output emotional response data corresponding to the predicted emotional response; an anchor determining unit configured to determine at least one anchor within the sensory media to which the emotional response data is attributable, and output the at least one anchor; and an emotion encoding unit configured to generate an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with data relating to the emotional response of the subject to the input sensory media, based on the input sensory media, the emotional response data and the at least one anchor.

Description

[0001] SYSTEM AND METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to computer implemented methods and systems for encoding a predicted emotional reaction of a user to input sensory media sensorially experienced by the user, and predicting an emotional reaction of a user to input sensory media sensorially experienced by the user.

[0004] BACKGROUND ART

[0005] "Sensory media" may refer to media content that engages senses beyond just sight and sound. Traditional media primarily rely on sight and sound, but sensory media may incorporate touch, smell, and sometimes even taste into the experience. This can include technologies like virtual reality (VR) and augmented reality (AR), which aim to immerse users in a multi-sensory environment. For example, VR headsets can provide visual and auditory experiences while also simulating touch through haptic feedback devices. Herein, the term sensory media is used to refer to media content that engages any one of the human senses, including but not limited to either sight or sound.

[0006] Increasingly, people are experiencing and interacting with sensory media. Users can experience these sensory media in different ways including using a VR headset that displays an image of a virtual environment as the sensory media. Users can interact with these virtual environments in different ways including with their own body, detected using sensors, or by using hardware controllers. Virtual environments are used particularly in gaming, but are becoming more prevalent in other contexts that replicate “everyday” real environments, such as shops.

[0007] In designing sensory media with particular attributes, e.g. that elicit a desired response in a subject, there is a distinct lack of tools available that can provide insights into the emotional reaction of a user to sensory media.

[0008] The present disclosure aims to at least partially solve the above problem. SUMMARY OF THE INVENTION

[0009] According to a first aspect of the disclosure, there is provided a computer system for encoding a predicted emotional reaction of a subject to sensory media to be sensorially experienced by the subject, comprising: an emotion prediction unit configured receive input data corresponding to sensory media to be sensorially experienced by a subject, to execute a first computer programme configured to determine a predicted emotional response of a subject to the input sensory media based on the input data and output emotional response data corresponding to the predicted emotional response; an anchor determining unit configured to determine at least one anchor within the sensory media to which the emotional response data is attributable, and output the at least one anchor; and an emotion encoding unit configured to generate an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with data relating to the emotional response of the subject to the input sensory media, based on the input sensory media, the emotional response data and the at least one anchor.

[0010] According to a second aspect of the disclosure, there is provided a computer-implemented method of encoding a predicted emotional reaction of a subject to sensory media to be sensorially experienced by the subject, comprising: receiving input data corresponding to sensory media to be sensorially experienced by a subject; determining a predicted emotional response of a subject to the input sensory media based on the input data and outputting emotional response data corresponding to the predicted emotional response; determining at least one anchor within the sensory media to which the emotional response data is attributable, and outputting the at least one anchor; and generating an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with data relating to the emotional response of the subject to the input sensory media, based on the input sensory media, the emotional response data and the at least one anchor.

[0011] According to a third aspect of the disclosure, there is provided a computer system for predicting an emotional response of a subject to sensory media experienced by the subject, comprising: an emotion prediction unit configured receive input data corresponding to sensory media to be sensorially experienced by a subject, to execute a computer programme configured to determine a predicted emotional response of a subject io me input sensory media based on the input data and output emotional response data corresponding to the predicted emotional response.

[0012] According to a fourth aspect of the disclosure, there is provided a computer-implemented method of predicting an emotional response of a subject to sensory media experienced by the subject, comprising: receiving an input dataset comprising data corresponding to sensory media sensorially experienced by a subject; determining a predicted emotional response of a subject to the input sensory media based on the input data, and at least one anchor within the sensory media to which the emotional response data is attributable; and producing an output dataset comprising emotional response data corresponding to predicted emotional response together with the at least one anchor.

[0013] According to a fifth aspect of the disclosure, there is provided a machine-learning model for predicting an emotional response of a subject to an sensory media experience by the subject, in particular for use in the method of the second or fourth aspects, comprising: an artificial neural network trained to determine the predicted emotional response of a subject to an input sensory media based on the input sensory media, having been trained based on training data comprising sensory media data relating to at least one training sensory media experienced by a real subject labelled with corresponding emotional response data relating the real subject’s emotional reaction to the training sensory media.

[0014] According to a sixth aspect of the disclosure, there is provided a computer-implemented method of training a machine-learning model for predicting an emotional response of a subject to sensory media experiences by the subject, in particular the machine-learning model of the fifth aspect, comprising: receiving an input training dataset comprising sensory media data relating to at least one training sensory media experienced by a real subject labelled with corresponding emotional response data relating the real subject’s emotional reaction to the training sensory media and training the machine learning algorithm to associate features of an input sensory media with emotional reactions.

[0015] According to a seventh aspect of the disclosure, there is provided a computer-implemented method of generating a training dataset for a machine-learning model, in particular the machine-learning model of the fifth aspect, comprising: obtaining physiological response data relating to a physiological response of a real subject to sensory media expedience uy the subject, and sensory media data relating to the sensory media experienced by the subject.

[0016] According to an eighth aspect of the disclosure, there is provided a training dataset for use in the method of training a machine-learning model of the sixth comprising: physiological response data relating to a physiological response of a real subject to sensory media experience by the subject, and sensory media data relating to the sensory media experienced by the subject.

[0017] According to a ninth aspect of the disclosure, there us provided a use of the method of the fourth aspect for at least one of: generating an emotionally encoded representation of input sensory media encoded with the predicted emotional response of a subject to the sensory media; and modifying sensory media based on the predicted emotional response of a subject to the sensory media.

[0018] According to a tenth aspect of the disclosure, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of the preceding aspects.

[0019] According to an eleventh aspect of the disclosure, there is provided a data processing device comprising means for carrying out the method of any one of the preceding aspects.

[0020] According to a twelfth aspect of the disclosure, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of the preceding aspects.

[0021] Optionally, the subject is a real or virtual subject.

[0022] Optionally, the first computer programme comprises a first machine learning model trained to determine the predicted emotional response of a subject to the input sensory media based on the input data. Optionally, training data for training the first machine learning model further comprises sensory media data relating to training sensory media experienced by a real subject, labelled with corresponding emotional response data. Optionally, the training emotional response data is based on at least one emotional response of a real subject to training sensory media. Optionally, the emotional response is determined based on physiological data relating to a physiological response of the subject to the training sensory media.

[0023] Optionally, the emotional response data for training the first machine learning model, is determined by a second computer programme configured to determine an emotional response based on input physiological data. Optionally, the second computer programme comprises a second machine learning model trained to determine a predicted emotional response based on input physiological data.

[0024] Optionally, the second machine learning model is trained based on training data comprising physiological data relating to a physiological response of the subject to the training sensory media labelled with a corresponding emotional response.

[0025] Optionally, the second computer programme is configured to label the sensory media data with emotional response data by determining the emotional response data based on physiological data corresponding to the sensory media data.

[0026] Optionally, the physiological response data relates to at least brain activity. Optionally, the physiological response data comprises EEG data. Alternatively, or additionally, the physiological response data relates to at least one of eye movement, pupil dilation, heart rate, and sweating.

[0027] Optionally, the training data is obtained by a training data obtaining system comprising: a physiological data collection unit configured to collect physiological response data relating to a physiological response of a training subject to an input sensory media. Optionally, the training data obtaining system further comprises a user interface unit configured to enable a user to sensorially experience the input sensory media.

[0028] Optionally, the training data on which the first machine learning model is trained additionally comprises subject data relating to characteristics of the training subject, such that the first machine learning model is configured to make the determination auuiuuiuuiy based on input subject data relating to the subject.

[0029] Optionally, the training data on which the second machine learning model is trained additionally comprises subject data relating to characteristics of the training subject, such that the second machine learning model is configured to make the determination additionally based on input subject data relating to the subject.

[0030] Optionally, the subject characteristics include one or more of age, gender, sex, sexual orientation, culture, nationality, neurodivergence (such as Autism, ADHD, Dyslexia).

[0031] Optionally, the emotionally encoded representation of the input sensory media is visually encoded with data relating to the emotional response of the subject to the input sensory media.

[0032] Optionally, the encoded data is configured to be visually decodable by a second user. Optionally, the encoded data represents the emotional response of the subject to the input sensory media using variation in colour.

[0033] Optionally, the encoded data represents the emotional response of the subject to the input sensory media using a heat map.

[0034] Optionally, the emotionally encoded representation of the input sensory media comprises emotionally encoded sensory media that can be sensorially experienced by the subject. Optionally, a user interface unit is configured to enable the user to sensorially experience the emotionally encoded sensory media.

[0035] Optionally, the emotionally encoded representation of the input sensory media comprises an image of the emotionally encoded sensory media.

[0036] Optionally, the emotional response data relates to at least one of a level of stress, a level of attentiveness, and a level of relaxation experienced by the subject. Optionally, the sensory media comprises elements that are sensorially experienced ai least one of visually, audibly, haptically, thermally, equilibrioceptively, nociceptively, olfactorially, and gustatorially.

[0037] Optionally, the anchor determining unit is configured to determine each anchor based on an interaction between the subject and the input sensory media. Optionally, the interaction comprises at least one of an experienced location of the subject within the input sensory media, an experienced orientation of the subject within the input sensory media, a region of the subject’s sensory attention within the sensory media, and an experienced event within the sensory media.

[0038] Optionally, one or both of the first machine learning model and the second machine learning model is an artificial neural network.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further features of the disclosure will be described below, by way of non-limiting examples and with reference to the accompanying drawings, in which:

[0041] Fig. 1 shows an example operational system;

[0042] Fig. 2 shows example emotionally encoded representations of a virtual environment;

[0043] Fig. 3 shows further example emotionally encoded representations of a virtual environment;

[0044] Fig. 4 shows an example training system for a machine learning model.

[0045] DETAILED DESCRIPTION

[0046] Fig. 1 shows a first example system 100 according to the disclosure. As shown, the example system 100 may comprise a computer system 101 and a display system 102. The computer system 101 comprises an emotion prediction unit 103. The emotion prediction unit 103 is configured receive input data corresponding to sensory media to be sensorially experienced by a subject and output emotional response data corresponding to a predicted emotional response. According to the example, the display system 102 is configured to display an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with the cuiuuuu response data.

[0047] The sensory media may comprise elements that are sensorially experienced at least one of visually, audibly, haptically, thermally, equilibrioceptively, nociceptively, olfactorially, and gustatorially, for example. In an example system, the sensory media may comprise visual and audible elements experienced by a subject though a combination of a visual display and a speaker. The visual display and / or the speaker may form part of a headset worn by a subject, e.g. a VR headset such as those from Oculus VR™. The sensory media may be, or form part of, a virtual environment in some examples.

[0048] The emotional response data may relate to at least one of a level of stress, a level of attentiveness, and a level of relaxation, experienced by the subject. Alternatively, or additionally, the emotional response data may relate to one or more of: happiness, sadness, anger and disgust. Alternatively, or additionally, the emotional response data may relate to one or more of: cognitive load, mental strain, cognitive fatigue, and cognitive relaxation. Preferably the emotional response data relates to a numerical value or score relating to the intensity of one or more emotions, e.g. any of those identified above.

[0049] Fig. 2 shows different emotionally encoded representations of an input virtual environment, as the sensory media, when a subject moved through a path in the virtual environment from A to B, as shown in the in the left-hand part of the Figure. The numbers 1 to 4 in the Figure denote points along the path. The left -hand parts of the Figure show a heat map 200 of the subject’s emotional reaction at different locations in the virtual environment 300. In this example, the emotional response data is attributable to the subject’s location in the virtual environment. The top right part of the Figure is an example of a three-dimensional emotionally encoded representation, whereas the bottom right part of the Figure is an example of a two-dimensional emotionally encoded representation.

[0050] Fig. 3 shows different emotionally encoded representations of an input virtual environment when a subject interacts with an object 400 in the virtual environment, as shown in the in the left-hand part of the Figure. In this example, the emotional response data is attributable to an object in the virtual environment with which the subject interreacted. As shown in the top right part of the Figure, in some examples, e.g. when a subject is in piuxinniy io a virtual object 400, the object itself may be emotionally encoded 500, e.g. with a colour corresponding with the subject’s emotional reaction. As shown in the bottom right part of the Figure, in some examples, e.g. where a subject moves a virtual object, a visual representation of the interaction may be provided together with emotional encoding 500 of the object 400.

[0051] The input data 104 corresponding to the sensory media may comprise data relating to a subject’s experience of the sensory media. This data may comprise visual and / or audio data, for example. This data may be pre-recorded or obtained live, as the subject is experiencing the sensory media. In an example where pre-recorded input data is provided, this may be provided by means of any suitable data storage medium.

[0052] The sensory media may be generated by a sensory media generating unit (not shown). In an example where live input data is obtained, the sensory media generating unit may be in data communication with the computer system 101 to provide the input data. The sensory media generating unit may provide data and / or instructions to a user interface unit for enabling a subject to sensorially experience an input sensory media.

[0053] In an alternative example, input data comprising data relating to a subject’s experience of sensory media may be generated based on an input sensory media. For example, a subject’s experience of the sensory media may be simulated. For example, movement of a virtual subject through an (e.g. virtual) environment, and / or interaction between a virtual subject and the environment may be simulated. Simulation may be performed with or without user input.

[0054] The computer system 101 may additionally comprise an anchor determining unit (not shown) configured to determine at least one anchor within the input sensory media to which the emotional reaction data is attributable. The anchor determining unit may be configured to determine each anchor based on an interaction between the subject and the input sensory media. The interaction may comprise at least one of an experienced location of the subject within an (e.g. virtual) environment corresponding to the input sensory media, an experienced orientation of the subject within an (e.g. virtual) environment corresponding to the input sensory media, a region of the subject’s sensory attention within the sensory media, an experienced event within the sensory media, or a time m me sensoi media.

[0055] An experienced location may be a location in the sensory media, at which the subject experiences the sensory media. This may be represented by coordinates within a coordinate system of the sensory media, for example. An experienced orientation may be the orientation in the sensory media at which the subject experiences the sensory media. This may be represented by a direction within the coordinate system of the sensory media, for example.

[0056] A region of the subject’s sensory attention may be a region of the sensory media that receives sensory attention from the subject. This may be visual sensory attention, for example based on the experienced location and / or orientation of the subject and / or eye tracking data to determine a region of the sensory media that the subject is looking at. However, sensory attention is not limited to visual attention. For example, if the subject interacts with the sensory media haptically, a region of the sensory media, such as a virtual object, that they experience touching may be a region of haptic sensory attention.

[0057] An experienced event may be an event within the sensory media that is experienced by the subject. An experienced event may be any substantial change in the sensory media experienced by the subject. The experienced event may relate to any of the senses that the subject is able to experience the sensory media through, for example a visual event or an audio event. For example, the event may be a planned narrative event within the sensory media, e.g. part of an unfolding story line.

[0058] Anchors may be determined based on the input sensory media data. The input sensory media data may provide data relating to an experienced location and / or an experienced orientation.

[0059] The sensory media data may include interaction data relating to subject interaction with the sensory media, for example if the subject is able to interact via a control means, data relating to the manner of control exercised by the subject may be used to determine an anchor. Control means may include one or more of movement sensors (e.g. cameras and associated movement recognition software), mechanical controllers (e.g. buttonsjoysticks, haptic gloves), means for voice control (e.g. microphone and associated voice leeogmuon software). Interaction data may also comprise gaze-tracking data relating to where the subject is looking in the sensory media at a given moment.

[0060] Sensory media data may also include data relating to the sensory media itself, for example, events within the sensory media or objects within the sensory media. Such sensory media data may be provided by a processing unit configured to generate the sensory media or a simulation.

[0061] The specific data used to determine the anchors may depend on the level of subject interaction permitted with the sensory media.

[0062] The data for determining the anchors may be processed by a processing unit of the system to determine the anchors. This processing may be performed by a different processing unit to that which generates the sensory media. However, in some examples, these processing units may be the same processing units, or different units within the same processing device.

[0063] The insights and inferences available from the sensory media data may differ depending on the data utilised. For example, sensory media data relating to the subject’s movement through an (e.g. virtual) environment corresponding to the sensory media may be used to determine which regions of the sensory media have elicited an emotional response. The sensory media data relating to specific objects within an (e.g. virtual) environment corresponding to the sensory media, such as the subject’s experienced proximity to an object or sensory interaction with an object may be used to determine which objects have elicited an emotional response.

[0064] The anchors may be parts of the sensory media with which emotional response data may be attributed. For example, the anchors may be one or more voxels within a three- dimensional sensory media, or pixels of a two-dimensional representation of the sensory media. These voxels (or pixels) may be associated with a specific location within the sensory media and / or may be associate with a specific object within an (e.g. virtual) environment corresponding to the sensory media, for example. In the first case, the anchor may be a fixed position within the sensory media. In the latter case, the anchor may not be in a fixed position within in the sensory media. Anchors may also be associated wnn a specific time-frame within the subject’s experience of the sensory media.

[0065] Data for determining anchors may be sampled at the same rate as data for determining emotional reactions. However, they may alternatively be sampled at different rates. If sampled at different rates, data sampled at the higher rate may be averaged over the interval of the lower sampling rate to provide correspondence. The sampling may be performed continuously for the period the subject experiences the sensory media.

[0066] The computer system 101 of the disclosure may further comprise an emotion encoding unit configured to generate an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with the emotional response data, based on the input sensory media, the emotional response data and the at least one anchor.

[0067] The emotionally encoded representation of the input sensory media may be visually encoded with data relating to the emotional response of the subject to the input sensory media. The encoded data may be configured to be visually decodable by a user of the system. The user may be different from the subject, or may be the subject. The encoded data may represent the emotional response of the user to the input sensory media using variation in colour, shade, tone or transparency. For example, the encoded data may represent the emotional response of the user to the input sensory media using a heat map, as shown in Fig. 2.

[0068] In some examples, the emotionally encoded representation of the input sensory media may comprise an image of an emotionally encoded sensory media. For example, the emotionally encoded representation of the input sensory media may comprise a two- dimensional image of a three-dimensional (e.g. virtual) environment corresponding to the input sensory media. This image may be a top-down (plan) view of an (e.g. virtual) environment corresponding to the sensory media, for example, as shown in Fig. 2.

[0069] According to the disclosure, the emotion prediction unit 103 is configured to execute a computer programme configured to determine a predicted emotional response of a subject to the input sensory media based on the input data, and output emotional response data corresponding to the predicted emotional response. In an example, the compuiei programme comprises a first machine learning model trained to determine the predicted emotional response of a subject to the input sensory media based on the input data.

[0070] Fig. 4 shows an example training system 200 for training the machine learning model 201. As shown, the machine learning model 201 may be trained on training data comprising sensory media data 202 relating to at least one training sensory media experienced by a real subject, labelled with emotional response data 203. Accordingly, the machine learning model 201 learns (in a supervised manner) to associate particular sensory media experiences with particular emotional responses. The emotional response data 203 may be based on at least one response of the real subject to the training sensory media.

[0071] As shown in Fig. 4, the emotional response may be determined based on physiological data 204 relating to a physiological response of the subject to the training sensory media. As shown in Fig. 4, the emotional response data 203 may be determined by a second computer programme 205 configured to determine an emotional response based on input physiological data. As in the example shown, the second computer programme may comprise a second machine learning model 205 trained to determine an emotional response based on input physiological data.

[0072] Fig. 4 shows a subsystem 206 for training the second machine learning model 205. The second machine learning model may be trained based on training data comprising physiological data relating to a physiological response of a subject to training sensory media, labelled with a corresponding emotional response. The training data for the second machine learning algorithm may be obtained by measuring test subjects’ physiological responses to tests that elicit particular emotions. Accordingly, the machine learning algorithm 205 learns (in a supervised manner) to associate particular physiological responses to particular emotional responses.

[0073] The tests may comprise real experiences of sensory media. As well as obtaining physiological response data, emotional responses may be obtained. The emotional responses may be self-recorded by the subjects, e.g. through a questionnaire or similar, or obtained by observation by a human tester. The emotional responses may be determined based on alternative physiological responses, such as body language and / or fa ia, expressions, by the human tester and / or by a computer programme.

[0074] As shown, the physiological response data and the emotional response data may be consolidated together to form the training dataset. As shown, the training data may additionally comprise subject data relating to characteristics of the training subject, such that the first machine learning-based model is configured to make the determination additionally based on input subject data relating to the subject. The subject characteristics include one or more of: age, gender, sex, sexual orientation, culture, nationality, neurodivergence (such as Autism, ADHD, Dyslexia). These characteristics may influence the emotional and physiological responses of different subjects.

[0075] The second machine learning model 205 may be an artificial neural network (ANN), such as a deep neural network (DNN) or multilayer perceptron (MLP).

[0076] In other examples, alternative methods for determining the emotional response based on physiological response data may be used. For example, a rules-based model may be used instead of a machine learning model.

[0077] Fig. 4 shows a subsystem 207 for obtaining training data for training the first machine learning model 201, including the sensory media data 203 and the physiological response data 204. The sensory media data 202 relates to sensory media experienced by a test subject and the physiological response data 204 relates to the physiological response of the test subject to that sensory media. Both sets of data may be time-series data matched in time.

[0078] As shown, the sensory media data 203 and the physiological response data 204 may be consolidated together to form a precursor to the training dataset. The physiological response data 204 may then be converted to emotional response data 203 to form the training dataset. The consolidated data may be divided into different data samples for training the first machine learning model. For example, time-series data may be segmented to have a predefined length in time, or segmented to have varying lengths (e.g. including single video frames). As shown, the training data may additionally comprise subject data relating to characteristics of the training subject, such that the first machine learning-based model is configured to make the determination additionally based on input subject data relating to the subject. The subject characteristics include one or more of: age, gender, sex, sexual orientation, culture, nationality. These characteristics may influence the emotional and physiological responses of different subjects.

[0079] The first machine learning model 201 may be an artificial neural network (ANN) model, such as a deep neural network (DNN) or multilayer perceptron (MLP).

[0080] The physiological data may relate to at least one of brain activity, eye movement, pupil dilation, heart rate, and sweating.

[0081] The training system 200, e.g. subsystems 207 and 206, may comprise a physiological data collection unit configured to collect physiological response data relating to a physiological response of a subject to sensory media. The physiological data collection unit may comprise corresponding sub-units to collect the respective data.

[0082] For example, the physiological data collection unit may comprise EEG sensors configured to sense brain activity. The physiological data collection unit may comprise a camera (e.g. visible or infrared light), and associated software, to track eye movement and / or pupil dilation. The physiological data collection unit may comprise a heart rate monitor (e.g. a Holter monitor). The physiological data collection unit may comprise galvanic skin response electrodes to collect data relating to sweating.

[0083] An EEG sensor system may comprise of the following, for example:

[0084] 1. Electrical activity measuring electrodes configured to be placed on or in the proximity to the head with the purpose of receiving and transmitting electrical activity travelling through the scalp having originated form the brain.

[0085] 2. An amplifier for amplifying and / or converting analogue electrical signals from the sensor into a digital signal that can be processed by a processor.

[0086] 3. A signal transmitter that will send the data from the amplifier to the processor.

[0087] An eye tracking system may comprise of the following, for example: 1. A visual or infrared light camera directed towards the eyes with the puipose 01 measuring the eye movement and pupil dilation changes of the system subject.

[0088] 2. A receiver unit for the input of visual data which can be translated into digital data.

[0089] 3. A transmission unit for the purpose of transmission of digital data to a processor.

[0090] A decrease in Alpha pattern brainwaves may indicate that the sensory media has elicited a higher than normal level of attention from the subject, for example. Increase in pupil dilation may indicate that the subject is attracted towards an object within the sensory media. Galvanic skin response and heart rate may indicate emotional arousal strength.

[0091] In other examples, alternative training methodologies may be used other than that shown in Fig. 4. For example, the physiological response data may not be converted to emotion response data prior to training the first machine learning model 201. Instead, the machine learning model may be trained to predict a physiological response to sensory media and a second computer programme may determine emotional response data based on the output predicted physiological response data.

[0092] Referring back to Fig. 1, the trained machine learning model 201 forms part of the emotional prediction unit 103 of the operational system 100. Predictions may be output to a data dashboard 105 to be viewed or otherwise used by a user. This may include the emotionally encoded representation of the sensory media, and may include additional forms of the data. These additional forms may include charts or graphs, for example, showing the changing level of emotion of a subject. The emotionally encoded representation and or additional forms of data may be output in real-time (or near realtime), as a subject experiences the sensory media.

[0093] In some examples, the emotionally encoded representation of the input sensory media comprises emotionally encoded sensory media that can be sensorially experienced by the user, e.g. via a user interface unit. For example, the emotionally encoded representation of the input sensory media may be a modified version of the input sensory media.

[0094] In some examples, the emotionally encoded representation of the input sensory media may be fed back to the subject or a user of the system 100 as it is generated. Alternatively, or additionally, the emotionally encoded representation may be stored in a memoiy 101 viewing later.

[0095] The emotionally encoded representation of the input sensory media may comprise a representation of the input sensory media integrated with the encoded emotional reaction data. In this case, the data relating to the encoding may be indistinguishable from of the data relating to the representation of the input sensory media. Alternatively, or additionally, the emotionally encoded representation of the input sensory media may comprise a representation of the input sensory media overlaid (or augmented) with the encoded emotional response data. In this case, the data relating to the encoding may be separate from the data relating to the representation of the input sensory media. For example, the data relating to the encoding may be a two-dimensional or three-dimensional heatmap.

[0096] Generating an emotionally encoded representation of the input sensory media may be preceded by combining emotional response data with the anchors. This may comprise mapping emotional response data to anchors, or vice versa.

[0097] In an example system, emotional response data for a specific time may be mapped to an anchor for the same time, i.e. relating to the subject’s interaction with the sensory media at the specific time. The emotional response data may then be determined and encoded as a heat map within encoded sensory media based on the anchors.

[0098] The emotional response data may indicate a level of one or more emotional states including but not limited to those identified above, namely stress, attention, relaxation, happiness, sadness, anger, disgust, cognitive load, mental strain, cognitive fatigue, and cognitive relaxation. Colours, changes in colours, changes in colour tones and strengths of colours or changes in opaqueness may be used to visually represent these emotional states, e.g. in a heat map. The emotional states displayed may be configurable by a user of the system. User configurations may be provided as input via a configuration unit 106, shown in Fig. 1

[0099] A colour range or colour strength may be assigned to correspond with each emotional state. The specific colour may be configurable by the user. These colours may represent the strength, decrease, increase or other change; for example, a light blue may repiesem a lower attention measurement, whilst a dark blue may represent a measurement of high attention. If the user moved from the co-ordinate of 0,0,0 to 0,10,10 in a virtual environment and their attention levels were measured to have increased from low attention to high attention at an even rate between the two points, the areas around which they began movement, the area in which they moved, and the destination area, may be coloured, starting with a light blue and demonstrating a gradual change to a dark blue across this path.

[0100] In the case of a two-dimensional visual representation, such as a top-down view or a subject-centric view of an (e.g. virtual) environment corresponding to the sensory media, a colour overlay may be placed over the image of the environment. This overlay is generated by analysing the data for its inference of each metric strength, and then applying the appropriate colour range to the correct spatial co-ordinates within the visual representation of the environment.

[0101] In the case of the utilisation of the original sensory media in the generation of the visual heatmap, such as overlaying the coloured heatmap onto the original sensory media itself, copies of the original files containing the representations of the 3D objects in the environment may be created and altered. These files may be located in or linked to the software generating the 3D environment. These files will follow the same 3D layout as the original file, however altering the colours in the appropriate manner to display the aforementioned metrics.

[0102] In the case the coloured overlay demonstrating which activities, or objects induce certain metrics, the files relating to or linked to these specific 3D objects may be copied, and altered colours will be applied in the same manner.

[0103] In some example systems, the predicted emotional response output by the emotion prediction unit 103 may be used to alter the sensory media experienced by the user. This may occur in real-time as the subject is experiencing the sensory media in some case. In other cases, the sensory media may be modified for future subjects to experience. In the example of a computer game, if a level of stress of the subject is determined to be too low or too high compared to a desired level, then sensory media may be modified to include stress inducing features and / or remove stress reducing features, or remove stiess inuuunig features and / or include stress reducing features respectively. The same principle applies for other emotional reactions experienced by the subject and any or all senses that the sensory media relates to. Examples may include modifying media content, visual image colour, hue or brightness, audio pitch, volume or tempo, or haptic feedback strength.

Claims

CLAIMS1. A computer system for encoding a predicted emotional reaction of a subject to sensory media to be sensorially experienced by the subject, comprising: an emotion prediction unit configured receive input data corresponding to sensory media to be sensorially experienced by a subject, to execute a computer programme configured to determine a predicted emotional response of a subject to the input sensory media based on the input data and output emotional response data corresponding to the predicted emotional response, and anchor determining unit configured to determine at least one anchor within the sensory media to which the emotional response data is attributable, and output the at least one anchor; an emotion encoding unit configured to generate an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with data relating to the emotional response of the subject to the input sensory media, based on the input sensory media, the emotional response data and the at least one anchor.

2. The computer system of claim 1, wherein the computer programme comprises a first machine learning model trained to determine the predicted emotional response of a subject to the input sensory media based on the input data.

3. Th computer system of claim 2, wherein the first machine learning model is an artificial neural network.

4. The computer system of claim 2 or 3, wherein training data for training the first machine learning model further comprises sensory media data relating to training sensory media experienced by a real subject, labelled with corresponding emotional response data.

5. The computer system of claim 4, wherein the training emotional response data is based on at least one emotional response of a real subject to training sensory media.

6. The computer system of claim 5, wherein the emotional response is duiuiiiniiuu based on physiological data relating to a physiological response of the subject to the training sensory media.

7. The computer system of any one of claims 4 to 6, wherein the emotional response data for training the first machine learning model, is determined by a second computer programme configured to determine an emotional response based on input physiological data.

8. The computer system of claim 7, wherein the second computer programme comprises a second machine learning model trained to determine a predicted emotional response based on input physiological data.

9. The computer system of claim 8, wherein the second machine learning model is trained based on training data comprising physiological data relating to a physiological response of the subject to the training sensory media labelled with a corresponding emotional response.

10. The computer system of claim 8 or 9, wherein the second computer programme is configured to label the sensory media data with emotional response data by determining the emotional response data based on physiological data corresponding to the sensory media data.

11. The computer system of any one of claims 6 to 10, wherein the physiological response data relates to at least brain activity,12. The computer system of claim 1, wherein the physiological response data comprises EEG data.

13. The computer system of any one of claims 6 to 12, wherein the physiological response data relates to at least one of eye movement, pupil dilation, heart rate, and sweating.

14. The computer system of any one of claims 2 to 13, wherein the training uaia is obtained by a training data obtaining system comprising: a physiological data collection unit configured to collect physiological response data relating to a physiological response of a training subject to an input sensory media.

15. The computer system of claim 14, wherein the training data obtaining system further comprises a user interface unit configured to enable a user to sensorially experience the training sensory media.

16. The computer system of any preceding claim, wherein the emotionally encoded representation of the input sensory media is visually encoded with data relating to the emotional response of the subject to the input sensory media.

17. The computer system of claim 16, wherein the encoded data represents the emotional response of the subject to the input sensory media using variation in colour.

18. The computer system of claim 16 or 17, wherein the encoded data represents the emotional response of the subject to the input sensory media using a heat map.

19. The computer system of claim 17 or 18, wherein the emotionally encoded representation of the input sensory media comprises an image of the emotionally encoded sensory media.

20. The computer system of any preceding claim, wherein the emotional response data relates to at least one of a level of stress, a level of attentiveness, and a level of relaxation experienced by the subject.

21. The computer system of any preceding claim, wherein the anchor determining unit is configured to determine each anchor based on an interaction between the subject and the input sensory media.

22. A computer-implemented method of encoding a predicted emotional reaction of a subject to sensory media to be sensorially experienced by the subject, comprising:receiving input data corresponding to sensory media to be sensorially cxpuuunvcu by a subject; determining a predicted emotional response of a subject to the input sensory media based on the input data and outputting emotional response data corresponding to the predicted emotional response, determining at least one anchor within the sensory media to which the emotional response data is attributable, and outputting the at least one anchor; generating an emotionally encoded representation of the input sensory media, whereby the emotionally encoded representation of the sensory media is encoded with data relating to the emotional response of the subject to the input sensory media, based on the input sensory media, the emotional response data and the at least one anchor.

23. A computer system for predicting an emotional response of a subject to sensory media experienced by the subject, comprising: an emotion prediction unit configured receive input data corresponding to sensory media to be sensorially experienced by a subject, to execute a computer programme configured to determine a predicted emotional response of a subject to the input sensory media based on the input data and output emotional response data corresponding to the predicted emotional response.

24. A computer-implemented method of predicting an emotional response of a subject to sensory media experienced by the subject, comprising: receiving an input dataset comprising data corresponding to sensory media sensorially experienced by a subject; determining a predicted emotional response of a subject to the input sensory media based on the input data, and at least one anchor within the sensory media to which the emotional response data is attributable; and producing an output dataset comprising emotional response data corresponding to predicted emotional response together with the at least one anchor.

25. A machine-learning model for predicting an emotional response of a subject to an sensory media experience by the subject, in particular for use in the method of claim 22 or 24 comprising: an artificial neural network trained to determine the predicted emotional response of a subject to an input sensory media based on the input sensory media, havingbeen trained based on training data comprising sensory media data relating to ai least one training sensory media experienced by a real subject labelled with corresponding emotional response data relating the real subject’s emotional reaction to the training sensory media.

Images