System and method for providing digital content to a user for improving mental health

Abstract

A system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the system includes: a measurement system configured to measure physiological data of the user; and a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user.

Description

[0001] SYSTEM AND METHOD FOR PROVIDING DIGITAL CONTENT TO A USER FOR IMPROVING MENTAL HEALTH

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a system and method for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief

[0004] TECHNICAL BACKGROUND

[0005] Some patients may have an anxiety before or during a medical treatment in a hospital or longterm rehabilitation in ambulatory care or at home in some cases. Other individuals may suffer from fatigue or distress or burnout risk which may be due to heavy mental load in some cases.

[0006] An impact on the mental health and wellbeing of an individual may occur in some cases due to one or more of the following effects:

[0007] An emotional deprivation, fear and concern about future and missing social contacts (e.g., during COVID19 pandemics) in some cases; visiting and talking to a psychologist may not happen promptly in some cases considering appointment requirements beforehand and some special situations (e.g., during COVID19 pandemics); long-term rehabilitation may impose high demands on surrounding environments in some cases and takes extra cost and physical burden such as travelling etc. in some cases; some factors may be difficult to be artificially manipulated and controlled in the real world in some cases to fit the requirements of patients such as weather, environmental noise, odors, lighting etc.; and in the real world, some stimulus or experience may be interrupted or weakened by the surroundings in some cases, which makes it challenging to let patients concentrate on the rehabilitation process.

[0008] Although there exist techniques for recovery, mental health, relaxation and mental stress relief of an individual, it is generally desirable to improve the existing techniques.

[0009] SUMMARY

[0010] According to a first aspect, the disclosure provides a system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the system comprises: a measurement system configured to measure physiological data of the user; and a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user. According to a second aspect, the disclosure provides a method for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the method comprises: measuring physiological data of the user; and providing cross-modal content to the user and adapting the provided cross-modal content in accordance with the physiological data of the user.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0014] Fig. 1 schematically illustrates in a block diagram an embodiment of a system for digital cross- modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief;

[0015] Fig. 2 schematically illustrates an embodiment of different mood regions of a user;

[0016] Fig. 3 schematically illustrates in a flow diagram an embodiment of a method for digital cross- modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief;

[0017] Fig. 4 schematically illustrates in a flow diagram an embodiment of a method for adapting cross- modal content to be provided to a user; and

[0018] Fig. 5 schematically illustrates in a flow diagram an embodiment of a method for adapting cross- modal content to be provided to a user.

[0019] DETAILED DESCRIPTION OF EMBODIMENTS

[0020] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0021] As mentioned in the outset, there may exist situations in some cases in which an impact on the mental health and wellbeing of an individual may occur.

[0022] It has thus been recognized that a system and method may be needed to provide mental relief and support in some cases, for example, in which individuals are subject to mentally demanding activities or work or social restrictions without a channel to release stress, or in which individuals are anxious and nervous before a complicated medical examination or treatment, or in which individuals are in a long-term hospital stay or rehabilitation process such that they may be depressed, exhausted or emotionally deprived about the persisting situation.

[0023] It has been recognized that synergetic digital cross-modal technologies may be applied to enhance multisensory support for a user for achieving a positive mood state, in particular, regarding audio-visual, olfactory and tactile interaction for synergizing several human perception channels for achieving a positive mood state transition of the user.

[0024] It has further been recognized that digital cross-modal interaction may be combinable with other technologies such as machine learning, cloud-based systems, etc.

[0025] It has thus been recognized that it is desirable to provide a system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief

[0026] Hence, some embodiments pertain to a system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the system includes: a measurement system configured to measure physiological data of the user; and a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user.

[0027] The system thus uses in some embodiments at least two different human perception channels (e.g., vision and hearing) to interact over a digital platform with the user, in particular, the system makes use of synchronized and customized (user specific) multisensory triggers providing user centric interactive contents.

[0028] In some embodiments, the measurement system includes at least one body-wearable physiological sensor to be attached to a body of the user for measuring the physiological data.

[0029] A body-wearable physiological sensor may be or may include a heartbeat sensor, a blood pressure sensor, a skin conductance sensor, a saliva stress marker sensor or the like.

[0030] In some embodiments, the measurement system includes at least one non-body-wearable sensor for measuring further physiological data.

[0031] A non-body-wearable sensor may be or may include any non-contact-based observation means, for example, a camera for detecting mimic, a microphone for detecting voice, a time-of-flight image sensor or a radar sensor (e.g., millimeter radar sensor) for detecting breathing and a breathing frequency, an infrared detector for detecting temperature, and a short-wave infrared sensor for detecting skin perspiration.

[0032] The non-body-wearable sensor may further function as a user input system for at least one of interaction with the provided cross-modal content and user feedback.

[0033] As mentioned above, the system includes a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user.

[0034] The adaption of the cross-modal content may be performed by the user via a user input system, by a medical expert or automatically by an information processing device.

[0035] The content provisioning system includes an information processing device such as a computer, a server or the like, wherein the information processing device includes circuitry configured to obtain the physiological data and to process the physiological data.

[0036] The circuitry may include one or more processors, one or more memory components, one or more data bus interfaces, one or more communications interfaces, etc.

[0037] The circuitry of the information processing device is configured to exchange data with other systems and devices via a wired or wireless connection.

[0038] Generally, the physiological data of the user may indicate a mood state of the user which may be classified as a positive mood state or a negative mood state.

[0039] In some embodiments, the mood state may be classified to be in an excitement and motivation region (a positive mood state), a tranquil and relaxed region (a positive mood state), an anxiety and distress region (a negative mood state) and a depression and deprivation region (a negative mood state) depending on the measured physiological data of the user.

[0040] It is envisaged in some embodiments to achieve a positive mood state transition from an estimated negative mood state to a positive mood state by adapting the cross-modal content to be provided to the user in accordance with the physiological data of the user.

[0041] In some embodiments, pleasant audio-visual scenarios are displayed to help users calm down and release their stress, anxiety and depression. This may be achieved by using a VR (“Virtual Reality”) HMD (“Head-Mounted Display”) or MR (“Mixed Reality”) or AR (“Augmented Reality”) via a curvature display or a projection screen to provide the users with immersive visual experience in virtual environments. For example, a pleasant and relaxing surrounding may be provided by panoramic videos or 3D (“three-dimensional”) scenarios, a realistic 3D sound for auditory solutions or an appropriate interactive experience to enhance the user’s sense of presence in virtual environments.

[0042] Hence, in some embodiments, the content provisioning system includes a visual content presentation device for providing visual content to the user, in particular, wherein the content provisioning system includes a virtual reality system which includes the visual content presentation device, wherein the virtual reality system is configured to allow the user to interact with the provided cross-modal content.

[0043] In general, the content provisioning system is configured to provide extended reality ("XR”) applications and content to the user - such as AR, MR and VR -, and wherein the content provisioning system is configured to allow the user to interact with the provided XR content, for example, via a user input system as described herein.

[0044] The visual content presentation device may be or may include a digital display device such as a wide-angle curvature screen or a head-mounted display device. The visual content presentation device may be or may include a light source arrangement which includes a plurality of light sources of one or more different colors.

[0045] The head-mounted display device may be part of a virtual reality system for providing the visual content of the virtual environment.

[0046] The virtual reality system may further include at least one of the audio output devices as described herein, the scent distribution device as described herein, and the tactile interaction device as described herein.

[0047] In such embodiments, the cross-modal content corresponds to virtual reality content.

[0048] The virtual reality system further includes a user input system configured to obtain a user input for interaction with the cross-modal content, i.e. the virtual reality content.

[0049] The user input system may include an audio input device (e.g., a microphone) for obtaining an audio input from the user for allowing the user to interact with the provided cross-modal content. For example, the audio input may include one or more voice commands from the user.

[0050] The user input system may include a visual input device (e.g., a camera) for obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content. For example, the visual input may include one or more gesture commands from the user (gesture recognition) or a tracking of the eyes (gaze tracking), a tracking of the face (face recognition or tracking) or arms of the user. The user input system may include a pointer device, a keyboard, a computer mouse, a joystick or the like.

[0051] The virtual reality system may further include the information processing device for running a virtual reality application.

[0052] The virtual reality system may allow a more interactive behavior and may enable interaction with a digital twin, wherein the virtual reality system may use, as mentioned above, eye or gaze tracking, face tracking or recognition, hand gesture recognition, voice recognition or the like for interaction with the provided cross-modal content.

[0053] This may allow establishing a real-time interaction between the user and a virtual environment such that the user may get involved into the interactive process in a more realistic, natural, interesting and immersive way, which may improve the system for the user in terms of at least one of recovery, mental health, relaxation and mental stress relief

[0054] The visual content may be image content, video content, lighting or the like.

[0055] The visual content may be real image or video content or may be computer-generated image or video content or a virtual environment or the like.

[0056] In some embodiments, the content provisioning system includes an audio output device for providing audio content to the user.

[0057] The audio output device may be headphones, a speaker, a sound system or the like and may be provided separately from the visual content presentation device or may be integrated in the visual content presentation device. The audio output device may be provided, for example, in a relaxing chair.

[0058] The audio content may be synchronized with the visual content in terms of timing and content.

[0059] In some embodiments, a scent distribution device is implemented in close user proximity to provide scent cues to enhance the user’s sense of presence, increase realism and enhance mood state. For example, a scent may contribute to attentiveness and concentration improvement in mental work or tasks, in faster stress relaxation, in faster recovery, in enhanced impressiveness and presence in VR, or specific scents may affect and may change the mood state (both valence and arousal), or a scent may affect pre-disposition to development of depression as there may be in some cases an existing mutual relationship between depression, olfaction, and quality of life.

[0060] Hence, in some embodiments, the content provisioning system includes a scent distribution device for providing scent content to the user. The scent distribution device may be provided by a multi-channel scent dispenser, for example in a relaxing chair for the user, including a plurality of different ingredients for release and mixing. The scent content may be synchronized with the visual content and the audio content in terms of timing, content (i. e. , mixture and component pattern) and intensity.

[0061] The scent distribution device may be integrated in the visual content presentation device, for example, in a head-mounted display for VR, MR or AR applications.

[0062] In some embodiments, tactile triggers and sensors are implemented in body proximity to apply at least one of informational and relaxing sensations on the body, e.g., for detecting and tracking forces for adaptive feedback, for transmitting haptic cues (e. g. vibrating or force feedback gloves) or for creating comfortable and immersive user experiences. In some embodiments, haptic feedback may be provided through low frequency tones of a provided audio content coupled to a relaxing chair (e.g., back haptic).

[0063] Hence, in some embodiments, the content provisioning system includes a tactile interaction device for providing tactile content and for detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

[0064] The tactile interaction device may be provided, for example, in a relaxing chair for the user or as a glove, a west or the like.

[0065] The tactile interaction device includes sensors for detecting user forces which may be used in a virtual environment to interact with the provided cross-modal content.

[0066] Generally, in some embodiments, the content provisioning system is further configured to provide the cross-modal content synchronized to the user.

[0067] The cross-modal content may be synchronized in terms of at least one of timing, content and intensity.

[0068] The system may be further optimized for intervention with the user by providing means for obtaining user feedback or psychological feedback (subjective questionnaires) may be shared with doctors via a network for helping to improve or adjusting therapy, rehabilitation or prevention plans.

[0069] Hence, in some embodiments, the system includes a user input system configured to obtain a user input from the user for at least one of interaction with the provided cross-modal content and user feedback. As discussed above, the user input system may include a visual input device for obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content and an audio input device obtaining an audio input from the user for allowing the user to interact with the provided cross-modal content.

[0070] As discussed above, the user input system may be part of a virtual reality system for interaction with the provided virtual reality content.

[0071] In some embodiments, the same user input system is used for allowing the user to provide user feedback. The user feedback may be processed by the information processing device of the content provisioning system.

[0072] In some embodiments, the user input system includes a first subsystem for obtaining a user input for interaction with the provided cross-modal content and a second subsystem for obtaining a user input for user feedback.

[0073] The user feedback may be used to re-train an artificial intelligence model for user-specific optimization of the selection of the cross-modal content to be provided to the user.

[0074] The user input system may further be used to obtain a user input for selecting the cross-modal content.

[0075] Hence, personalized multi-sensory rehabilitation settings may be achieved with user-specific combinations of the digital cross-modal content.

[0076] As mentioned above, it is envisaged in some embodiments to achieve a positive mood state transition from a negative mood state to a positive mood state by adapting the cross-modal content to be provided to the user in accordance with the physiological data of the user.

[0077] Thus, in some embodiments, the content provisioning system is further configured to input the physiological data into an artificial intelligence which is trained to select the synchronized cross- modal content to be provided to the user for achieving a positive mood state transition of the user.

[0078] Generally, each piece of content of each type of content (visual, audio, scent, tactile) may have a different mood effect on a user.

[0079] For example, a lighting or, generally, visual content may be calming or energizing by selecting lighting spectrum, color, temperature and dynamic pattern.

[0080] For example, a scent content may be relaxing or invigorating by selecting a specific ingredient or a combination of ingredients. For example, a tactile interaction may be selected to simulate a massage, or a movement correlated to the visual content, audio content or scent content.

[0081] It is thus envisaged that the content provisioning system selects various stimuli patterns and stimuli combinations automatically according to the current physiological data and the desired positive mood state transition.

[0082] In some embodiments, each piece of content of each type of content (visual, audio, scent, tactile) may be stored in a database (for scent may be an indicator for the mixture) together with a unique indicator identifying the respective piece of content.

[0083] The content provisioning system may thus select various stimuli patterns and stimuli combinations automatically according to the desired mood state transition.

[0084] The artificial intelligence may be provided by an artificial neural network, which may include one or more fully-connected layers, one or more convolutional layers, one or more recurrent layers or the like or a combination thereof.

[0085] The artificial intelligence is pre-trained on a large database of the content provisioning system to learn the patterns between physiological data and the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

[0086] The training of the artificial intelligence may be performed, for example, as follows:

[0087] At first, a plurality of different combinations of each piece of content of each type of content (visual, audio, scent, tactile) in the database may be provided to a plurality of different users.

[0088] The physiological data of each user for each combination of cross-modal content are obtained before providing the respective cross-modal content combination and after providing the respective cross-modal content combination.

[0089] A medical expert labels, based on the physiological data before and after providing the respective cross-modal content combination, each cross-modal content combination where a positive mood state transition has been achieved or a positive mood state has been maintained.

[0090] Moreover, the medical expert labels each cross-modal content combination where a negative mood state transition has been achieved or a negative mood state has been maintained.

[0091] The artificial intelligence is then trained by inputting the physiological data measured before providing a cross-modal content combination, wherein the artificial intelligence then outputs either a set of content indicators indicating a particular cross-modal content combination likely to achieve a positive mood state transition or maintain a positive mood state or outputs a zero if no cross-modal content is likely to achieve a positive mood state transition.

[0092] The output of the artificial intelligence may be improved by connecting the concrete physiological data of the user with the previously provided synchronized cross-modal content.

[0093] Hence, in some embodiments, the content provisioning system is further configured to further input the previous synchronized cross-modal content into the artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user further based on the previous synchronized cross-modal content.

[0094] For example, the artificial intelligence may recognize images in the visual content or may recognize specific tone sequences in an audio piece or the like and learns the connection with the physiological data of the user.

[0095] Generally, the system may be applied to two different scopes, wherein the first scope is treatment and rehabilitation support, and the second scope is relief and prevention of mental stress.

[0096] Scope 1:

[0097] The system may be used at a hospital (with strict regulation) or at an ambulatory service or by a home healthcare or rehabilitation service.

[0098] The patients typically undergo a treatment in the hospital or need long-term rehabilitation.

[0099] The system may be used to solve distress, anxiety, fear, feeling of uncertainty about progress of treatment, where the patient is depressed by the current situation, and it is not possible to adapt to normal environment again after being under heavy long-term treatment.

[0100] The system may be used for generating pleasant, relaxing or motivating environments for patients such that, in some cases, less medication may be needed for treatment, a shorter recovery duration may be achieved, the patients assess and visualize the rehabilitation progress positively, motivation is generated (e.g., due to intrinsic, interactive tasks, rewards, concentration, reminders, etc.).

[0101] Scope 2:

[0102] The system may be used at the workplace, at home or in wellness real estates, spa or the like.

[0103] The people may be prone to high mental pressure, fatigue or burnout due to demanding work or private issues or are otherwise emotionally deprived. The system may be used to solve fatigue, sleep disorder, nervousness, low mood, demotivation, feeling of being deenergized and tend to develop depressive state.

[0104] The system may be used for generating relaxing and recovering environments in some cases, for example, for faster mental relaxation and recovery, for mood regulation without medication (e.g., help to energize with blue light and scent or become calm with red light and scent) or for reducing or preventing consumption of non-prescription medication.

[0105] Generally, the contents may be available through a streaming library.

[0106] An artificial intelligence may be implemented to provide individual recommendations on visual content, auditory cues (audio content) and matching scent contents and tactile content.

[0107] Depending on the scope of application, different modalities may be used:

[0108] For scope 1 : The selection of contents (visual, audio, scent, tactile) may be done by a medical expert and prescribed to the user, wherein the medical expert typically tracks the success of intervention and adjusts the content proposal (adapts the cross-modal content provided to the user). The selection of contents may be done automatically by an artificial intelligence.

[0109] For scope 2: The user may be able to select and combine contents individually. The selection of contents may be done automatically by an artificial intelligence.

[0110] The simultaneous addressing of several human perception channels (senses) may enhance the feeling of presence in the proposed environment and may, in some cases, provide for faster mental recovery, stress relaxation, and reduce anxiety and fatigue.

[0111] The system may allow a high level of customization by providing individually curated audiovisual, olfactory and tactile stimuli due to easier adaptability of the different trigger in virtual world.

[0112] The system may allow for real-time monitoring of the mood state and vital state of the user by collecting physiological data of the user.

[0113] The system may provide a possibility to combine with other technologies such as machine learning, cloud-based systems, etc.

[0114] Some embodiments pertain to a (corresponding) method for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the method includes: measuring physiological data of the user; and providing cross-modal content to the user and adapting the provided cross-modal content in accordance with the physiological data of the user.

[0115] The method may be performed by the system as described herein.

[0116] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0117] Returning to Fig. 1, an embodiment of a system 1 for digital cross-modal interaction with a user 4 for at least one of recovery, mental health, relaxation and mental stress relief is discussed in the following under reference of Fig. 1, which schematically illustrates the embodiment in a block diagram.

[0118] The system 1 includes a measurement system configured to measure physiological data of the user 4, wherein the measurement system includes body-wearable physiological sensors 2 and non-body -wearable sensors 3.

[0119] The body-wearable physiological sensors 2 include, for example, a heartbeat sensor, a blood pressure sensor and a skin conductance sensor.

[0120] The non-body-wearable sensors 3 include, for example, a camera and a microphone.

[0121] The non-body-wearable sensors 3 further function as a user input system for at least one of interaction with provided cross-modal content and user feedback. The non-body -wearable sensors 3 are thus shared, for example, with a virtual reality system.

[0122] The system 1 further includes a relaxation chair 5 for user 4, for example, the relaxation chair 5 may be provided at home or at the workplace or in a hospital or at an ambulatory care unit or in wellness real estates.

[0123] The system 1 further includes a content provisioning system configured to provide cross-modal content to the user 4 and to adapt the provided cross-modal content in accordance with the physiological data of the user 4.

[0124] The content provisioning system includes either a wide-angle screen 6a (or projection screen in other embodiments) for providing visual content to the user 4 or a virtual reality device 6b for providing the visual content to the user 4. The virtual reality device 6b is part of a virtual reality system which further includes the nonbody-wearable sensors 3 for allowing the user 4 to interact with the provided visual content. In particular, the wide-angle screen 6a or the virtual reality device 6b (including a head-mounted display) provides the user 4 with immersive visual experience in virtual environments.

[0125] The content provisioning system further includes an audio output device 7a for providing audio content to the user 4, wherein the audio output device 7a are a pair of separate speakers in this embodiment. Alternatively, the content provisioning system further includes an audio output device 7b for providing audio content to the user 4, wherein the audio output device 7b are headphones, wherein the headphones may be included in the virtual reality device 6b.

[0126] The audio output device 7a or 7b provides pleasant audio content synchronized with the visual content for helping the user 4 to calm down and release stress, anxiety and depression.

[0127] The content provisioning system further includes a scent distribution device 8a for providing scent content to the user 4, wherein the scent distribution device 8a is a separate device. Alternatively, the content provisioning system further includes a scent distribution device 8b for providing scent content to the user 4, wherein the scent distribution device 8b is integrated in the relaxation chair 5 or in the virtual reality device 6b. Generally, the scent distribution device 8b may be integrated in any other device, in particular, in any other wearable device.

[0128] The scent distribution device 8a or 8b provides corresponding scents to the user 4 for enhancing a sense of presence, enhance mood and increase realism.

[0129] The content provisioning system further includes a tactile interaction device 9 for providing tactile content and for detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

[0130] The tactile interaction device 9 is shown as attached to the user 4, for example, as a glove or west.

[0131] The tactile interaction device 9 is further provided in the relaxation chair 5 (e.g., in back, seat or arm rests) to apply informational or relaxing sensations on the body of the user 4.

[0132] The audio output device 7a or 7b, the scent distribution device 8a or 8b and the tactile interaction device 9 may further be part of the virtual reality system.

[0133] The content provisioning system further includes an information processing device 10 which may be a computer, a server or the like. The information processing device 10 is connected in a wired or wireless manner with the wide- angle screen 6a or virtual reality device 6b, the audio output device 7a or 7b, the scent distribution device 8a or 8b and the tactile interaction device 9.

[0134] The information processing device 10 is further connected in a wired or wireless manner with the measurement system including the body-wearable physiological sensors 2 and the non-bodywearable sensors 3 to obtain the physiological data or the user input indicating an interaction with the provided cross-modal content or user feedback.

[0135] The information processing device 10 controls the other devices of the content provisioning system to provide cross-modal content to the user 4 in accordance with a selected cross-modal content, wherein the cross-modal content is selected based on the obtained physiological data of the user 4, thereby adapting the provided cross-modal content in accordance with the physiological data of the user 4.

[0136] The information processing device 10 stores the content in a database. The database may be updated by remembering the user preferences and choices at specific conditions or when fulfilling specific tasks. The user 4 may upload own content to the database (e.g., video clips taken in pleasant surroundings, during vacation, during memoizable experiences or the like).

[0137] The information processing device 10 may transmit the physiological data to a screen 11 of a medical expert 12.

[0138] The medical expert may then select the cross-modal content according to the physiological data of the user 4 and may transmit the decision to the information processing device 10 which then controls the other devices of the content provisioning system to provide cross-modal content to the user 4 in accordance with a selected cross-modal content.

[0139] In other embodiments, the information processing device 10 runs an artificial intelligence to which it inputs the physiological data of the user 4, wherein the artificial intelligence is trained to select the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

[0140] Fig. 2 schematically illustrates an embodiment of different mood state regions 15 of the user 4, which is discussed in the following under reference of Fig. 1 and Fig. 2.

[0141] Depending on a valence and arousal indicated by the physiological data, the user 4 may be in a mood state corresponding to an excitement and motivation region 15a (a positive mood state), a tranquil and relaxed region 15d (a positive mood state), an anxiety and distress region 15b (a negative mood state) and a depression and deprivation region 15c (a negative mood state).

[0142] The valence is negative on the left side of the arousal axis. The arousal increases from bottom to top.

[0143] Fig. 3 schematically illustrates in a flow diagram an embodiment of method 20 for digital cross- modal interaction with the user 4 for at least one of recovery, mental health, relaxation and mental stress relief, which is discussed in the following of Fig. 1 and Fig. 3.

[0144] The method 20 is performed by the system 1 of Fig. 1.

[0145] At 21, physiological data is measured by the body-wearable sensors 2.

[0146] The body-wearable sensors 2 transmit the physiological data to the information processing device 10.

[0147] At 22, the information processing device 10 obtains the physiological data of the user 4 and adapts the provided cross-modal content in accordance with the physiological data.

[0148] At 23, the information processing device 10 waits a predetermined amount of time before requesting further physiological data to allow the user 4 to immerse into the provided cross- modal content.

[0149] Fig. 4 schematically illustrates in a flow diagram an embodiment of a method 22 for adapting cross-modal content to be provided to the user 4, which is discussed in the following.

[0150] At 30, the information processing device 10 transmits the physiological data to the screen 11 of the medical expert 12.

[0151] At 31, the medical expert 12 selects the cross-modal content from the database of the information processing device 10.

[0152] Then, at 32, the information processing device 10 controls the other devices of the content provisioning system to provide the selected cross-modal content to the user 4.

[0153] Fig. 5 schematically illustrates in a flow diagram an embodiment of a method 22 for adapting cross-modal content to be provided to a user.

[0154] At 40, the information processing device 10 inputs the physiological data into an artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user 4 for achieving a positive mood transition of the user 4. Moreover, the information processing device 10 inputs the previous synchronized cross-modal content into the artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user further based on the previous synchronized cross-modal content.

[0155] The artificial intelligence outputs the selected synchronized cross-modal content to be provided to the user 4.

[0156] At 41, the information processing device 10 selects the suggested cross-modal content selected by the artificial intelligence from the database.

[0157] At 42, the information processing device 10 controls the other devices of the content provisioning system to provide the selected cross-modal content to the user 4.

[0158] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0159] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0160] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0161] Note that the present technology can also be configured as described below.

[0162] (1) A system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the system includes: a measurement system configured to measure physiological data of the user; and a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user.

[0163] (2) The system of (1), wherein the content provisioning system includes a visual content presentation device for providing visual content to the user, in particular, wherein the content provisioning system includes a visual input device for obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content, in particular, wherein the content provisioning system includes a virtual reality system which includes the visual content presentation device, wherein the virtual reality system is configured to allow the user to interact with the provided cross-modal content.

[0164] (3) The system of (1) or (2), wherein the content provisioning system includes an audio output device for providing audio content to the user, in particular, wherein the content provisioning system includes an audio input device for obtaining an audio input from the user to allow the user to interact with the provided cross-modal content.

[0165] (4) The system of anyone of (1) to (3), wherein the content provisioning system includes a scent distribution device for providing scent content to the user.

[0166] (5) The system of anyone of (1) to (4), wherein the content provisioning system includes a tactile interaction device for providing tactile content and for detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

[0167] (6) The system of anyone of (1), wherein the content provisioning system is further configured to provide the cross-modal content synchronized to the user.

[0168] (7) The system of (6), wherein the content provisioning system is further configured to input the physiological data into an artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

[0169] (8) The system of anyone of (1) to (7), wherein the system further includes a user input system configured to obtain a user input from the user for at least one of interaction with the provided cross-modal content and user feedback.

[0170] (9) The system of anyone of (1) to (8), wherein the measurement system includes at least one body -wearable physiological sensor to be attached to a body of the user for measuring the physiological data.

[0171] (10) The system of (9), wherein the measurement system further includes at least one nonbody-wearable sensor for measuring further physiological data.

[0172] (11) A method for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the method comprises: measuring physiological data of the user; and providing cross-modal content to the user and adapting the provided cross-modal content in accordance with the physiological data of the user. (12) The method of (11), wherein providing the cross-modal content to the user includes providing visual content and audio content to the user, in particular, wherein the method further comprises obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content, in particular, further comprising allowing the user to interact with the provided cross-modal content via a virtual reality system.

[0173] (13) The method of (11) or (12), wherein providing the cross-modal content to the user includes providing audio content to the user, in particular, wherein the method further comprises obtaining an audio input from the user for allowing the user to interact with the provided cross- modal content.

[0174] (14) The method of anyone of (11) to (13), wherein providing the cross-modal content to the user includes providing scent content.

[0175] (15) The method of anyone of (11) to (14), wherein providing the cross-modal content to the user includes providing tactile content, and wherein the method further comprises detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

[0176] (16) The method of anyone of (11) to (15), wherein the cross-modal content is provided synchronized to the user.

[0177] (17) The method of (16), further including inputting the physiological data into an artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

[0178] (18) The method of anyone of (11) to (17), further including obtaining a user input from the user for at least one of interaction with the provided cross-modal content and user.

[0179] (19) The method of anyone of (11) to (18), wherein the physiological data is measured with at least one body -wearable physiological sensor to be attached to a body of the user.

[0180] (20) The method of anyone of (11) to (19), wherein further physiological data is measured with at least one non-body -wearable sensor.

[0181] (21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer.

[0182] (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.

Claims

CLAIMS1. A system for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the system comprises: a measurement system configured to measure physiological data of the user; and a content provisioning system configured to provide cross-modal content to the user and to adapt the provided cross-modal content in accordance with the physiological data of the user.

2. The system of claim 1, wherein the content provisioning system includes a visual content presentation device for providing visual content to the user, in particular, wherein the content provisioning system includes a visual input device for obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content, in particular, wherein the content provisioning system includes a virtual reality system which includes the visual content presentation device, wherein the virtual reality system is configured to allow the user to interact with the provided cross-modal content.

3. The system of claim 1, wherein the content provisioning system includes an audio output device for providing audio content to the user, in particular, wherein the content provisioning system includes an audio input device for obtaining an audio input from the user to allow the user to interact with the provided cross-modal content.

4. The system of claim 1, wherein the content provisioning system includes a scent distribution device for providing scent content to the user.

5. The system of claim 1, wherein the content provisioning system includes a tactile interaction device for providing tactile content and for detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

6. The system of claim 1, wherein the content provisioning system is further configured to provide the cross-modal content synchronized to the user.

7. The system of claim 6, wherein the content provisioning system is further configured to input the physiological data into an artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

8. The system of claim 1, further comprising a user input system configured to obtain a user input from the user for at least one of interaction with the provided cross-modal content and user feedback.

9. The system of claim 1, wherein the measurement system includes at least one bodywearable physiological sensor to be attached to a body of the user for measuring the physiological data.

10. The system of claim 9, wherein the measurement system further includes at least one non-body -wearable sensor for measuring further physiological data.

11. A method for digital cross-modal interaction with a user for at least one of recovery, mental health, relaxation and mental stress relief, wherein the method comprises: measuring physiological data of the user; and providing cross-modal content to the user and adapting the provided cross-modal content in accordance with the physiological data of the user.

12. The method of claim 11, wherein providing the cross-modal content to the user includes providing visual content and audio content to the user, in particular, wherein the method further includes obtaining a visual input from the user for allowing the user to interact with the provided cross-modal content, in particular, further comprising allowing the user to interact with the provided cross-modal content via a virtual reality system.

13. The method of claim 11, wherein providing the cross-modal content to the user includes providing audio content to the user, in particular, wherein the method further includes obtaining an audio input from the user for allowing the user to interact with the provided cross-modal content.

14. The method of claim 11, wherein providing the cross-modal content to the user includes providing scent content.

15. The method of claim 11, wherein providing the cross-modal content to the user includes providing tactile content, and wherein the method further comprises detecting a tactile input from the user for allowing the user to interact with the provided cross-modal content.

16. The method of claim 11, wherein the cross-modal content is provided synchronized to the user.

17. The method of claim 16, further comprising inputting the physiological data into an artificial intelligence which is trained to select the synchronized cross-modal content to be provided to the user for achieving a positive mood state transition of the user.

18. The method of claim 11, further comprising obtaining a user input from the user for at least one of interaction with the provided cross-modal content and user feedback.

19. The method of claim 11, wherein the physiological data is measured with at least one body -wearable physiological sensor to be attached to a body of the user.

20. The method of claim 11, wherein further physiological data is measured with at least one non-body-wearable sensor.

Images