Method and arrangement to recognize emotional state and support mood

Abstract

The subject of the invention is the recognition of a user's emotional state and supporting their mood, particularly when experiencing loneliness. A specific goal is to help the user obtain sufficient interaction with relatives and friends when their mood is slipping into a concerning area. The objective is also that the connection is automatically opened to exactly the right person who can best help the user return to a safe and healthy mood and mental state. As an alternative to making contact, for example, when a suitable contact person is not reachable, providing the user with music, games, podcasts, videos, or other mood interventions appropriate to the emotional state is also targeted. These objectives are achieved through a system built into a smartwatch (101) and smartphone (121) that measures, recognizes, and monitors the user's vitality (107) and emotional state (127) throughout the day, and opens a communication link (140) or a mood intervention suitable for the situation when set conditions are met.

Description

[0001] Method and arrangement to recognize emotional state and support mood

[0002] The invention relates to the recognition and support of an individual's emotional state through various interventions, particularly when experiencing loneliness. A specific objective is to assist in facilitating adequate interaction with family and friends. For elderly and those suffering from health issues, the invention also aims to enhance communication with caregivers. The arrangement according to the invention is also suitable for younger users, where the need may arise from health reasons, living environment, life situation, or an interest in monitoring and managing their emotional well-being. The term 'user' encompasses children, adolescents, youth, adults, and the elderly.

[0003] Loneliness is a vast, global issue that is not alleviated but rather exacerbated by rapidly spreading social media and online services. As of 2024, there are five billion social media users worldwide, of which 20-40% are estimated to suffer from loneliness. The withdrawal from face-to- face contacts with others has detrimentally altered living environments and led to a decline in mental and social well-being.

[0004] The increasing use of social media services has brought with it several specific problems, not only for the elderly but also particularly for young people and young adults. While these platforms offer significant advantages, such as improved communication and access to information, they also pose unique challenges that can impact mental health, social skills, and overall wellbeing. Paradoxically, the more time spent online, the less time is available for face-to-face interactions, which are crucial for developing deep and meaningful relationships.

[0005] Social media platforms are filled with opportunities for comparison, as users often present idealized versions of their lives. This can lead to feelings of inadequacy, decreased self-esteem, and in some cases, body image issues among youth and young adults. The anonymity and distance provided by social media can lead to an increase in bullying and harassment. Young people are particularly vulnerable to cyberbullying, which can have serious consequences for their mental health and well-being.

[0006] Social media platforms are designed to be addictive and can lead to dependency. Young individuals may find themselves compulsively checking their phones and social media accounts, which can disrupt daily life, academic performance, and personal relationships. Young users are not always aware of the privacy implications of their online activities. Excessive sharing of personal information can lead to privacy violations and potentially dangerous situations if such information falls into the wrong hands.

[0007] Short videos (shorts) are a particular social media service that poses problems for children, young people, and also adults. Typically lasting from a few seconds to a couple of minutes, regular exposure to such short content may condition the brain to expect constant stimulation and immediate gratification. Their rapid pace and high content volume can also lead to hyperactivity. Children may find it difficult to detach from them or may feel restless and anxious when not consuming media. This hyperactivity can affect the ability to engage in deep, reflective thinking, impacting learning and creativity. Many children use electronic devices to watch short videos before bedtime. The blue light from screens can disrupt the production of melatonin, the hormone that regulates sleep. Disturbed sleep can affect cognitive functions, mood, and overall health, impairing the brain's ability to learn and retain information. Additionally, short videos are linked to emotional states caused by loneliness, as they attempt to redirect thoughts and numb the mind to cope with loneliness.

[0008] There is a growing number of research papers linking excessive use of social media to mental health issues, including anxiety, depression, and suicidal thoughts. The constant pressure for connectivity and the flood of information can be overwhelming and detrimental to both the brain and mental health. Furthermore, the passive nature of social media use can lead to a decrease in physical activity among all age groups, potentially leading to health issues such as obesity and related diseases. Excessive use of social media can impair or even prevent the development of essential life skills, such as social skills, problem-solving abilities, and emotional regulation. This can be particularly damaging for children, whose brains are undergoing various sensitive periods and rapid developmental phases. Real interactions are crucial for the development of these skills, and excessive digital communication can hinder the development of brain functions and cognitive abilities.

[0009] Loneliness can be particularly acute among young people. Young adults and teenagers often report higher feelings of loneliness or isolation. Among middle-aged individuals, the experience of loneliness can vary widely and may be shaped by life events such as divorce or the pressures of balancing work and family life. Among the elderly, particularly those who live alone, have lost a spouse, or whose mobility is limited, are susceptible to problems caused by loneliness. Health problems stemming from loneliness can include cardiovascular diseases, high blood pressure, obesity, a weakened immune system leading to increased susceptibility to infections, poor sleep quality, and diminished physical activity. Mental health issues can range from depression, anxiety, stress, cognitive decline, and, in the elderly, an increased risk of dementia. Among older individuals, loneliness is also linked to a higher risk of developing Alzheimer's disease and other forms of dementia due to elevated levels of stress and stress hormones such as cortisol, which can damage brain cells and increase inflammation. Additionally, loneliness can lead to feelings of hopelessness and helplessness, which can further exacerbate mental health problems.

[0010] Loneliness is a problem for many who live alone or are in problematic relationships. Maintaining connections with family and friends may gradually diminish, and as depression sets in, the threshold for reaching out only increases. Loneliness or a problematic relationship can also lead to uncontrolled mood swings and overreactions, potentially endangering one's own or a loved one's health or property. Making contact in an agitated state of mind is not always easy, as it can be difficult to decide who to call, who would understand, or who would have the time to listen. Making phone calls can become overwhelmingly difficult and be postponed from day to day. While social media services readily offer chat partners or gaming companions, their quality and reliability are questionable. Contacting caregivers and health services may also be hindered by a high barrier. As mood declines towards depression or becomes uncontrollably agitated, or as mental health otherwise becomes unstable, it is difficult to self-assess the need for help. Therefore, help often comes too late.

[0011] The mood of someone living alone varies for many reasons and in many ways depending on the activities of the day. Browsing news, message threads, and social media updates or engaging in addictive games can elicit a variety of emotions: joy and excitement, sadness and distress, worry and fear, even shame. An active imagination may further intensify these extreme emotional states. Calling a family member or friend could help to stabilize emotions and prevent them from becoming too frightening or depressive, but even the thought of making contact can be distressing. How could the threshold for making contact or taking other actions be lowered? Or where could someone suffering from depression due to social media addiction find support to improve their mood?

[0012] The mood of someone living in a problematic relationship varies for many reasons and in many ways depending on the events of the day. Mood fluctuations may be similar to those experienced by a person living alone if the relationship is diluted and does not provide support to a darkening mood. On the other hand, the mood may receive impulses towards either depression or agitation if there are disagreements in the relationship that are not properly addressed or managed. How could the threshold for making contact or taking other actions be lowered in such situations as well?

[0013] The recognition of emotional states using technical solutions has been disclosed, for example, in patent application US20220223064A1, where the user's emotional state and reactions are analyzed and the user aided in recognizing their moods and reactions, thereby reducing stress and improving emotional well-being.

[0014] In the publication "Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems" by Deger Ayata et al., published in the Journal of Medical and Biological Engineering (2020) 40: 149-157, there is a review of emotion recognition using wearable technology. The article focuses on monitoring the emotional and health states of elderly individuals living alone. According to the article, emotions can be categorized dimensionally, where two important dimensions are Valence and Arousal.

[0015] In the patent US 11545173B2, mood analysis using mobile technology is presented by recording and analyzing an audio sample with a trained machine learning model and determining the user's mood based on one or more emotional values.

[0016] The use of wearable technology in the recognition of emotional states is also examined in the articles:

[0017] - ’’Emotion Recognition Using Wearables: A Systematic Literature Review - Work-in-progress” by Stanislaw Saganowski et al.,

[0018] - ’’Emotion-Recognition Using Smart Watch Sensor Data: Mixed Design Study” by Juan C Quiroz,

[0019] - ’’Wearable Emotion Recognition Using Heart Rate Data from a Smart Bracelet” by Lin Shu et al.,

[0020] - ’’Predicting Emotion with Biosignals: A Comparison of Classification and Regression Models for Estimating Valence and Arousal Level Using Wearable Sensors” by Pekka Siirtola et al., Sensors 2023, - ’’Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables” by Stanislaw Saganowski et al., Scientific Data 9, 2022.

[0021] Valence, Arousal, and Dominance are components of the so-called VAD model, which is used in the description and analysis of emotions. According to this model, emotions can be described along the following three dimensions:

[0022] - Valence (V, pleasantness): This dimension describes the pleasantness or unpleasantness of an emotion. Positive Valence refers to pleasant feelings, such as happiness, joy, or satisfaction. Negative Valence refers to unpleasant feelings, such as sadness, shame, anger, or fear. Some researchers use the Pleasure dimension instead of Valence, referring to it as the PAD model.

[0023] - Arousal (A, excitement): This dimension describes the intensity or level of activation of an emotion. High Arousal indicates a strong emotional reaction and physical vitality, whether the emotion is positive or negative. Low Arousal, on the other hand, refers to calmness, relaxation, or indifference.

[0024] - Dominance (D, sense of control): This dimension describes an individual's perception of their control over an emotion or situation. High Dominance indicates a feeling of controlling one's emotions or the situation, whereas low Dominance indicates a feeling of being controlled by the situation or emotion.

[0025] These three dimensions enable a more comprehensive and precise description of emotions than simply categorizing them as positive and negative. For instance, both anger and fear can be intense (high Arousal) negative (negative Valence) emotions, but the difference between them is evident in Dominance: anger typically involves higher Dominance (the individual wants to control or change the situation), whereas fear is associated with lower Dominance (the individual feels controlled by the situation or external factors).

[0026] The dimension of emotional state can also include Vitality, or level of vitality, which is easier to measure with wearable technology than the aforementioned VAD dimensions. It describes an individual's experience of energy or vitality and can be measured in various levels of vitality. It indicates how energetic (vital) or exhausted (powerless) an individual feels. This dimension can be associated with both positive and negative emotions. For example, an individual may experience a high level of vitality when they are excited or happy, but also when they are angry or anxious. A low level of vitality may be related to apathy or depression. There is a strong, direct correlation between Vitality and Arousal, so Arousal can be assessed by measuring vitality with wearable sensors, which is particularly utilized in this invention for assessing emotional states.

[0027] On the other hand, measuring Valence, the pleasantness or unpleasantness of an emotional state, with wearable sensors is challenging because it is a subjective emotional experience that can vary greatly from one individual to another. Similarly, measuring Dominance, the feeling of control or domination, with wearable sensors is also challenging. In addition to or instead of wearable technology, microphones and the analysis of voice tone and content, or cameras and the analysis of facial expressions and gestures using image recognition technologies, can be used to measure Valence and Dominance.

[0028] The analysis of speech and the recognition of emotions from voice is a field that methods of machine learning and artificial intelligence have significantly advanced. Many companies and applications focus on this area. Here are some examples:

[0029] - Beyond Verbal has developed technology that analyzes voice based on tone, pitch, rate of change, and other vocal characteristics to recognize various emotions.

[0030] - Affectiva, known for its facial expression recognition technology, has also expanded into voice recognition. Their technology aims to detect emotions and social signals from human voice.

[0031] - Cogito has developed software that analyzes voice and recognizes emotions, helping to improve customer service and reduce customer churn.

[0032] - Vokaturi has developed an open-source SDK (Software Development Kit) that recognizes five basic emotions: happiness, sadness, anger, fear, and neutral by analyzing vocal spectral and prosodic features. These emotional assessments are based on complex modeling developed from extensive audio data. The model is trained with voice samples from various cultures and languages to make it as universal as possible. The modeling process includes multiple machine learning methods, including time-series analysis, which recognizes voice in sequences.

[0033] To determine Valence from speech, complex methods of machine learning and signal processing can be used. Here are some general guidelines related to recognizing Valence from speech:

[0034] - Pitch: A higher pitch usually indicates positive Valence, while a lower pitch usually indicates negative Valence. - Intensity: A louder voice may indicate a more intense (high Arousal) emotional state, whose Valence could be either positive or negative.

[0035] - Prosody: The rhythm, emphasis, and tone of speech can provide clues about Valence. For example, fast, loud, and rhythmic speech might indicate positive Valence, while slow, low, and monotone speech might indicate negative Valence.

[0036] - Speech Content: Word choices and linguistic structures can also reflect Valence. Positive words or expressions generally indicate positive Valence, whereas negative words or expressions usually indicate negative Valence.

[0037] The learning capability of the recognition algorithm is crucial because people express and experience emotions differently, and cultural, social, and individual differences can affect how Valence is manifested in speech.

[0038] Recognizing Dominance, or the sense of control in an emotional state from speech, also requires signal processing and machine learning methods. Here are some general guidelines related to recognizing Dominance from speech:

[0039] - Pitch: A lower pitch might indicate greater Dominance, as a deep voice is often associated with authority and self-assurance. Conversely, a high pitch might indicate lesser Dominance.

[0040] - Intensity: A louder voice can indicate greater Dominance, as it can be interpreted as a sign of confidence and power.

[0041] - Prosody: A fast speech rate and clear emphasis can suggest greater Dominance, as these can convey confidence and decisiveness.

[0042] - Speech Content: Dominance may also be reflected in word choices and linguistic structures. For example, imperative sentences or strong expressions can indicate greater Dominance.

[0043] There are many mobile applications for recognizing facial expressions, often using artificial intelligence and machine learning to detect and analyze human facial expressions and gestures, particularly microexpressions and microgestures. Here are some examples of applications suitable for emotion recognition:

[0044] - Affectiva specializes in recognizing emotions based on facial expressions and voice tone. Their SDK (Software Development Kit) can be used in many different applications. - Microsoft's Azure Face API cloud service offers a facial recognition API that includes a facial expression recognition feature. This API can be integrated into any application.

[0045] - Noldus's FaceReader is an advanced software for analyzing facial expressions. It can analyze multiple universal expressions and other facial movements.

[0046] - Kairos provides various facial recognition services, including expression recognition.

[0047] - Apple's Emotient has developed technology that can analyze microexpressions in faces, enabling emotion recognition.

[0048] Determining Valence from facial expressions often involves face recognition technologies and facial expression analysis techniques. These technologies use artificial intelligence and machine learning to recognize different parts of the face, such as the eyes, eyebrows, nose, and mouth, and to analyze their movements and positions. Common principles used in determining Valence include:

[0049] - Joy and Happiness: In joyful or happy facial expressions, the mouth usually curves upward (smile), the eyes may crinkle slightly at the corners, and the cheeks may rise.

[0050] - Sadness: In sad facial expressions, the corners of the mouth may curve downward, the eyes may appear "heavier" or directed downward, and the eyebrows may arch slightly upward.

[0051] - Anger: In angry facial expressions, the eyebrows often furrow, and the mouth may be tightly closed or slightly open.

[0052] - Fear: In fearful facial expressions, the eyes are often larger or open, the eyebrows are raised, and the mouth may be open.

[0053] Each individual's facial expressions are unique, and cultural differences can affect how emotions are expressed and interpreted. Additionally, a person's internal emotional experience may not always be directly visible in facial expressions. For example, a person might smile even while experiencing a negative Valence internally. Training the system at the individual level can more easily accommodate personal deviations from general expression usage. A system that determines Valence generally from facial expressions needs to be carefully trained with a large and diverse dataset that accounts for these individual and cultural differences. Additionally, it should be flexible and consider contextual factors that can affect how emotions are expressed and interpreted. Determining Dominance, or the sense of control from facial expressions, is challenging, but there are indications that Dominance may be associated with certain facial expressions or features. For example, strong or intense facial expressions, such as a stern look or tightly pursed lips, which can be recognized by artificial intelligence, may indicate higher Dominance. Additionally, the direction and intensity of gaze can influence the interpretation of Dominance — a direct and intense gaze may convey stronger Dominance than an avoiding gaze.

[0054] Artificial intelligence can be trained to recognize emotions directly from facial images or voice without defining the previously described VAD or other dimensions. Recognition can use measurements from wearable technology in addition to facial images, voice, and / or responses to questions to determine emotional state using a discrete model, i.e., by categorizing them into different categories. In the categorical classification of emotional states, emotions are named or numbered and divided into distinct, clearly defined feelings. This approach is derived from traditional emotion theory, which emphasizes the universality of certain basic emotions.

[0055] Commonly accepted basic emotions are:

[0056] - Joy: A positive emotion associated with pleasure, satisfaction, or happiness.

[0057] - Sadness: A negative emotion associated with loss, disappointment, or despair.

[0058] - Anger: A negative emotion associated with insult, injustice, or threat.

[0059] - Fear: A negative emotion associated with the perception of danger or threat.

[0060] - Disgust: A negative emotion associated with unwillingness or repulsion to encounter something or someone.

[0061] - Surprise: A neutral, positive, or negative emotion associated with encountering an unexpected event or information.

[0062] In addition to these basic emotions, there are many other emotional states that can be seen as derivatives or combinations of basic emotions. In one study ("Emoji fans take heart: Scientists pinpoint 27 states of emotion" by Yasmin Anwar, Berkeley News, September 6, 2017), 27 different emotional states were recognized: admiration, adoration, aesthetic appreciation, amusement, anger, anxiety, awe, embarrassment, boredom, calmness, confusion, craving, disgust, empathic pain, entrancement, excitement, fear, horror, interestjoy, nostalgia, relief, romance, sadness, satisfaction, sexual desire, and surprise. In a mobile application (How we feel), which is based on research from YUEI (Yale University Emotional Intelligence), as many as 144 different emotional states have been defined, which are provisionally placed in a quadrant formed by the Valence- Arousal axes.

[0063] Emotional state can also be mapped subjectively using questions, for example, using Likert- scale questions that ask the user to rate their prevailing emotional state as a combination of different emotional options. Questions can use categorized emotional states or they can be directed to dimensions of emotional states such as Valence (e.g., on an unpleasantness-pleasantness scale of -5 to +5), Arousal (e.g., on an intensity scale of 1 to 7), or Dominance (e.g., on a sense of control scale of 1 to 7).

[0064] Here are some examples of questions used to map emotional state:

[0065] - Assessment of prevailing emotional state: "How strongly do you feel the following emotions right now: joy, sadness, fear, anger, disgust, surprise (e.g., on a scale of 1 = not at all, 7 = very strongly)."

[0066] - Valence (pleasantness): "Rate the pleasantness of your prevailing emotional state on a scale of -5 to +5, where -5 means very unpleasant and +5 very pleasant."

[0067] - Arousal (intensity): "How active or alert do you feel right now on a scale of 1 to 7, where 1 means 'very calm or tired' and 7 'very active or alert'?"

[0068] - Dominance (sense of control): "How controlling or dominating do you feel right now on a scale from -5 to +5, where -5 means 'I feel that the situation controls me' and +5 'I feel that I control the situation'?"

[0069] The survey can also be conducted by presenting the questions with synthesized voice. Responses can provide information not only from words and sentences but also from the user's voice and facial expressions and gestures.

[0070] The objective of this invention is to assist users suffering from loneliness by providing mood support measures and particularly connections to relatives, friends, caregivers, therapy services, or similar when their mood is slipping into an abnormal or non-neutral area. The invention lowers the threshold for taking action through a technical solution that measures, recognizes, and monitors the individual's vitality and emotional state throughout the day, and according to certain algorithms, determines when it is appropriate to contact a relative, friend, caregiver, therapy service, or artificial intelligence service, and automatically opens a connection to the suitable service or person for that emotional state, so the individual does not need to make the decision to initiate contact themselves. The scope of the invention also includes various mood support interventions such as games, music, brain exercises, physical exercises, virtual therapy services, artificial intelligence therapy services etc., which can be used instead of conversation-based video calls, audio calls, and texting connections if they are not available at the needed moment or if the user prefers them.

[0071] The objectives of this invention are achieved in its simplest form with an arrangement programmed into a smartwatch and smartphone, where the user's vitality and emotional state are measured, recognized, and monitored throughout the day and, upon meeting set conditions, a phone connection is automatically opened to the most suitable person to assist at that moment, or alternatively, it offers the user mood support interventions appropriate for different emotional states, such as games, music, brain exercises, physical exercises, etc.

[0072] In the arrangement according to the invention, the smartwatch is programmed to function so that it processes a vitality level value and trend describing the user's vitality at any given moment from the data produced by its movement and other sensors. It is known that vitality directly correlates with the intensity of the emotional state known as Arousal, thus providing an initial indication of the emotional state and mood. The vitality value and trend are compared against threshold values set using historical vitality data, and if the comparison indicates that the vitality level is too low or too high, a more detailed assessment of the emotional state is initiated. For a more detailed assessment, the smartphone is programmed to analyze the user's voice and facial expressions and gestures using the previously described known methods and techniques. To analyze the user's voice and facial image, the smartwatch or smartphone is programmed to prompt the user with questions or a motivating command to say something aloud and look towards the camera. After collecting sufficient voice and facial image data, the smartphone initiates their analysis. The smartphone's memory stores annotated data from the user's previous voice and facial image and processed parameters, against which the newly obtained voice and facial image data are compared or reflected, and from these characteristics, the user's current emotional state or certain dimensions of it (e.g., V, A, and / or D) are determined using the aforementioned known methods. The smartphone's memory has previously stored information on which emotional states require contact and with whom at that particular time of the week and day. Thus, when the action threshold is exceeded, the smartphone automatically establishes a phone or video connection with that person. After the call, the smartphone asks the user for feedback on whether the call was necessary and if the person was suitable for the situation. This information is used to refine the parameters of the algorithms stored in the arrangement's memory, which determine in what emotional states contact should be initiated and with whom.

[0073] In case no suitable contact is available when the action threshold is exceeded, or the user does not want to bother their friends and relatives, the arrangement offers mood support interventions suitable for different emotional states, such as games, music, brain exercises, podcasts, videos, guided physical exercises, etc. During or after the use of these interventions, the arrangement continues to monitor changes in emotional state and refine its algorithms so that in a similar situation next time, it can offer a mood support intervention that causes a better change in emotional state. The arrangement gives the user the option to prioritize these non-intrusive mood support interventions to operate primarily, and only if they do not help in correcting the emotional state, then move to establishing a phone or video connection.

[0074] The advantage of this invention is that it ensures the user receives sufficient interaction with relatives, friends, and caregivers, as well as other mood-supporting activities as needed. It minimizes inconvenience to the user by initiating the determination of emotional state only within specific vitality levels when the need for action is clear. It also minimizes unnecessary actions by basing the decision to initiate contact or other actions on reliable emotional state assessments derived from the analysis of voice and / or facial images. It is capable of selecting the most appropriate conversation partner or mood intervention for each situation. Thus, it knows the suitability values of the designated contacts and, if necessary, can choose an intervention such as music, games, brain exercises, podcasts, videos, or other services or activities suitable for the situation.

[0075] Fundamentally, the invention is an arrangement and method designed for recognizing and supporting emotional states, which not only maintains user interaction with relatives, friends, caregivers, and other contacts but also offers other mood-supporting interventions. It measures the user's vitality using a smartwatch, recognizes the emotional state by analyzing the voice and facial images, and initiates video calls to designated contacts or offers mood interventions when necessary. Specifically, the vitality measurement is programmed to monitor the user’s vitality level and send an alert to the smartphone when the vitality level drops below a predefined lower threshold or rises above an upper threshold for that specific time; upon receiving an alert, the smartphone is programmed to activate the user by voice or text message to speak, and while the user speaks, it records the user's voice and facial image, analyzes the emotional state, and activates contact if the user’s emotional state meets the criteria for exceeding the action threshold as set for that particular time. Furthermore, the smartphone is programmed to initiate contact via a video call with the contact who has the highest suitability value for that user’s particular emotional state at that moment, or alternatively, to initiate the most appropriate mood intervention for the situation.

[0076] Some preferred embodiments of the invention are described in the dependent patent claims.

[0077] The invention is described in detail below with reference to the accompanying drawings:

[0078] - Figure 1 shows the arrangement according to the invention in a block diagram.

[0079] - Figure 2 illustrates the vitality level data resulting from vitality measurement stored at the daily, monthly, and yearly levels.

[0080] - Figures 3a-3d show the Valence- Arousal-Dominance maps used in the analysis of emotional states according to the VAD model in a three-dimensional coordinate system.

[0081] - Figure 4 shows a flowchart of one operational mode of the arrangement according to Figure 1.

[0082] - Figure 5 shows the integration of mood interventions into the arrangement according to the invention.

[0083] - Figure 6 shows a flowchart of one operational mode of the arrangement according to Figure 5.

[0084] The embodiments described in the following explanation are exemplary only. A professional in the field may implement the basic idea of the invention in ways other than those described. For instance, the analysis of the emotional state as described is predominantly conducted within a three-dimensional framework. However, it is equally feasible to perform this analysis in one, two, four, or even higher dimensions.

[0085] Although the description may refer to one or more embodiments in several places, this does not imply that the reference is only to the described embodiment, or that the described feature is only useful in one described embodiment. Features of two or more embodiments may be combined to create new embodiments of the invention.

[0086] Figure 1 presents a block diagram of an embodiment of the arrangement according to the invention. The measurement and analysis of the user's vitality are programmed to occur in section 100 of the block diagram. The measuring device may be, for example, a smartwatch 101 worn on the user’s wrist, whose motion sensors 102, heart rate sensors 103, respiration sensors 104, and galvanic skin response sensors (GSR) 105 generate measurement data for the vitality determination application 106. Some of these sensors, especially the respiration sensors 104, may be located outside the smartwatch, positioned on different parts of the user’s body, and transmit their measurement data wirelessly to the smartwatch 101 or directly to the smartphone 121. The vitality determination application 106 is programmed to calculate the user's vitality level using known algorithms employed in smartwatches and activity bracelets and to store the vitality level value, for instance, every 1-30 seconds in the vitality history database 109. The processing may use continuous averaging of measurement values over a set period, such as one minute, to prevent random rapid changes in vitality from causing unnecessary alerts. The threshold value processing application 108 is programmed to determine and set the lower and upper limits of the normal vitality level, or alert thresholds, for each moment, which are also referred to as alert limits. These alert thresholds may be manually set during the deployment phase of the arrangement, for example, hour by hour or as preset threshold functions where thresholds are predefined for each season, each day of the week, and each hour, roughly estimated as starting values. In a more advanced embodiment of the invention, the determination of these thresholds also defines the lower and upper limits for the rate of change of the vitality level for each moment. The vitality analysis application 107 is programmed to determine whether the vitality level exceeds the upper limit or falls below the lower limit set for that moment. In a more advanced embodiment, the comparison also takes into account the rate of change, whether it exceeds the upper limit or falls below the lower limit. If the vitality level exceeds the upper limit or falls below the lower limit, an alert occurs, and the vitality analysis application 107 sends a command to the smartphone 121 to start measuring and analyzing the emotional state. It also sends information about the vitality level to be used as additional information in the emotional state recognition by the emotional state analysis application 127.

[0087] As use continues, the threshold values are automatically fine-tuned based on feedback received from and / or measured from the user. The fine-tuning can also be set to occur automatically, for example, in 1-5% increments (with the min-max scale corresponding to 100%). This means that if the alert-triggered emotional state analysis does not lead to a contact or mood intervention, the alert threshold value for that particular day and time is raised by one increment in a direction that reduces the sensitivity of future alerts at the same weekday and time. Therefore, if an alert is caused by exceeding the upper limit, the upper limit is raised by one increment, and if the alert occurs from falling below the lower limit, the lower limit is lowered by one increment. The adjustment of threshold values occurs over a set period, for instance, hourly. For example, if an alert occurs at 7:38 AM and it does not lead to contact or a mood intervention, the threshold value is adjusted for the 7:00-8:00 AM period. The adjustment can also be made smoothly so that the threshold value for that time period is raised, but the threshold curve is maintained as continuous over time, resulting in a greater change in the middle of the interval and less at the edges, and the threshold values for adjacent periods are adjusted at their edges so that there are no jumps in the threshold curve.

[0088] The determination of the user's emotional state is programmed to occur in section 120 of the block diagram in Figure 1. Upon receiving the command, the smartphone 121 alerts, and when the user picks up the smartphone, it presents a question or several questions or tells a joke through the voice and video call application 122, to which the user normally responds by speaking, laughing, and / or expressing emotions while looking at the smartphone. The voice recording application 123 records and stores the user's voices and speech and sends the generated data to the speech analysis application 124 for processing for emotional state recognition. The camera application 125 photographs and stores the user's facial expressions and gestures, and sends the generated data to the image analysis application 126 for processing for emotional state recognition. The emotional state analysis application 127 recognizes the emotional state using data from the speech analysis application 124, the image analysis application 126, and the vitality analysis application 107. The recognized emotional state is stored in the emotional state history database 128. The emotional state analysis application 127 also makes a decision on whether contact or a mood intervention is needed at that moment based on the emotional state. Decision-making is based on the VAD threshold value application 129’s set threshold values for emotional state dimensions, which are compared with the dimensions of the emotional state at that moment. Initially, the threshold values are set using experience-based knowledge learned from other users, where the user provides personal information such as age, gender, personality type, physical health, mental health, accustomed daily rhythm, or other mood management-related information during the deployment phase, and an average of the threshold values from similar users is used to set the initial threshold values for the user. As use progresses, the threshold values are adjusted individually based on accumulated experience. Thus, the determination of threshold values takes into account information from the emotional state histories of the user and other similar users about whether a contact or mood intervention in a similar emotional state has previously been deemed necessary and led to a desired change in emotional state. If the decision on the intervention is negative, this information is sent to the vitality threshold value processing application 108 for fine-tuning the threshold values to reduce unnecessary determinations of the emotional state. If, however, the decision on making contact is positive, meaning the intervention threshold is exceeded, a command is sent to the contact application 140 to initiate contact. Similarly, if the decision on the mood intervention is positive, a command is sent to the mood intervention application 540 (Figure 5) to initiate the mood intervention.

[0089] Upon receiving the activation command, the contact application 140 analyzes the current emotional state by comparing it with data stored in the emotional state history database 128, especially related to the corresponding day of the week and time, and data stored in the contact history database 141, particularly the outcomes resulting from previous contacts made in similar emotional states. Based on this comparison, the contact application 140 decides which contact person is the best option in this situation and initiates the opening of a call or video call using the smartphone's voice and video call application 122 to the selected contact person's communication device 142-146. If the contact attempt fails, the contact application 140 selects the next best option and initiates connection to their communication device.

[0090] The number of contact persons can vary according to the user's choice. Contact persons can include close family members and friends, as well as public and private healthcare professionals. A contact service can be established with public and private healthcare services, from which users can select suitable healthcare or care professionals as their contact persons. The arrangement according to the invention can also be linked to social media platforms, such as Facebook, so that people designated as friends there can be invited to act as contact persons in a service operating based on this arrangement.

[0091] In the aforementioned comparison, for each emotional state, a suitability value, for example, on a scale of 0 to 100, is stored for each contact person in the contact history database 141 for each day of the week and time of day. The contact application 140 uses this suitability value to select the best option for each situation from the contact persons. The calculation of the suitability value involves the contact application 140 comparing the effect of different contact persons on the user’s emotional state in each emotional state. This effect is stored in the contact history database 141 not only as a change and trend in the emotional state (e.g., changes in the VAD dimensions resulting from the contact, or the direction and speed of change during the contact) but also as user feedback about the conversation with each contact person. In a more advanced embodiment of the invention, feedback is also sought from the contact person about the conversation, and the emotional state of the contact person during and after the conversation can be recorded in a similar method on their smartphone. To determine the suitability value of contact persons, the arrangement is programmed to perform one or more of the following operations:

[0092] - The arrangement asks the user’s opinion on which contact person they wish to connect with in each emotional state, for example, on a scale of 0 to 10,

[0093] - The arrangement monitors the change in the user’s emotional state during contacts and records the direction and magnitude / speed of the change caused by each conversation with each contact person, for example, as a percentage of the maximum scale in the VAD coordinate system,

[0094] - The arrangement asks the user's opinion about the change in their emotional state after each contact, for example, on a scale of -5 to +5, and

[0095] - The arrangement asks each contact person's opinion about the change in the user’s emotional state after each contact, for example, on a scale of -5 to +5, and calculates the suitability value as a weighted average of the results from the selected operations, weighted by set coefficients, and scales the result, for example, to a scale of 0 to 100.

[0096] The user's emotional state and its changes can be recorded and stored in the emotional state history database 128 even without a command from the vitality analysis application 107 whenever the user utilizes their smartphone 121 and their voice is within the hearing range of the smartphone's microphone and / or their face is within view of the smartphone's camera. In these situations, too, if the emotional state exceeds the thresholds set by the VAD (Valence, Arousal, Dominance) models, the contact application 140 receives a command to initiate contact, even if the vitality measurement arrangement 100 has not issued an alert. Information about the need for contact occurring without an alert from the vitality measurement is used to automatically adjust the thresholds in the threshold value processing application 108 so that similar future situations would trigger an alert. The thresholds can also be made more sensitive incrementally, for example, in 1-5% increments, unlike in cases where an alert proved unnecessary, where they are adjusted to be less sensitive.

[0097] The sensors described in Figure 1 can measure many different parameters to produce data for determining vitality. For example, motion sensors 102 can measure 1-3-axis linear movement (e.g., via an accelerometer) and 1-3 -directional rotational movement (e.g., via a gyroscope) as well as orientation (e.g., via a magnetometer); heart rate sensors 103 measure heart rate (HR) and heart rate variability (HRV); respiration sensors 104 measure breathing frequency and depth; and GSR sensors 105 measure skin conductivity or its changes (GSR, galvanic skin response, or alternatively EDA, electrodermal activity).

[0098] In addition to the smartwatch, the user may use other wearable sensors, such as an activity bracelet, smart ring, heart rate belt, separate glucose sensor, blood oxygen sensor, wearable blood pressure monitor, etc.

[0099] Vitality strongly correlates with physical activity. As vitality levels increase, so does physical activity, while low vitality leads to reduced physical activity. High vitality typically indicates increased alertness, concentration, and energy levels, which can lead to greater physical activity. This might manifest as faster movement, more frequent movement, or more vigorous movement. Conversely, low vitality can lead to decreased physical activity, which might manifest as slower movement, less frequent movement, or less vigorous movement.

[0100] Vitality also affects heart rate (HR) in various ways. When vitality is high, such as in emotional states of stress, fear, or excitement, the sympathetic nervous system is activated, which generally leads to an acceleration of the heart rate. When vitality is low, such as in emotional states of relaxation or sleepiness, the parasympathetic nervous system is more active, which generally leads to a slowdown in heart rate.

[0101] Vitality is also reflected in heart rate variability (HRV), which measures the variation in time intervals between heartbeats and can be processed from heart rate data. High HRV generally means that the heart can quickly adapt to varying situations, a sign of good heart health and autonomic nervous system balance. High vitality (e.g., in a state of stress) can lead to a decrease in HRV, as the sympathetic nervous system is more active and the influence of the parasympathetic nervous system weakens, reducing heart rate variability. On the other hand, low vitality (e.g., in a state of relaxation) typically leads to an increase in HRV, as the parasympathetic nervous system is more active and heart rate variability increases.

[0102] Vitality affects respiration in multiple ways. A high level of vitality, such as in emotional states of stress, fear, or excitement, typically leads to accelerated and more superficial breathing. This is due to the activation of the sympathetic nervous system, which prepares the body for a 'fight or flight' response. Conversely, when the vitality level decreases, as in relaxed or sleepy emotional states, breathing generally slows and deepens. This results from the activation of the parasympathetic nervous system, which returns the body to a calmed state. Thus, the depth and frequency of breathing usually correlate with the level of vitality and can be used as aids in measuring vitality. Additionally, the patterning of breathing, i.e., the ratio of inhalation to exhalation duration, and the frequency of yawning can be utilized as additional data in determining vitality levels and emotional states. Breathing depth, patterning, and yawns can be processed from data collected by motion sensors placed on the chest. At its simplest, respiratory rate and depth can be processed from heart rate data or from motion sensors in a smartphone in a chest pocket, thereby eliminating the need for separate respiratory sensors.

[0103] Skin conductivity, often measured using GSR sensors to utilize the galvanic skin response (GSR), generally correlates with vitality levels as well. Higher levels of vitality (for example, in emotional states such as stress, fear, or excitement) generally lead to increased skin conductivity, as these states stimulate the sympathetic nervous system, which regulates sweating. Increased sweating raises skin conductivity. As the vitality level decreases, such as in calm or relaxed emotional states, skin conductivity typically decreases. This is because the parasympathetic nervous system, which activates in calming conditions, reduces sweating, thereby reducing skin conductivity. Therefore, there is generally a direct correlation between skin conductivity and vitality levels. Another possibility for utilizing sweat is to measure the chemical compounds, nanoparticles, and biological components such as exosomes it contains. Since the emotional state is known to affect the composition of sweat, measuring this composition provides information about the emotional state.

[0104] Thus, information about vitality can be processed from data obtained solely from motion sensors but also by using other measurement data (such as heart rate, HRV, respiratory rate, respiratory depth, skin conductivity) and combining their data for a more reliable measurement of vitality levels. For example, a high heart rate, rapid breathing, and high GSR value together more reliably indicate a high level of vitality. Known algorithms used in activity applications developed for smartwatches and bracelets are also useful in various embodiments of this invention. In these, combining and analyzing different sensor data is done using multivariable methods, which are statistical and machine learning methods and can handle multiple variables simultaneously. These methods include the following:

[0105] - Regression analysis, which aims to predict a dependent variable based on one or more independent variables. For example, one might try to predict a person's vitality level based on heart rate, respiration, movement, and GSR. - Principal component analysis (PCA) is a statistical method used to reduce data dimensions while retaining as much data variation as possible. It can help identify common features from various measurements.

[0106] - Clustering is a machine learning method that can divide data into similar groups. For example, groups can be found that have similar heart rate, respiration, movement, and GSR profiles.

[0107] - Various machine learning algorithms, such as Decision Trees, Random Forests, Deep Learning Algorithms, Reinforced Learning Algorithms, and Neural Networks, can handle multiple variables simultaneously and identify complex patterns in data.

[0108] Figure 2 illustrates vitality level data generated from measuring vitality, stored on a daily scale at 200, a monthly scale at 210, and an annual scale at 220. A weekly scale could also be used instead of a monthly scale to prominently highlight the weekly rhythm, including weekends. The figure exemplarily depicts a vitality curve 201, starting at midnight 00:00 and ending the following midnight at 24:00, along with a vitality level variation range that extends from zero to a maximum value of 209. During night hours, the vitality level is typically lowest, and the lower alert threshold 203 is also at its lowest. The upper limit 202 is typically also at its lowest during night hours. During morning wake-up and activities, the vitality level typically rises and may briefly peak at 204, but does not trigger an alert because the upper limit 202 has risen to its higher daytime level. Post-lunch, there is typically a calming period and, for many elderly users, a time for naps, during which the vitality level goes down to 205. In the example, something stimulating occurs in the afternoon (such as failing to pay bills on the computer, stock market diving, or getting engrossed in an online game), causing the vitality level to rise above the upper limit at the moment 206 and continue rising in a clear trend, prompting an alert 207 to go off soon thereafter. From the figure, it can be inferred that the alert has led to a contact, as a result of which the user's vitality level quickly drops and settles back to a normal level by the present moment 208, well before bedtime. The daily rhythm can also be considered when adjusting the alert thresholds for vitality levels, for example, by lowering the upper alert threshold in the early evening, so that a user in an excited state can receive a calming contact if necessary. Even a simple invitation to analyze the emotional state can help the user recognize the need to calm down and prepare for bedtime.

[0109] Figure 2 illustrates vitality data also on a monthly scale 210. In the vitality curve 211, slow changes including peaks and troughs and a trend-like change 212 are depicted, from which the change information can be used to fine-tune the thresholds of learning algorithms. The figure also demonstrates weekly systematic fluctuations in vitality levels. Additionally, the alerts that have occurred during the current month are marked as 213 and 214. In the example shown in the figure, the current day's vitality level 215 significantly deviates from the previous similar weekdays' vitality levels 216, 217, and 218, which can also be considered in adjusting the alert thresholds.

[0110] In the yearly vitality data 220, the vitality curve 221 shows very slow changes, which can be interpreted, for example, as variations in vitality levels according to seasons. In the example of Figure 2, the vitality level peaks in April 222. The alert thresholds can be fine-tuned by distinguishing systematic variations in the user's vitality data and refining the alert thresholds accordingly. In the example of Figure 2, the upper alert threshold can be raised during spring because the natural rhythm of that user includes a higher vitality level in spring. Similarly, systematic variations in life rhythm related to weekends can be automatically considered in fine-tuning the algorithms for alert thresholds. In algorithms that use neural network computation, training can occur, for example, self-directedly (SOM, Self Organizing Map), allowing the alert thresholds to set and change automatically during self-directed learning so that the alerts lead to a maximum positive change in the user's various emotional states and do not burden contacts with unnecessary or ineffective communications. When using neural networks, self-directed learning is advisable to be applied to the entire system so that the entire event chain, vitality level measurement — emotional state analysis — contact initiation, produces the maximum positive change in the user's emotional state relative to the time and effort of the contacts.

[0111] Figure 3a presents a Valence- Arousal-Dominance map 300 according to the VAD model in a three-dimensional orthogonal coordinate system, which can be used in one embodiment of the invention. The axes and their scales can be arranged in many other ways, but for the purpose of illustrating this invention, all axes in Figure 3a use positive and negative value ranges so that the neutral or normal value area is near zero on each axis and thus near the origin 316 from all dimensions' perspectives. For data processing, a maximum is set in both directions for each axis, allowing values exceeding these maxima to saturate to their maximum value if necessary. The Arousal vertical axis (ESA) 301 describes the intensity of the emotional experience. Upwards, the intensity of the emotional experience increases, in the middle, it is normal, and downwards, it decreases. The Valence horizontal axis (ESV) 302 describes the pleasantness of the emotional experience such that moving to the right increases pleasantness, in the middle, it is neutral, and 1 moving to the left increases unpleasantness. In the third dimension, the Dominance axis (ESD) 303 describes the controllability of the emotional experience. In the depicted case, the feeling of control increases along the receding axis, in the middle, it is neutral, and along the approaching axis, the feeling of control weakens, meaning the emotional state increasingly dominates the user.

[0112] In Figure 3 a, dashed lines are drawn at boundaries where the emotional experience moves outside the neutral and / or normal area near the origin 316, thus initiating an action process. At these axes, the threshold values are marked in Figure 3a as follows: the upper limit of normal Arousal 310 is AH and the lower limit 311 is AL, the upper limit of neutral Valence 312 is VH and the lower limit 313 is VL, the upper limit of neutral Dominance 314 is DH and the lower limit 315 is DL.

[0113] In a simple embodiment of the invention depicted in Figure 3b, the condition that an emotional state outside of which triggers the initiation of an action process is a sphere with a radius r 317, the surface intersection points of which with each axis of the coordinate system match the previously described threshold values AH, AL, VH, VL, DH, and DL. The emotional state appears as a vector 318 originating from the origin, moving within the sphere as the emotional state varies, and upon breaching the sphere's surface, the criteria for crossing the action threshold are met, initiating the action process. In another simple embodiment (Figure 3c), the condition that an emotional state outside of which triggers the initiation of an action process is a cube 319, the walls of which align in each direction with the previously described threshold values AH, AL, VH, VL, DH, and DL. The emotional state appears as a vector 320 originating from the origin, moving within the cube as the emotional state varies, and upon breaching one of the cube's 319 walls, the criteria for crossing the action threshold are met, initiating the action process. In a more advanced embodiment (Figure 3d), the threshold values for different axes depend on each other, and thus the condition is a complex space 321, the surface distance from the origin in each direction determined by the processed combination of three threshold values in the VAD threshold value application 129. The emotional state appears as a vector 322 originating from the origin, moving within the space as the emotional state varies, and upon breaching the surface of the space 321, it triggers the action process. A practical example is mentioned where if Dominance is high (strong feeling of control), Valence and / or Arousal might rise high before the action threshold is exceeded. On the other hand, if Valence is strongly negative and Arousal high, it might indicate a feeling of anger, hence the action threshold should be lower. Negative Dominance with high Arousal and negative Valence means a feeling of fear, where the action threshold should also be lower. During the teaching period of the system, the interface describing the action threshold, also called the threshold function, is shaped individually for each user, and during use, this interface is adjusted based on the feedback received in the VAD threshold value application 129.

[0114] In a preferred embodiment of the invention, the space described by the VAD model is divided into sections that represent different emotional states according to a discrete model. In this way, the emotional state space analyzed according to the VAD model can be divided into 2-6 parts for each dimension, resulting in 8-216 named or numbered possible emotional states, into which the user's prevailing emotional state is classified at each moment in time, for example, using probability values. Note that instead of three dimensions the emotional space can also be defined by one, two, four or more dimensions and subdivided into two or more segments per dimension, thereby forming a plurality of distinct emotional states, each uniquely named or numbered. Criteria can be defined for each possible emotional state that determine when an action threshold is exceeded. The criteria can consider the probability values of the user's prevailing emotional state targeting different emotional states, from which the exceeding of the action threshold can be calculated using weighted values. Contact person suitability values can also be set for each possible emotional state separately, and the most suitable contact person for the user’ s prevailing emotional state can be calculated from weighted suitability values. In a simple embodiment of the invention, emotional states outside the neutral and / or normal range may include, for example: 1) Excited / Pleased, 2) Happy / Content, 3) Exhausted / Bored, 4) Sad / Downhearted, 5) Irri- tated / Frustrated, and 6) Anxious / Stressed. The connection application 140 defines, based on the contact history 141, to which contact person the connection should be opened at each moment in the aforementioned emotional states. Note that connections should also be made in positive emotional states such as Excited / Pleased, allowing the user to share good experiences with the contact person and the contact person to hear positive news from the user, thus improving the emotional state of both parties.

[0115] Figure 4 illustrates an operational mode of the arrangement according to Figure 1 as a flowchart. The arrangement consists, for example, of a smartwatch worn on the user's wrist, which is in constant connection with their smartphone, such as an iPhone. Instead of a smartphone, a tablet device, such as an iPad, can also be used. Additionally, it is possible that the smart devices are connected to a local data network and use the computing, memory, and database capacities of connected computers or similar. Furthermore, the smart devices may be connected to the internet and use the computing, memory, and database capacities of connected systems, such as cloud services.

[0116] After deployment, the arrangement is typically always kept on. Sensor data flows in at step 401 and the vitality level is processed at step 402. The processing of the vitality level may occur in the smartwatch or smartphone, or in cooperation with services available from the data network. Vitality data is stored at step 403 in the vitality history database 404, and alert thresholds are adjusted continuously with historical data according to set algorithms at step 405. At step 406, the processed vitality level is tested against the thresholds. If the vitality level exceeds the lower threshold Vita and is below the upper threshold Vita, processing and testing of the vitality level continue at step 402. When the vitality level falls below the lower threshold Vita or exceeds the upper threshold Vita, a notification message is sent to the user at step 407. Simultaneously, the processing of the vitality level from sensor data 402 and its storage in the vitality history data 403 continues, so that vitality level information is also available for the processing of the emotional state.

[0117] In step 407 of Figure 4, the smartwatch or smartphone alerts and prompts the user to respond. The alert can come as an auditory question when the vitality level falls below the alert threshold, such as: “How are you? Is something bothering you?” Or, when the vitality level rises above the alert threshold, as a question: “Is everything alright? Could you stop to answer a few questions?” The questions may also appear as text or icons on the display of the smartwatch or smartphone. Following the opening question, the arrangement according to the invention may pose questions related to current news or previous conversations with the user. The aim is to genuinely engage the user in the conversation, thereby expressing their emotional state, which the arrangement measures and stores at step 408 and analyzes at step 409 based on the clues provided by the user's voice and / or content and facial expressions and / or gestures. The arrangement might also tell a joke and observe the user's reaction to it. For instance, a hearty laugh indicates a positive Valence, a stifled 'huh' indicates a negative one. The arrangement may also use emotion-stimulating music or video by playing it from the smartphone or tablet and monitoring the user's reaction, recording vocalizations and facial expressions and gestures. Analyzing the prevailing emotional state at step 409, when operating according to the VAD model, produces values for the three dimensions: Valence, Arousal, and Dominance. These dimensions of the emotional state can change rapidly during the conversation, necessitating their averaging over, for example, a few minutes to obtain a reliable estimate of the emotional state. The speed and extent of changes in the dimensions of the emotional state also contain information about the user’s actual emotional state, so they can be used to fine-tune the algorithms for emotional state analysis. Alternatively, when using discrete modeling of emotional state, the analysis may classify the prevailing emotional state into a specific named emotional state, such as cheerful, depressed, or frightened, or into a percentage probability of different emotional states. The flowchart in Figure 4 is drawn with an emphasis on the VAD model.

[0118] In step 410 of Figure 4, the emotional state information obtained from the analysis of the prevailing emotional state is stored in the emotional state history database 411. In addition to timestamps indicating the season, day of the week, and time of day, the database may also contain health and wellness data produced by the applications of the smartwatch 101 and smartphone 121. These health and wellness details can include data on the day's, week's, or longer period's physical activities, active and resting energy consumption, heart rate, step count, walking and running distances, standing versus sitting, floors climbed, resting heart rate, heart rate variability, blood oxygen levels, sweating, sweat composition, respiratory rate, sleep duration and its distribution across different types of sleep: light, baseline, deep NREM, and REM sleep. This data can be used as additional inputs in the processing of VAD threshold values at step 412. Particularly when processing is performed using neural computation, these additional inputs are valuable as features that allow the pattern recognition to autonomously extract important characteristics and form complex classifications. In its simpler embodiments, the arrangement according to the invention can be based on the processing of VAD thresholds at 412 using only the recorded emotional state data 410 and timestamp information.

[0119] In step 413 of Figure 4, the arrangement according to the invention makes a decision on the necessity of contact or mood intervention. It compares the emotional state ES obtained from the analysis at 409 with the boundaries resulting from the VAD thresholds. In a simple embodiment of the invention, as described in Figure 3, this comparison involves comparing the three dimensions of the emotional state, ESV, ESA, and ESD, individually against their respective upper and lower limits to determine if all three dimensions are within these limits, which is marked in the figure by VADL < ES < VADH. It means that all tests VL < ESV < VH, AL < ESA < AH, and DL < ESD < DH are true, in which case no contact or mood intervention is needed. In this case, the boundaries form a cube as described in Figure 3 c. In a more advanced embodiment (Figure 3d), the boundaries form a complex interface against which the emotional state vector 322, or distance from the origin, is compared. If the emotional state vector remains within the interface defined by the threshold function, the criteria for contact or mood intervention are not met, and no action is needed.

[0120] If the result is that no contact or mood intervention is necessary, the process moves to step 414, i.e., refining the algorithm for vitality threshold values. If the emotional state is clearly within the normal and / or neutral bounds, i.e., does not exceed any dimension, for example, beyond 70% of the boundary interface, necessary fine-tunings are made to the vitality threshold value processing algorithms 404, such as loosening the thresholds by 1-5% to prevent overly sensitive triggering of emotional state analysis in similar future situations. If the emotional state exceeds the normal and / or neutral interface, the process moves to step 415, i.e., initiating contact. Similarly, in the arrangement shown in Figure 6, the process moves to implementing the mood intervention at 615.

[0121] Once the intervention process is initiated, step 416 analyzes the emotional state by comparing it to the data stored in the emotional state history database 411 and the contact history database 418, especially focusing on changes resulting from previous contacts made in a similar emotional state. At step 417, the arrangement selects from the contact persons the one whose previous contacts have led to the greatest positive change in the emotional state. At step 419, the arrangement initiates a call or video call. If the contact attempt fails, the process (shown by a dashed line) returns to step 417 to select the next best contact person alternative. This continues until the contact is successful.

[0122] Once the contact is successful, a call or video call with the contact person begins at step 420 and ends at step 423. During the call, the user’s speech and / or video of their face are recorded (step 421), and changes in the user’s emotional state are analyzed in step 422. After the call, at step 424, feedback on the improvement of the emotional state is requested from the user via one to three written or spoken questions on the smartphone, and in step 425, the data on the emotional state changes in this contact are updated in the contact history database 418.

[0123] Figure 5 illustrates an arrangement according to the invention where, as an alternative to contact, various mood-supporting and enhancing programs and therapy services are used. These may become necessary, for example, when no suitable contact person is available. The user can also prioritize certain support programs or virtual therapy services for desired emotional states, and only if these are not helpful, then make contact with the right person. Mood-supporting programs can include calming music, relaxing games, inspiring podcasts and videos, mindfulness and meditation exercises, brain exercises, etc. Mood-supporting therapy services can include virtual psychotherapy and group therapy, mood-lifting interactive entertainment or conversation with a therapy robot, various therapy forms based on generative artificial intelligence, etc. In this document, mood- supporting programs and therapy services are briefly referred to as mood interventions.

[0124] The arrangement shown in Figure 5 operates as described in Figure 1 for the measurement and analysis of vitality 100, determination of emotional state 120, and contact 520. Additional blocks are the selection of the intervention 510 and the implementation of mood interventions in block cluster 530. Thus, in this arrangement, the interventions include not only contact options but also mood interventions.

[0125] The selection of the intervention 510 in one embodiment of the invention operates such that if the contact person selection 417 shown in Figure 4 results in a situation where none of the suitable contact persons are available and the contact attempt fails, it moves to select the best suitable mood intervention 542-546 from the block diagram of Figure 5, which could be one or several instead of the six shown.

[0126] Upon receiving a start command from the intervention selection unit 510, the mood intervention application 540 analyzes the current emotional state by comparing it to the data stored in the emotional state history database 128, especially related to the corresponding day of the week and time, and to the mood intervention history database 541, especially focusing on changes resulting from previous mood interventions made in a similar emotional state. As a result of the comparison, the mood intervention application 540 decides which of the mood interventions 542-547 is the best option in this situation and initiates it using the smartphone’s audio and video call application 122. If the selected mood intervention fails, the mood intervention application 540 selects the next best option and initiates it.

[0127] For determining the suitability values of mood interventions, the arrangement is programmed to perform one or more of the following operations:

[0128] - a) The arrangement asks the user’s opinion when selecting mood interventions into the system, which mood intervention he wants to be implemented in each emotional state, for example, on a scale of 0 to 10, - b) The arrangement monitors the change in the user’s emotional state during the use of mood interventions and stores in the mood intervention history database 541 the direction and magnitude / speed of the change caused by each mood intervention in each emotional state, for example, as a percentage of the maximum scale in the V D coordinate system, and

[0129] - c) The arrangement asks the user’s opinion on the change in their emotional state after each mood intervention, for example, on a scale of -5 to +5, and calculates the suitability value as a weighted average of the results of the selected operations using set weighting factors and scales the result, for example, between 0 to 100. Additionally, after each mood intervention, the arrangement may ask the user to update the rating given in part a) regarding the desirability of that mood intervention in that emotional state, which change will be reflected in the suitability value of that mood intervention the next time it is selected.

[0130] Figure 6 presents a flowchart where the upper part 610 is the same as the corresponding part in Figure 4, but the lower part describes the events in the case that, instead of a contact, a mood intervention is implemented. Once the intervention process is initiated, step 616 analyzes the emotional state by comparing it to the data stored in the emotional state history database 411 and the mood intervention history database 618, especially focusing on changes resulting from previous mood interventions made in a similar emotional state. At step 617, the arrangement selects from the mood interventions the one that previously led to the greatest positive change in the emotional state. At step 619, the arrangement initiates the selected mood intervention. If the intervention is not available for some reason, the process returns (shown by a dashed line) to step 617 to select the next best option. This continues until the intervention starts.

[0131] The mood intervention is carried out at step 620 and ends at step 623. During the intervention, if possible, the user’s speech and / or video of their face are recorded (step 621), and changes in the user’s emotional state are analyzed in step 622. After the intervention ends at 623, feedback on the improvement of the emotional state is requested from the user via one or more written or spoken questions on the smartphone at step 624, and in step 625, the data on the emotional state changes in this mood intervention are updated in the mood intervention history database 618.

[0132] The choice of intervention between contact and mood interventions can be set to operate such that mood interventions are the primary interventions and contact is only initiated if they do not achieve sufficient correction in the user’s emotional state. Additionally, the user can choose to turn off either type of intervention entirely if desired.

[0133] This invention can also apply a learning system that discovers the association between inputted data and correct classification, which requires training material from the modeled process, i.e., measured data and their corresponding classifications. The system forms an approximation of the function that can perform the desired classification based on input features. Methods can also be used where the learning system does not need to be told the desired classification but automatically searches for various groups (classes) from the input. This is referred to as unsupervised learning. Available learning methods include, among others, Decision tree learning, Artificial neural networks, Bayesian learning, Instance-based learning, Analytical learning, and Reinforcement learning.

[0134] In determining VAD thresholds, i.e., the threshold function, the learning system can be programmed to regulate the interface of the normal and / or neutral state such that the threshold function is initially set as a sphere representing a constant distance from the origin in all directions, or as a cube, the faces of which are set at the desired distance from the origin in each of the six directions, and the surface defining the space is allowed to be shaped according to the learning algorithm by shaping the surface distance from the origin in each direction such that each realized contact event produces the maximum positive change in the user’s emotional state. The surface modification can be done smoothly so that, while moving the interface in that direction, the surrounding interface is also moved so that the interface remains intact and does not have sharp change boundaries.

[0135] In neural learning, which is very suitable for this invention, the computation is performed by neural networks consisting of simple computational elements, neurons, which have multiple inputs and one output. Typically, there is a weight coefficient for each input, which either strengthens or weakens the significance of the inputs. The weight coefficients are usually determined by teaching with examples. The response value of the neuron is calculated from the weighted inputs. Fundamentally, neural computation is linear algebra, where, at its simplest, matrix multiplication and function evaluations are performed.

[0136] The arrangement and method according to the invention can also be applied to the diagnosis of various mood-related disorders, whether psychological or physical, and to assist patients in coping with the problematic emotional states associated with them. For example, the depression and mania phases of bipolar mood disorder strongly affect the patient’s emotional state and can be recognized by the emotional state analysis according to the invention. Similarly, mood fluctuations associated with Alzheimer's disease, Parkinson's disease, and other memory and mental illnesses can be recognized by the emotional state analysis according to the invention, and patients suffering from these can be helped with a video or audio call with a suitable contact person. In cases of illnesses, the emotional state map can be defined disease-specifically for emotional states that are typically associated with the mood states resulting from the illness for each user. The smartphone 121 can also be programmed and / or trained to analyze the emotional state 409 of a mentally sick or traumatized user during their daily life, with or without concurrent vitality measurements from a smartwatch 101, either continuously or during regularly scheduled intervals. This analysis identifies changes in the user’s emotional state profile to detect shifts in mental or psychological stability, and initiates a special contact call or a mood intervention designed to facilitate access to therapy or assistance for their symptoms and related issues.

[0137] The arrangement and method according to the invention can also be applied to recognize instances of cyberbullying based on changes in the user's emotional state while the user is using the smartphone during daily work or leisure routines. The recognition of cyberbullying can be programmed and taught in the emotional state analysis 409 application. This embodiment could include special contact persons or a reporting procedure for reporting of cyberbullying.

[0138] The smartphone 121 can also be programmed and trained to recognize instances of overwhelming use of addicting and / or toxic internet content, based on changes in the user's emotional state profile while engaging with short videos, online games, or similar services on the smartphone 121; and to initiate a special contact call or a mood intervention aimed at recovery from the addictive and toxic emotional experiences.

[0139] The foregoing has described some preferred embodiments of the arrangement and method according to the invention. The invention is not limited to the examples described herein, but the inventive idea can be applied in numerous ways within the limits set by the patent claims.

Claims

Claims1. An arrangement for monitoring and supporting a user's emotional state by activating or maintaining the user's interaction with relatives, friends, caregivers, and other contacts, as well as by offering various mood interventions affecting the mood, wherein the arrangement consists of- a device attached to the user for measuring vitality, for example, a smartwatch (101);- a mobile application installed on the user's smartphone (121) that recognizes and analyzes the user's emotional state from voice audio and / or facial images; and- a video call application installed on the user's smartphone (122), characterized in that- the vitality measurement device (101) is programmed to monitor the user's vitality level and send an alert (407) to a smartphone (121) when the vitality level falls below a preset lower threshold (203) for that specific time, or when the vitality level rises above a preset upper threshold (202) for that specific time;- upon receiving the alert, the smartphone (121) is programmed to activate the user to speak via a voice or text message, and while the user is speaking, to record the user's voice and / or facial image (408), analyze them to assess the user's emotional state (409), and to initiate contact (415) or mood intervention (615) if the user's emotional state meets the criteria (413) for exceeding the action threshold set for that moment; and- the smartphone (121) is programmed to initiate a contact (415, 419) as a video call, voice call or texting with a connection device of one of the contact persons who has the highest suitability value for that user in that emotional state at that moment (417), or- the smartphone (121) is programmed to initiate a mood intervention (615, 619) with one of the set mood interventions which has the highest suitability value for that user in that emotional state at that moment (617).

2. An arrangement according to claim 1, characterized in that the vitality measuring device (101) measures one or more of the following parameters from the user: physical activity (102), heart rate (103), heart rate variability, respiratory rate (104), respiratory depth, skin conductivity (105), composition of sweat and chemical compounds of sweat, and processes the vitality level (106, 201) from the data obtained from these parameters and their changes by applying one ormore of the following known technologies: regression analysis, principal component analysis, clustering, machine learning, deep learning, neural network technology, and any other technology used in activity meters for processing vitality levels.

3. An arrangement according to claim lor 2, characterized in that the smartphone (121) is programmed to record and analyze the user's voice and / or facial image (408) at times other than upon receiving an alert from the vitality measuring device (101), to analyze the emotional state (409), and to initiate contact (415) or a mood intervention (615) if the user's emotional state meets the criteria for exceeding the action threshold (413) set for that particular time.

4. An arrangement according to claim 1 or 3, characterized in that the threshold values (405) for the vitality monitoring system are set manually during the implementation phase for each day of the week and each hour of the day, for example, in hourly intervals, and are finetuned automatically during continuous use such that any alert, which does not lead to contact or a mood intervention (414) in the emotional state analysis (409, 413), causes the threshold value to be adjusted, for example, by 1-5% of the entire range of vitality level variations in the direction that decreases the sensitivity of generating future alerts on the same weekday and at the same time, and that any contact or mood intervention (414) initiated by the emotional state analysis (409, 413), which was not triggered by a vitality analysis alert, causes the threshold value to be adjusted, for example, by 1-5% of the entire range of vitality level variations in the direction that increases the sensitivity of generating future alerts on the same weekday and at the same time.

5. An arrangement according to any of claims 1 -4, characterized in that the user is activated to speak and display their facial expressions and gestures to the smartphone camera through text, image or voice messages presented with stimulating questions, comments, or jokes.

6. An arrangement according to any of claims 1-5, characterized in that the analysis of the user's emotional state (409) is carried out by recording the user's voice and / or facial image or image stream with the smartphone's (121) microphone and / or video camera (408) and processing the emotional state using known methods of emotional state recognition.

7. An arrangement according to any of claims 1-6, characterized in that in the analysis of vitality and / or emotional state, in addition to the voice and / or facial images, health and wellness data produced by applications of the smartwatch (101) and smartphone (121) are also used,which may indicate any set of activities and measures of the past day, week, and longer periods such as physical activity, active energy expenditure, resting energy expenditure, heart rate, step count, walking and running distances, standing versus sitting, floors climbed, resting heart rate, heart rate variability, blood oxygen levels, skin conductance, composition of sweat, breathing rate, sleeping time and its distribution into different types of sleep: light sleep, basic sleep, deep NREM sleep, and REM sleep, and which health and wellness data are used, for example, as additional inputs in the fine-tuning of VAD threshold values (412).

8. An arrangement according to any of claims 1-7, characterized in that for determining the suitability value of each contact person and mood intervention, the arrangement is programmed to perform one or more of the following operations:- the arrangement asks the user's opinion on the suitability of each contact person for use in each emotional state, for example, on a scale of 0 to 10,- the arrangement asks the user's opinion on the suitability of each mood intervention for use in each emotional state, for example, on a scale of 0 to 10,- the arrangement monitors changes in the user's emotional state during contacts (421, 422) and stores (425) the contact history in the database (418) for each emotional state with each contact person, recording the direction and magnitude and / or speed of the change, for example, as a percentage of the maximum scale in the VAD coordinate system,- the arrangement monitors changes in the user's emotional state during the use of mood interventions (621, 622) and stores (625) the histories of mood interventions in the database (618) for each emotional state with each mood intervention, recording the direction and magnitude and / or speed of the change, for example, as a percentage of the maximum scale in the VAD coordinate system,- the arrangement asks for the user's opinion (424) on the change in their emotional state after each contact, for example, on a scale of -5 to +5,- the arrangement asks for the user's opinion (624) on the change in their emotional state after each mood intervention, for example, on a scale of -5 to +5, and- the arrangement asks for the opinion of the contact person (424) on the change in the user's emotional state after each contact, for example, on a scale of -5 to +5, and then calculates the suitability value as a weighted average of the results of the selected operations, scaled, for example, between 0 to 100.

9. An arrangement according to any of claims 1-8, characterized in that, in the event that the chosen contact person is not available when contact is initiated, the smartphone (121) is programmed to initiate contact with the person who has the next highest suitability value for the user's emotional state at that particular moment, and for situations where no contact person is available or the user has prioritized mood interventions over contacts, the smartphone (121) is programmed to initiate the mood intervention (530, 617) with the highest suitability value for the user's emotional state at that particular moment.

10. A method for monitoring and supporting a user's emotional state by activating or maintaining interaction with family, friends, caregivers, and other contacts, as well as, through various mood-affecting interventions, wherein the method involves measuring the user's vitality level, for example, with a smartwatch (101), and recognizing the user's emotional state, for example, with a smartphone (121) by recording and analyzing the user’s voice and / or facial images (408, 409), characterized in that- the vitality level measurement is programmed to monitor the user's vitality level (401, 402) and to send an alert to the smartphone (121) when the vitality level falls below a predetermined lower limit for that particular time (406, 203) or when the vitality level rises above a predetermined upper limit for that particular time (406, 202),- the smartphone (121) is programmed, upon receiving the alert, to activate by voice, image or text message, prompting the user to speak and watch the smartphone and, while the user is speaking, to record the user's voice and / or facial image stream (408), to analyze these to determine the user's emotional state (409), and to initiate a contact (415) or a mood intervention (615) if the user's emotional state meets the criteria (413) for exceeding the action threshold set for that moment; and- the smartphone (121) is programmed to establish a communication link with the contact person among those set, for whom the suitability value at that moment, for that user’s particular emotional state, is highest (417); or- the smartphone is programmed to initiate a mood intervention (530) with one of the set mood interventions for which the suitability value at that moment, for that user’s particular emotional state, is highest.

11. A method according to claim 10, characterized in that the determination of the emotional state from voice sound and / or facial video is based on a VAD model, where the dimensions ofthe emotional state form a three-dimensional orthogonal coordinate system (300), whose dimensions are:- Valence (V), which describes the pleasantness of the emotional state (302);- Arousal (A), which describes the intensity of the emotional state (301); and- Dominance (D), which describes the controllability of the emotional state (303), and in which coordinate system the emotional state is defined around the origin within a boundary set by a threshold function (412) that defines a normal and / or neutral state (317, 319, 321), beyond which the action threshold (413) is exceeded.

12. A method according to claim 11 , characterized in that the threshold function defining the normal and / or neutral emotional state is initially set as a sphere (317), representing a constant distance from the origin in all directions, or as a cube (319), whose sides are set at the desired distance from the origin in each of the six directions, and the surface defining the state is allowed to adapt according to a learning algorithm that shapes the surface distance from the origin in each direction, such that each realized contact event or mood intervention produces a maximal positive change in the user's emotional state.

13. A method according to claim 11 , characterized in that the threshold function defining the normal and / or neutral emotional state is initially set for a new user to a state around the origin as it has settled for a similarly profiled user previously using the method, and the surface (321) defining the state is fine-tuned according to a learning algorithm that shapes the surface distance from the origin in each direction, such that each realized contact event or mood action produces a maximal positive change in the user's emotional state.

14. A method according to claim 13 , characterized in that the threshold function defining the normal and / or neutral emotional state is determined during the setup phase so that the user provides information about themselves, such as age, gender, personality type, physical health condition, psychological health condition, accustomed day rhythm, or other mood management- related information, based on which one or more similar users are selected from previous users, and an average threshold function suitable for the user is determined during the setup phase.

15. A method according to any of claims 10-14, characterized in that the analysis of the vitality and / or emotional state employs a learning system operating on the principle of supervised or unsupervised learning, which is programmed and / or trained to automatically discoverthe association between the input data and useful classification, and to search for various vitality and / or emotional state classes in the input data using known learning methods such as Decision tree learning, Artificial neural networks, Bayesian learning, Instance-based learning, Analytical learning, or Reinforcement learning.

16. A method according to claim 10, characterized in that the analysis of emotional states is performed using a discrete model, wherein the emotional space, defined by one or more dimensions, is subdivided into at least two segments per dimension, thereby forming a plurality of distinct emotional states, each uniquely named or numbered, in which the prevailing emotional state of the user at any given moment is classified based on probability values, and wherein each emotional state is associated with predefined suitability values and criteria for initiating contacts and / or mood-related actions when an action threshold is met or exceeded.

17. A method according to any of claims 10-16, characterized in that the mood interventions include a set comprising one or more of the following: mood-affecting music, mood-affecting games, brain exercises, podcasts, videos, virtual therapy services, artificial intelligence therapy services, and other mood-affecting services.

18. A method according to any of claims 10-17, characterized in that the smartphone (121) is programmed and / or trained to analyze the emotional state (409) of a mentally sick or traumatized user during their daily life, with or without concurrent vitality measurements from a smartwatch (101), either continuously or during regularly scheduled intervals; and this analysis identifies changes in the user’s emotional state profile to detect shifts in mental or psychological stability, and initiates a special contact call or a mood intervention designed to facilitate access to therapy or assistance for their symptoms and related issues.

19. A method according to any of claims 10-18, characterized in that the analysis of emotional states is taught to recognize different emotional states in various disease states of diseases, whether psychological or physical, such as the depressive and manic phases of bipolar mood disorder or the depressive and aggressive phases of memory disorders, and that contact is set either empirically or as a result of an automatic learning function to target each emotional state to the contact person or mood action that produces the best change.

20. A method according to any of claims 10-19, characterized in that the analysis of emotional state (409) is programmed and taught to recognize instances of cyberbullying based onRECTIFIED SHEET (RULE 91)changes in the user's emotional state while using the smartphone (121) during daily work or leisure routines, and to initiate a special contact call or a reporting procedure for the cyberbullying event.

21. A method according to any of claims 10-19, characterized in that the analysis of emo- tional state (409) is programmed and trained to recognize instances of overwhelming use of addicting and / or toxic internet content, based on changes in the user's emotional state profile while engaging with short videos, online games, or similar services on the smartphone (121); and to initiate a special contact call or a mood intervention aimed at recovery from the addictive and toxic emotional experiences.RECTIFIED SHEET (RULE 91)

Images