Method, device, electronic device and storage medium for voice interaction

Abstract

The application relates to the technical field of smart homes, and discloses a voice interaction method and device, electronic equipment and a storage medium, which comprise the following steps: detecting emotions held by all persons in a target area; determining one emotion from all the detected emotions as a target emotion; adjusting the timbre of a broadcast voice according to the target emotion; and carrying out voice interaction with the person holding the target emotion in the target area by using the adjusted timbre. The application identifies the emotional states of all persons in the whole target area, realizes the leap from 'perceiving voice' to 'perceiving person', and provides a more natural, more humanized and more reliable man-machine interaction experience for users.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and in particular to a method, device, electronic device, and storage medium for voice interaction. Background Technology

[0002] With the development of artificial intelligence technology, smart home devices with voice assistants are becoming increasingly popular. People can control devices or systems to execute commands or complete question-and-answer dialogues through voice conversations. However, current technologies typically use a single, preset voice tone for voice interaction, resulting in a rather monotonous presentation. To enhance the fun and personalization of the interaction, some devices offer multiple voice tone libraries, requiring users to manually switch between them in the system settings. However, this approach cannot automatically switch voice tones during real-time voice interaction or recommend different voice to different users, failing to provide a truly personalized service. Summary of the Invention

[0003] The purpose of this invention is to provide at least one method, apparatus, electronic device and storage medium for voice interaction, which can at least solve the technical problem that the existing voice interaction devices still lack the ability to provide personalized services, and can at least achieve automatic switching of voice timbre to interact with different people in real-time voice interaction.

[0004] To address the aforementioned technical problems, at least one embodiment of this application provides a method for voice interaction, comprising: To detect the emotions held by individuals within the target area; Identify one emotion as the target emotion from all detected emotions; Adjust the tone of the broadcast voice according to the target emotion; The adjusted timbre is used to conduct voice interaction with individuals within the target area who are experiencing the target emotion.

[0005] At least one embodiment of this application also provides a voice interaction device, comprising: The emotion detection module is used to detect the emotions held by individuals within the target area. The target emotion determination module is used to identify one emotion as the target emotion from all detected emotions. A timbre adjustment module is used to adjust the timbre of the broadcast speech according to the target emotion; An interaction module is used to conduct voice interaction with a person in the target area who is experiencing the target emotion, using an adjusted timbre.

[0006] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described voice interaction method.

[0007] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method of voice interaction.

[0008] The methods, apparatus, electronic devices, and storage media for voice interaction provided in the embodiments of this application enable voice interaction devices to no longer process only a single sound signal, but to recognize the emotional states of everyone within the entire target area, achieving a leap from "perceiving speech" to "perceiving people." From all detected emotions, a target emotion is identified to allocate interaction resources to the user most in need of assistance when resources are limited. The timbre of the broadcast voice is adjusted according to the target emotion, making each interaction "tailor-made" for the specific scenario and emotion, providing a highly personalized experience and avoiding a monotonous, mechanical feel. Ultimately, the interaction behavior is precisely directed to the "person holding the target emotion," providing the responding user with a timely response that matches their emotional state. This application transforms the interaction method from conventional "passive response" to "active intelligence." The system is no longer a simple command executor, but an intelligent agent capable of perceiving the environment, understanding emotions, making decisions, and communicating with empathy, providing users with a more natural, human-centered, and reliable human-computer interaction experience.

[0009] In some optional embodiments, determining an emotion as the target emotion from all detected emotions includes: Based on the emotion weight corresponding to each emotion, the distance weight between each person and the voice interaction device, and / or the age weight corresponding to each person, the emotion priority value corresponding to each person is calculated. The emotion held by the person with the highest emotion priority value is identified as the target emotion.

[0010] In this embodiment, the traditional voice interaction decision-making method is improved from a single dimension (such as who speaks first) to a multi-dimensional one (emotion + distance + age), making the system's decision-making ability more refined and closer to human judgment, and enhancing the system's humanistic care.

[0011] In some optional embodiments, the emotion weight corresponding to each emotion is defined as follows: the greater the emotional urgency, the greater the emotion weight.

[0012] In this embodiment, the emotional weight is positively correlated with the degree of urgency, which endows the artificial intelligence system with "values" and "judgment", ensuring that the interaction is not cold and impersonal, but reliable and trustworthy.

[0013] In some optional embodiments, the distance weight between each person and the voice interaction device is defined as follows: the closer the person is to the voice interaction device, the greater the corresponding distance weight.

[0014] In this embodiment, the system prioritizes responding to commands from nearby users, which can significantly improve the success rate of first-time recognition and reduce repeated interactions or erroneous executions caused by mishearing or missing information. At the same time, this interaction method also conforms to human social intuition and expectations.

[0015] In some optional embodiments, the age weight for each person is defined as follows: when the detected age corresponds to an elderly person or a child, the corresponding age weight is greater than the weight of other age groups.

[0016] In this embodiment, priority is given to the emotional needs of the elderly and children, transforming smart devices from mere tools into empathetic "family partners" that understand respect for the elderly and care for the young.

[0017] In some optional embodiments, the emotion priority value is calculated according to the following formula: Emotional priority = ×Emotional weight ×Age weight+ ×Distance weight; in, and To adjust the coefficient, > , + =1.

[0018] In this embodiment, factors such as emotion, distance, and age are comprehensively considered when calculating the emotion priority, making the final target emotion more natural and fair.

[0019] In some optional embodiments, adjusting the timbre of the broadcast speech according to the target emotion includes: The fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech are adjusted according to the target emotion.

[0020] In this embodiment, at least one of the fundamental frequency, spectrum tilt, and broadcast rhythm is adjusted to make the timbre adjustment more delicate.

[0021] In some optional embodiments, adjusting at least one of the fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech according to the target emotion includes: When the target user's emotion is anxiety, the base frequency is lowered; When the target user's emotion is urgent, the base frequency is increased; When the target user's emotion is loneliness, increase the spectrum tilt. When the target user's emotion is anger, reduce the spectrum tilt. When the target user's emotions reach an alarm level, the broadcast speed is increased.

[0022] In this embodiment, different timbre parameters are adjusted according to different emotions, so that the timbre is no longer monotonous, but more emotionally rich.

[0023] In some optional embodiments, detecting the emotions held by individuals within the target area includes: Millimeter-wave radar is used to detect the emotions held by individuals within a target area.

[0024] In this embodiment, using millimeter-wave radar to detect emotions can accurately detect emotions without excessively collecting users' private information.

[0025] In some alternative embodiments, the detection of the emotions held by individuals within a target area via millimeter-wave radar includes: Millimeter-wave radar is used to acquire vital sign data and / or behavioral dynamic data of individuals within the target area. An emotion quantification value is obtained by analyzing the vital signs data and / or behavioral dynamic data, and the corresponding emotion is determined based on the emotion quantification value.

[0026] In this embodiment, millimeter-wave radar can capture dynamic data of human behavior, such as movement speed, direction, and posture. Through this data, vital signs data and typical behavioral dynamic data of different emotions can be calculated directly or indirectly.

[0027] In some optional embodiments, obtaining the emotion quantification value based on the vital sign data and / or behavioral dynamic data includes at least one of the following steps: An anxiety index is obtained based on the entropy of respiratory signal samples in the vital signs data, the low-frequency / high-frequency power ratio of heart rate variability in the vital signs data, and the energy-weighted fusion of microlimb tremors in the behavioral dynamics data. The loneliness index is determined based on the duration of time spent facing away from the device and the density of human voices in the environment from the aforementioned behavioral dynamic data; The anger index is obtained by weighting and fusing the surface blood flow acceleration in the vital signs data and the attack probability in the behavioral dynamics data.

[0028] This embodiment integrates information from multiple dimensions, including vital sign data (such as respiratory signal sample entropy, low-frequency / high-frequency power ratio of heart rate variability, and surface blood flow acceleration) and behavioral dynamic data (such as microlimb tremor energy, duration of facing away from the device, ambient sound density, and attack probability). Different types of data reflect the physiological state and behavioral characteristics of the human body from different perspectives. Weighted fusion of these data can more comprehensively and accurately capture emotion-related signals, avoiding the bias and errors that may be caused by a single data source, and improving the accuracy and reliability of emotion recognition.

[0029] In some optional embodiments, the method further includes: Based on the target emotion, a matching sound effect enhancement method is determined, and the timbre of the broadcast speech is enhanced with sound effects.

[0030] In this embodiment, sound enhancement can create a more realistic and immersive atmosphere, better soothing the user's emotions.

[0031] In some optional embodiments, the sound effect enhancement processing of the broadcast speech includes: When the target emotion is loneliness, the adjusted timbre is enhanced with reverb. When the target emotion is fear, the adjusted timbre is enhanced with penetration.

[0032] In this embodiment, reverberation causes sound to reflect and delay, creating a rich echo effect that helps dispel feelings of loneliness, while penetration enhancement makes the sound clearer and more powerful, helping to dispel feelings of fear. Attached Figure Description

[0033] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0034] Figure 1 This is a flowchart of a voice interaction method provided in one embodiment of this application; Figure 2 This is a system architecture diagram of a voice interaction system provided in one embodiment of this application; Figure 3 This is a schematic diagram illustrating the types of data acquired by a millimeter-wave radar according to another embodiment of this application; Figure 4 This is a schematic diagram of the decision logic of the decision layer of a voice interaction system provided in another embodiment of this application; Figure 5 This is a schematic diagram of the execution logic of the execution layer of a voice interaction system provided in another embodiment of this application; Figure 6 This is a schematic diagram of a voice interaction device provided in another embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0036] To address the technical problem that existing voice interaction devices still lack the ability to provide personalized services, this invention proposes a voice interaction method. The implementation details of the voice interaction method in this embodiment are described below. The following content is only for ease of understanding and is not necessary for implementing this solution.

[0037] Example 1: The voice interaction method of this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes: Step 110: Detect the emotions held by each person in the target area; Specifically, voice interaction devices can proactively identify the emotions of people within a target area. For example, a camera can capture facial images of people within the target area, and computer vision algorithms can analyze facial muscle movements and feature point changes to identify different emotions. Alternatively, a microphone can collect speech signals from people within the target area and analyze the acoustic features of the speech, such as pitch, intensity, speech rate, and timbre. Different emotions cause these acoustic features to change; for example, anger results in faster speech and higher pitch, while calmness results in moderate speech rate and stable pitch. Machine learning algorithms, such as support vector machines and decision trees, can be used to model and classify the acoustic features of speech to identify emotions.

[0038] The target area can be determined based on the actual application scenario. For example, in a home, the target area could be the living room area or the maximum range that the emotion recognition device can detect; in a shopping mall scenario, the target area could be a specific product display area.

[0039] Step 120: Identify one emotion as the target emotion from all detected emotions; Specifically, the target emotion is the basis for timbre adjustment in this voice interaction. There are several ways to determine the target emotion: In some embodiments, a priority principle is adopted: a priority order is established based on the importance and urgency of different emotions. For example, in security monitoring scenarios, emotions such as fear and tension may have higher priority. When these emotions are detected, they are prioritized as target emotions so that appropriate measures can be taken in a timely manner.

[0040] In some embodiments, the majority principle can also be used: count the number of people experiencing each emotion within the target area, and select the emotion with the most participants as the target emotion. For example, if two people are detected as happy, one as sad, and one as angry, then according to the majority principle, happy is selected as the target emotion.

[0041] In some embodiments, the target emotion can be selected based on the specific application scenario. For example, in entertainment venues, emotions such as joy and excitement are more in line with the atmosphere and can be considered as the primary target emotion.

[0042] Step 130: Adjust the timbre of the broadcast voice according to the target emotion; Specifically, this embodiment utilizes text-to-speech (TTS) technology to dynamically adjust the timbre characteristics of the system's output speech to match the "target emotion," thereby achieving empathy and enhancing the naturalness of the interaction and user experience. Matching means that the adjusted timbre can soothe or positively guide the target emotion. For example, if the target emotion is sadness, the adjusted timbre can alleviate this sadness, such as by becoming gentler. If the target emotion is happiness, the adjusted timbre becomes more cheerful to further enhance the user's happy emotion.

[0043] In some optional embodiments, a set of timbre parameters is pre-set for each target emotion and directly invoked. In other optional embodiments, timbre enhancement can be achieved by combining the user's personalized settings and historical preferences. For example, the system can record the timbre types that the user prefers under different emotions, and automatically adjust the timbre based on the target emotion and user preferences during the next voice interaction.

[0044] Step 140: Use the adjusted timbre to conduct voice interaction with the person in the target area who is experiencing the target emotion.

[0045] Specifically, the interaction process can generate targeted voice interaction content based on the target's emotion. For example, when the target's emotion is happy, the voice content can be positive encouragement, blessings, or sharing more interesting information; when the target's emotion is sad, the voice content can be comforting, supportive, or providing solutions to problems.

[0046] In some optional embodiments, during voice interaction, the voice interaction device needs to respond to user feedback and emotional changes in real time, adjusting the interaction strategy and tone accordingly. For example, if a user shows impatience during voice interaction, the voice interaction device can speed up the speech and simplify the content to improve interaction efficiency.

[0047] In some optional embodiments, the voice interaction device detects the emotions of each person in the target area in real time, continuously determines the target emotion, and readjusts the tone of the broadcast based on the newly determined target emotion.

[0048] In summary, the voice interaction device in this embodiment no longer processes only a single sound signal, but instead identifies the emotional state of everyone within the entire target area, achieving a leap from "perceiving voice" to "perceiving people." It identifies a target emotion from all detected emotions to allocate interaction resources to the user most in need of help, given limited resources. The tone of the broadcast voice is adjusted according to the target emotion, ensuring that each interaction is "tailor-made" for the specific scenario and emotion, providing a highly personalized experience and avoiding a monotonous, mechanical feel. Ultimately, the interaction is precisely directed to the "person holding the target emotion," providing the responding user with a timely and emotionally appropriate response. This application transforms the interaction method from conventional "passive response" to "active intelligence." The system is no longer a simple command executor, but an intelligent agent capable of perceiving the environment, understanding emotions, making decisions, and communicating with empathy, providing users with a more natural, human-centered, and reliable human-computer interaction experience.

[0049] In some embodiments, determining an emotion as the target emotion from all detected emotions includes: calculating an emotion priority value for each person based on the emotion weight corresponding to each emotion, combined with the distance weight between each person and the voice interaction device, and / or the age weight corresponding to each person; and determining the emotion held by the person with the highest emotion priority value as the target emotion.

[0050] Specifically, in one embodiment, the target emotion can be determined solely based on emotion weights. For example, if there are two people in the target area, A (anxious) and B (happy), and the weight of anxiety is greater than that of happiness, then anxiety is chosen as the target emotion, and the voice interaction device interacts with person A.

[0051] In one embodiment, the target emotion is determined by combining the emotion weight corresponding to each emotion with the distance weight between each person and the voice interaction device. For example, if there are people A (anxious) and people B (happy) in the target area, the distance weight between person B and the voice interaction device is greater than that between person A and the voice interaction device. After comprehensively considering both the emotion weight and the distance weight, the emotion priority value of person B is greater than that of person A. Therefore, the emotion of happiness is chosen as the target emotion, and the voice interaction device interacts with person B.

[0052] In one embodiment, the target emotion is determined by combining the emotion weight corresponding to each emotion with the age weight corresponding to each person. For example, in the target area, there are person A (anxious) and person B (happy). Person A is a middle-aged man, and person B is an elderly person. After combining the emotion weight and the age weight, the emotion priority value of person B is greater than that of person A. Therefore, the emotion of happiness is selected as the target emotion, and the voice interaction device interacts with person B.

[0053] In one embodiment, the target emotion is determined by combining the emotional weight corresponding to each emotion with the distance weight between each person and the voice interaction device, and the age weight corresponding to each person. The more factors are considered, the more refined and human-like the voice interaction device's decision-making becomes when resolving multi-user conflicts, providing users with a more humanized interactive experience.

[0054] In this embodiment, the traditional voice interaction decision-making method is improved from a single dimension (such as who speaks first) to a multi-dimensional one (emotion + distance + age), making the system's decision-making ability more refined and closer to human judgment, and enhancing the system's humanistic care.

[0055] In some optional embodiments, the emotion weight corresponding to each emotion is defined as follows: the greater the emotional urgency, the greater the emotion weight.

[0056] Specifically, the emotional priority can be alarm > emergency > anger > fear > anxiety > loneliness > sadness > happiness, following the principle of safety first. The emotional priority order in this embodiment is merely an example and can be modified according to actual needs in practical applications; no restrictions are imposed here. For example, the priority of fear can be set to be greater than that of anger.

[0057] In this embodiment, the emotional weight is positively correlated with the degree of urgency, which endows the artificial intelligence system with "values" and "judgment", ensuring that the interaction is not cold and impersonal, but reliable and trustworthy.

[0058] In some optional embodiments, the distance weight between each person and the voice interaction device is defined as follows: the closer the person is to the voice interaction device, the greater the corresponding distance weight.

[0059] Specifically, people closer to the voice interaction device are more likely to engage in real-time and effective interaction with the system. Prioritizing the emotions of these individuals can improve the efficiency and quality of interaction, better meeting user needs.

[0060] In some alternative embodiments, an inverse proportional function can be used to calculate the distance weights. Let the distance between the person and the voice interaction device be d, and the maximum detection distance be... Then distance weight It can be represented as: ; When d=0 =1 indicates the closest distance and the largest weight; when d= hour, =0 indicates the furthest distance and the smallest weight.

[0061] In some optional embodiments, the distance can be divided into several intervals according to the actual situation, and a fixed distance weight value can be set for each interval. For example, the distance can be divided into three intervals: near, medium, and far, and assigned distance weights of 0.8, 0.5, and 0.2 respectively.

[0062] In this embodiment, the system prioritizes responding to commands from nearby users, which can significantly improve the success rate of first-time recognition and reduce repeated interactions or erroneous executions caused by mishearing or missing information. At the same time, this interaction method also conforms to human social intuition and expectations.

[0063] In some optional embodiments, the age weight for each person is defined as follows: when the detected age corresponds to an elderly person or a child, the corresponding age weight is greater than the weight of other age groups.

[0064] Specifically, people of different ages differ in their cognitive levels and expressive abilities. Children and the elderly express emotions differently than adults. Therefore, when identifying target emotions, age factors need to be considered, and appropriate weighting should be applied to specific age groups. For example, children's emotions may be more easily influenced by the external environment and are expressed more directly; the system can assign a higher weight to children's emotions.

[0065] In some optional embodiments, different age weight values ​​can be set according to age group divisions. For example, the age groups can be divided into children (0-12 years old), teenagers (13-18 years old), adults (19-60 years old), and the elderly (60 years old and above), and assigned age weights of 1.2, 1, 0.8, and 1.2 respectively.

[0066] In some alternative embodiments, a linear function can be used to calculate the age weights. Let the age of the person be 'a', and the slope and intercept of the function are determined based on the needs of different age groups to calculate the corresponding age weights. .

[0067] In this embodiment, priority is given to the emotional needs of the elderly and children, transforming smart devices from mere tools into empathetic "family partners" that understand respect for the elderly and care for the young.

[0068] In some optional embodiments, the emotion priority value is calculated according to the following formula: Emotional priority = ×Emotional weight ×Age weight+ ×Distance weight; in, and To adjust the coefficient, > , + =1.

[0069] In this embodiment, factors such as emotion, distance, and age are comprehensively considered when calculating the emotion priority, making the final target emotion more natural and fair.

[0070] In some optional embodiments, adjusting the timbre of the broadcast speech according to the target emotion includes: adjusting at least one of the fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech according to the target emotion.

[0071] Specifically, the fundamental frequency refers to the basic frequency of vocal cord vibration, perceptually manifested as the pitch of a sound, and is key to conveying questions, affirmations, emphasis, and emotions. When the fundamental frequency is high, the sound sounds bright, lively, and energetic. Lowering the fundamental frequency makes the sound sound low, monotonous, and lacking energy, similar to a person speaking weakly and in a flat tone when in low spirits. Spectral tilt refers to the ratio of low-frequency energy to high-frequency energy in the speech signal spectrum, directly affecting the "tone quality" or "color" of the sound. Increasing the negative tilt, that is, weakening high-frequency energy and enhancing low-frequency resonance, makes the sound deeper and softer, producing an intimate and soothing effect.

[0072] In this embodiment, adjustments to at least one of the fundamental frequency, spectral tilt, and broadcast rhythm result in more nuanced timbre tuning. By precisely and collaboratively controlling these acoustic parameters, the speech synthesis system can transcend "mechanical broadcasting" and achieve truly "warm" voice interaction that matches the user's emotional state.

[0073] In some optional embodiments, adjusting at least one of the fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech according to the target emotion includes: When the target user's emotion is anxiety, the base frequency is lowered; When the target user's emotion is urgent, the base frequency is increased; When the target user's emotion is loneliness, increase the spectrum tilt. When the target user's emotion is anger, reduce the spectrum tilt. When the target user's emotions reach an alarm level, the broadcast speed is increased.

[0074] In this embodiment, different timbre parameters are adjusted according to different emotions, so that the timbre is no longer monotonous, but more emotionally rich.

[0075] In some optional embodiments, detecting the emotions held by individuals within the target area includes detecting the emotions held by individuals within the target area using millimeter-wave radar.

[0076] In this embodiment, millimeter-wave radar is used to acquire minute limb movements of the user, such as finger twitches, facial expressions, breathing patterns, and heart rate. Head orientation can also be determined by identifying the user's facial orientation using a 3D point cloud image. Simultaneously, the distance between the user and the voice interaction device can also be detected by millimeter-wave radar. In other words, millimeter-wave radar can acquire sufficient information for emotion recognition. Furthermore, millimeter-wave radar does not excessively collect the user's private information.

[0077] In some optional embodiments, the step of detecting the emotions held by individuals within a target area using millimeter-wave radar includes: acquiring vital sign data and / or behavioral dynamic data of individuals within the target area using millimeter-wave radar; parsing the vital sign data and / or behavioral dynamic data to obtain an emotion quantification value; and determining the corresponding emotion based on the emotion quantification value.

[0078] In this embodiment, millimeter-wave radar can capture dynamic data of human behavior, such as movement speed, direction, and posture. Through this data, vital signs data and typical behavioral dynamic data of different emotions can be calculated directly or indirectly.

[0079] In some optional embodiments, obtaining the emotion quantification value based on the vital sign data and / or behavioral dynamic data includes at least one of the following steps: An anxiety index is obtained based on the entropy of respiratory signal samples in the vital signs data, the low-frequency / high-frequency power ratio of heart rate variability in the vital signs data, and the energy-weighted fusion of microlimb tremors in the behavioral dynamics data. The loneliness index is determined based on the duration of time spent facing away from the device and the density of human voices in the environment from the aforementioned behavioral dynamic data; The anger index is obtained by weighting and fusing the surface blood flow acceleration in the vital signs data and the attack probability in the behavioral dynamics data.

[0080] Specifically, in one embodiment, the anxiety index is calculated according to the following formula: Anxiety index = 0.4 × SampEn(R) + 0.3 × log(LF / HF) + 0.3 × E_tremor; Wherein, SampEn(R) is the respiratory signal sample entropy, LF / HF is the low-frequency / high-frequency power ratio of heart rate variability, and E_tremor is the energy of microlimb tremors.

[0081] Anxiety assessment: The anxiety index is calculated and mapped to a range of 0-100. A score >75 indicates anxiety.

[0082] In one embodiment, the loneliness index is calculated according to the following formula: Loneliness Index = min(1, T_back / 300) × (1 - N_voice / 5); Where T_back is the duration of time the device is turned away from it, and N_voice is the ambient sound density.

[0083] In this embodiment, the duration of time spent facing away from the device reflects the degree of social avoidance, while the density of ambient human voices reflects social activity. N_voice refers to the number of independent human voice events detected in the past 5 minutes, mapped to a range of 0-5. (1 - N_voice / 5) indicates that fewer human voices indicate a higher loneliness factor (5 times is the saturation value). It should be noted that the upper limit of 5 from 0-5 is not absolute; 5 represents a critical value for the duration of silence indicating loneliness. Here, 5 is a general value and can be adjusted according to actual circumstances.

[0084] The detection of human voice events is primarily achieved through the Voice Activity Detection (VAD) algorithm, which detects the number of sentences spoken by people in the environment, representing the number of human voice events. The mapping method is as follows: For the number of human voice events detected above, if the number of detections is less than 5, it can be mapped to N_voice=0; 5~15 times can be mapped to 1; 15~25 times can be mapped to 2; 35~45 times can be mapped to 4; and more than 45 times can be mapped to 5. This is a mapping example, and the specific correspondence between the number of detections and the value of N_voice can be determined according to the actual situation.

[0085] N_voice=0 indicates that there are few human voices in the environment, which can be considered as people being relatively lonely; N_voice=5 indicates that the environment is very lively, with people constantly talking, and people being relatively active.

[0086] Loneliness assessment: Loneliness is determined only when both "avoidance behavior" and "quiet environment" are met. A dynamic threshold is used: the daytime threshold is 0.8, meaning a loneliness index less than 0.8 indicates loneliness; the nighttime threshold automatically drops to 0.6 (10 PM - 6 AM), meaning a loneliness index less than 0.8 indicates loneliness.

[0087] In one embodiment, the anger index is calculated according to the following formula: Anger Index = 0.6 × d(blood_flow) / dt + 0.4 × F_clench; Where d(blood_flow) / dt is the acceleration of blood flow on the body surface, and F_clench represents the attack probability.

[0088] Specifically, when emotionally agitated, the minute movements of the body surface (hands, arms, etc.) caused by heartbeats are captured by millimeter-wave signals to indirectly assess blood flow velocity.

[0089] Millimeter-wave radar detects subtle human movements, such as: increased breathing, accelerated heartbeat, trembling, taut lips, staring or glaring, bulging eyes, raised upper eyelids, lowered and converged eyebrows, and flared nostrils. The more of these movements detected, the higher the level of anger and the greater the probability of aggression.

[0090] For example, if two of the above micro-actions are detected, the attack probability is 20%; if five are detected, the attack probability is 50%. That is, the number of matches detected by micro-actions corresponds to the probability of an attack, and the matching relationship between the number of detections and the probability can be determined according to the actual situation.

[0091] This embodiment integrates information from multiple dimensions, including vital sign data (such as respiratory signal sample entropy, low-frequency / high-frequency power ratio of heart rate variability, and surface blood flow acceleration) and behavioral dynamic data (such as microlimb tremor energy, duration of facing away from the device, ambient sound density, and attack probability). Different types of data reflect the physiological state and behavioral characteristics of the human body from different perspectives. Weighted fusion of these data can more comprehensively and accurately capture emotion-related signals, avoiding the bias and errors that may be caused by a single data source, and improving the accuracy and reliability of emotion recognition.

[0092] In some optional embodiments, the method further includes: determining a matching sound effect enhancement method based on the target emotion, and performing sound effect enhancement processing on the timbre of the broadcast speech. In this embodiment, the sound effect enhancement method can further create a more realistic and immersive atmosphere, better soothing the user's emotions.

[0093] In some optional embodiments, the sound effect enhancement processing of the timbre of the broadcast speech includes: when the target emotion is loneliness, performing reverberation enhancement processing on the adjusted timbre; when the target emotion is fear, performing penetration enhancement processing on the adjusted timbre.

[0094] In this embodiment, reverberation causes sound to reflect and delay, creating a rich echo effect that helps dispel feelings of loneliness, while penetration enhancement makes the sound clearer and more powerful, helping to dispel feelings of fear.

[0095] Example 2: Based on the above embodiments, this embodiment provides an application example. This embodiment provides a voice interaction system, and the system architecture diagram of the system is as follows. Figure 2 As shown, it is divided into a perception layer, a decision-making layer, and an execution layer. The core of the perception layer is to identify the emotions held by people in the target area through millimeter-wave radar, microphone arrays, and environmental sensors. The decision-making layer is used to determine the target emotion among multiple emotions, and the execution layer is used to adjust the tone of voice according to the target emotion.

[0096] Specifically, such as Figure 3 The diagram illustrates the types of data acquired by millimeter-wave radar. Millimeter-wave radar primarily collects three types of data: vital sign data, behavioral dynamic data, and spatial perception data. This data is then input into the decision-making layer for processing. Figure 4 The diagram illustrates the decision-making logic of the decision-making layer. The decision-making layer determines the target emotion through a pre-defined emotion quantification model, a scenario rule base, and a user arbitration base. Figure 5 The diagram illustrates the execution logic of the execution layer. Based on the target emotion, the execution layer finely adjusts the timbre by modifying the fundamental frequency, spectrum tilt, and speech rate (rhythm) to match the target emotion. Then, the sound effects enhancement module further enhances the perceived effect of the timbre, enabling interaction with individuals exhibiting the target emotion.

[0097] The specific methods of voice interaction are as follows: Step 1: Acquire multimodal data using millimeter-wave radar Specifically, a 77GHz FMCW radar was used to acquire multi-dimensional raw data: Vital signs: respiratory rate (0.1-0.5 Hz), heart rate variability (HRV). Behavioral intention: head orientation (to determine the focus of attention), frequency of micro-movements (used for anxiety index calculation). Spatial relationship: The distance between the user and the device (0.1-5m) and their relative position in a multi-person scenario.

[0098] Step 2: Multimodal Emotion Feature Extraction Quantitative analysis was performed using the data collected in step 1: 2.1 Emotion Quantification The emotion quantification model makes the following quantitative judgments: 2.1.1 Anxiety Index = 0.4 × SampEn(R) + 0.3 × log(LF / HF) + 0.3 × E_tremor; Where SampEn(R) represents the respiratory signal sample entropy (respiratory irregularity); LF / HF is the low-frequency / high-frequency power ratio of heart rate variability (sympathetic / parasympathetic balance); E_tremor is the microlimb tremor energy (muscle tension); weighted fusion: respiratory entropy (40%), heart rate variability (30%), tremor energy (30%). Result Interpretation: The anxiety index calculation result is mapped to the 0-100 range, and >75 is judged as anxiety.

[0099] 2.1.2 Loneliness Index = min(1, T_back / 300) × (1 - N_voice / 5); T_back represents the duration of time with back to the device (degree of social avoidance), and N_voice represents the density of human voices in the environment (social activity); min(1, T_back / 300): the duration of time with back to the device exceeds 5 minutes and is calculated as the maximum value of 1; (1 - N_voice / 5): the fewer human voices, the higher the loneliness factor (5 times is the saturation value); Result determination: Loneliness is determined only when both "avoidance behavior" and "environmental silence" are met. The dynamic threshold is adjusted, with a daytime threshold of 0.8 and a nighttime threshold that automatically drops to 0.6 (22:00-6:00).

[0100] 2.1.3 Anger Index = 0.6 × d(blood_flow) / dt + 0.4 × F_clench; d(blood_flow) / dt represents the acceleration of blood flow to the skin surface (adrenaline surge); F_clench represents the probability of attack; the weighted average is blood flow acceleration (60%): directly reflecting the degree of sympathetic nerve activation, and behavioral characteristics (40%): posture recognition as an auxiliary verification. It captures sudden changes in blood flow velocity (which can increase by 200% within 0.5 seconds when angry), and an anger index greater than 60 immediately triggers a de-escalation protocol (to prevent emotional escalation). For example: a blood flow acceleration of 3.2 mm / s is detected. If the probability of clenching a fist is 0.9, then the anger value is 0.6×96 + 0.4×90 = 93.6. Immediately switch to a calm tone and activate the fragrance to cool down.

[0101] 2.2 Scene Judgment Based on environmental monitoring data, scenario analysis is performed. For example, if the judgment time is between 23:00 and t ≤ 6:00 and anxiety level > 75, then sleep mode is activated. If a fall is detected and heart rate > 120, then an emergency protocol is activated. If an external trigger signal, such as a smoke sensor, is detected, then an alarm protocol is activated.

[0102] 2.3 User Priority Arbitration To meet the needs of multi-user conflict resolution: In multi-user scenarios, the priority judgment rule is as follows: Emotional priority value = 0.7 × emotional weight × age weight + 0.3 × distance weight, where the age weight is 1.2 when the person is an elderly person or a child.

[0103] When multiple users experience emotional conflict, the order of urgency is: medical emergency > security alert > emotional need. Specifically, the interaction between people and devices in this invention does not necessarily require the person to initiate it. When the device detects a person with the highest priority, it will respond. That is, it does not necessarily require user interaction with the device, nor does it require the user to initiate it. The device detects the person with the highest priority in the current scene in real time, and adjusts the tone of the audio played by the device based on that person's emotions, and selects a soothing reassurance message corresponding to that emotion.

[0104] Example scenario: Dad: Anxiety = 70 (Preparing for an urgent meeting due to work requirements) Mom: Happiness = 60 (Putting on makeup, getting ready to go shopping) Child: Anger = 60 (Can't put the toy together, throws a tantrum) If the emotional priority score is calculated as "child > father > mother", then the system will prioritize soothing the child's emotions. Voice quality: Use a lively cartoon voice (cheerful tone). Content: "Hey little guy! Having trouble with something? Want me to help you find a puzzle guide? Or how about we listen to a Power Rangers story to relax?" The system continuously monitors and calculates the priority of people in the scene, and provides emotional support to the person with the highest priority (through voice adjustment, broadcast content, and scene-based coordination). 2.4 Rule base update The emotion decision engine rule base is updated in real time through user feedback received via voice or the app.

[0105] Step 3: Dynamic rendering of timbre parameters 3.1. Based on the quantization results from step 2, adjust the timbre matrix parameters: Example as follows: If anxiety is detected, adjust the base frequency to -15% (male voice 140→119Hz). If the situation is deemed urgent, adjust the base frequency to +30% (female voice 200→260Hz). If the condition is determined to be loneliness, the spectrum tilt will be adjusted to +12dB / oct tilt (warmth). If the judgment is anger: then adjust the spectrum tilt to: -6dB / oct tilt (coldness); If an alarm is detected, adjust the pace to: 4.8 syllables / second. The priority of emotions is determined by the principle of safety first (alarm > emergency > anger > anxiety > loneliness).

[0106] 3.2 After determining the parameters, speech synthesis is performed.

[0107] 3.3 After speech synthesis, the effect is enhanced through a sound effects enhancement module, as shown in the following example: Judged as loneliness: +0.5s hall reverberation; The cause of fear is identified as +3kHz harmonics (enhanced penetration). If you suspect anxiety: Add a pure tone or very gentle Brownian noise in the background at 80-120Hz to alleviate anxiety with a smooth, slow, low-frequency hum. Judged as anger: +broad, deep ambient sound pads, extended dynamic range.

[0108] 3.4 The final audio will be played through the device's speakers, while the home scene will be linked to enhance the atmosphere.

[0109] Example 3: Another embodiment of this application relates to a voice interaction device. The implementation details of the voice interaction device of this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. The schematic diagram of the voice interaction device of this embodiment can be seen as follows: Figure 6 As shown, it includes an emotion detection module 610, a target emotion determination module 620, a voice adjustment module 630, and an interaction module 640.

[0110] The emotion detection module 610 is used to detect the emotions held by individuals within the target area. The target emotion determination module 620 is used to determine one emotion as the target emotion from all detected emotions; The timbre adjustment module 630 is used to adjust the timbre of the broadcast speech according to the target emotion; The interaction module 640 is used to conduct voice interaction with a person in the target area who is experiencing the target emotion, using an adjusted timbre.

[0111] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0112] In some optional embodiments, the target emotion determination module includes: The emotion priority value calculation unit is used to calculate the emotion priority value of each person based on the emotion weight corresponding to each emotion, combined with the distance weight between each person and the voice interaction device, and / or the age weight corresponding to each person. The target emotion determination unit is used to determine the emotion held by the person with the highest emotion priority value as the target emotion.

[0113] In some optional embodiments, the tone adjustment module is used for: The fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech are adjusted according to the target emotion.

[0114] In some optional embodiments, the tone adjustment module is specifically used for: When the target user's emotion is anxiety, the base frequency is lowered; When the target user's emotion is urgent, the base frequency is increased; When the target user's emotion is loneliness, increase the spectrum tilt. When the target user's emotion is anger, reduce the spectrum tilt. When the target user's emotions reach an alarm level, the broadcast speed is increased.

[0115] In some optional embodiments, the emotion detection module is used to: Millimeter-wave radar is used to detect the emotions held by individuals within a target area.

[0116] In some optional embodiments, the emotion detection module includes: The data acquisition unit is used to acquire vital sign data and / or behavioral dynamic data of individuals within the target area using millimeter-wave radar. The emotion quantification unit is used to parse the vital sign data and / or behavioral dynamic data to obtain an emotion quantification value, and to determine the corresponding emotion based on the emotion quantification value.

[0117] In some alternative embodiments, the emotion quantification unit is used to achieve one of the following: An anxiety index is obtained based on the entropy of respiratory signal samples in the vital signs data, the low-frequency / high-frequency power ratio of heart rate variability in the vital signs data, and the energy-weighted fusion of microlimb tremors in the behavioral dynamics data. The loneliness index is determined based on the duration of time spent facing away from the device and the density of human voices in the environment from the aforementioned behavioral dynamic data; The anger index is obtained by weighting and fusing the surface blood flow acceleration in the vital signs data and the attack probability in the behavioral dynamics data.

[0118] In some alternative embodiments, the apparatus further includes: The sound effect enhancement module is used to determine a matching sound effect enhancement method based on the target emotion and to perform sound effect enhancement processing on the timbre of the broadcast speech.

[0119] In some optional embodiments, the sound enhancement module is specifically used for: When the target emotion is loneliness, the adjusted timbre is enhanced with reverb. When the target emotion is fear, the adjusted timbre is enhanced with penetration.

[0120] Example 4: Another embodiment of this application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the voice interaction methods in the above embodiments.

[0121] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0122] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0123] Example 5: Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0124] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for voice interaction, characterized in that, include: To detect the emotions held by individuals within the target area; Identify one emotion as the target emotion from all detected emotions; Adjust the tone of the broadcast voice according to the target emotion; The adjusted timbre is used to conduct voice interaction with individuals within the target area who are experiencing the target emotion; The step of identifying a target emotion from all detected emotions includes: Based on the emotion weight corresponding to each emotion, the distance weight between each person and the voice interaction device, and / or the age weight corresponding to each person, the emotion priority value corresponding to each person is calculated. The emotion held by the person with the highest emotion priority value is identified as the target emotion; The emotional weight for each emotion is defined as follows: the greater the severity of the emotion, the greater the emotional weight. The distance weight between each person and the voice interaction device is defined as follows: the closer the person is to the voice interaction device, the greater the corresponding distance weight. The age weight for each person is defined as follows: when the detected age corresponds to the elderly or children, the corresponding age weight is greater than the weight of other age groups. The emotion priority value is calculated using the following formula: Emotional priority = ×Emotional weight ×Age weight+ ×Distance weight; in, and To adjust the coefficient, > , + =1.

2. The voice interaction method according to claim 1, characterized in that, The step of adjusting the tone of the broadcast voice according to the target emotion includes: The fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech are adjusted according to the target emotion.

3. The voice interaction method according to claim 2, characterized in that, The adjustment of at least one of the fundamental frequency, spectral tilt, and broadcast rhythm of the broadcast speech according to the target emotion includes: When the target user's emotion is anxiety, lower the base frequency; When the target user's emotion is urgent, increase the base frequency; Increase the spectrum tilt when the target user's emotion is loneliness; Reduce the spectrum tilt when the target user's emotion is anger; When the target user's emotions reach an alarm level, the broadcast speed is increased.

4. The voice interaction method according to claim 1, characterized in that, The emotions held by individuals within the target detection area include: Millimeter-wave radar is used to detect the emotions held by individuals within a target area.

5. The voice interaction method according to claim 4, characterized in that, The method of detecting the emotions held by individuals within a target area using millimeter-wave radar includes: Millimeter-wave radar is used to acquire vital sign data and / or behavioral dynamic data of individuals within the target area. An emotion quantification value is obtained by analyzing the vital signs data and / or behavioral dynamic data, and the corresponding emotion is determined based on the emotion quantification value.

6. The voice interaction method according to claim 5, characterized in that, The process of obtaining an emotion quantification value based on the vital sign data and / or behavioral dynamic data includes at least one of the following steps: An anxiety index is obtained based on the entropy of respiratory signal samples in the vital signs data, the low-frequency / high-frequency power ratio of heart rate variability in the vital signs data, and the energy-weighted fusion of microlimb tremors in the behavioral dynamics data. The loneliness index is determined based on the duration of time spent facing away from the device and the density of human voices in the environment from the aforementioned behavioral dynamic data; The anger index is obtained by weighting and fusing the surface blood flow acceleration in the vital signs data and the attack probability in the behavioral dynamics data.

7. The voice interaction method according to any one of claims 1-6, characterized in that, The method further includes: Based on the target emotion, a matching sound effect enhancement method is determined, and the timbre of the broadcast speech is enhanced with sound effects.

8. The voice interaction method according to claim 7, characterized in that, The sound enhancement processing for the timbre of the broadcast speech includes: When the target emotion is loneliness, the adjusted timbre is enhanced with reverb. When the target emotion is fear, the adjusted timbre is enhanced with penetration.

9. An apparatus for implementing the voice interaction method according to any one of claims 1-8, characterized in that, include: The emotion detection module is used to detect the emotions held by individuals within the target area. The target emotion determination module is used to identify one emotion as the target emotion from all detected emotions. A timbre adjustment module is used to adjust the timbre of the broadcast speech according to the target emotion; The interaction module is used to conduct voice interaction with people in the target area who are experiencing the target emotion, using an adjusted timbre; The device is also used for: Based on the emotion weight corresponding to each emotion, the distance weight between each person and the voice interaction device, and / or the age weight corresponding to each person, the emotion priority value corresponding to each person is calculated. The emotion held by the person with the highest emotion priority value is identified as the target emotion; The emotional weight for each emotion is defined as follows: the greater the emotional urgency, the greater the emotional weight. The distance weight between each person and the voice interaction device is defined as follows: the closer the person is to the voice interaction device, the greater the corresponding distance weight; the age weight of each person is defined as follows: when the detected age corresponds to the elderly or children, the corresponding age weight is greater than the weight of other age groups. The emotion priority value is calculated using the following formula: Emotional priority = ×Emotional weight ×Age weight+ ×Distance weight; in, and To adjust the coefficient, > , + =1.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the voice interaction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the voice interaction method according to any one of claims 1 to 8.

Images