Smart home control method and system based on emotion recognition

Abstract

The application discloses an intelligent home control method and system based on emotion recognition, relates to the technical field of intelligent home control, and discloses the intelligent home control method and system based on emotion recognition, which acquires multi-modal emotion data, performs context perception fusion analysis, verifies historical modes, determines a hierarchical response strategy based on credibility, accurately controls intelligent home equipment, can improve the accuracy of emotion recognition, realizes the hierarchical response strategy, avoids user misoperation, and improves the use experience of users.

Description

Technical Field

[0001] This application relates to the field of smart home control technology, and in particular to a smart home control method and system based on emotion recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, emotion-based smart home systems are gradually being integrated into daily life. Existing solutions typically rely on sensors such as cameras, microphones, and wearable devices to collect multimodal data on users' facial expressions, voice tone, and physiological signals. These data are then analyzed using pre-set algorithms to determine emotional states and trigger corresponding adjustments to the home environment. However, in real-world home environments, these systems exhibit serious technical flaws: the emotion recognition process is affected by individual differences, changes in ambient lighting, background noise, and fluctuations in physiological signals, leading to significant uncertainty in the recognition results. For example, when a user is immersed in a work environment, focused facial micro-expressions can easily be misinterpreted as irritability; while watching a film, tears evoked by emotion are often incorrectly identified as sadness. These misinterpretations directly lead to inappropriate control commands, such as the system suddenly dimming the lights, playing soothing music, or lowering the room temperature while the user is watching a movie, forcibly interrupting normal entertainment; or initiating relaxation interventions when the user is handling important tasks, disrupting their thought process. Such interference not only damages the user experience but may also trigger a crisis of trust in the system, reducing the user's willingness to use it long-term. The core issue lies in the fact that existing technologies generally employ a rigid, one-way "recognition-execution" logic. They lack both a dynamic, quantitative evaluation mechanism for the credibility of emotion recognition results and a strategy system for implementing tiered responses based on credibility levels. Specifically, this manifests in two ways: the system applies a uniform intensity of response to all recognition results, leading to excessive interference when low-credibility false positives occur; or, to avoid false positives, it sets stringent recognition thresholds, causing highly credible, genuine emotional needs to be systematically ignored. This either-or approach prevents the system from achieving a balance between the timeliness of emotional care and the accuracy of control commands, ultimately limiting the practical value of smart home emotional interaction functions.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a smart home control method and system based on emotion recognition, which aims to improve the accuracy of emotion recognition, implement a hierarchical response strategy, and avoid user misoperation.

[0005] To achieve the above objectives, this application proposes a smart home control method based on emotion recognition, the method comprising: Acquire the user's current context and multimodal emotional data; the multimodal emotional data includes facial images, speech audio, and physiological signals; Based on the current context, the multimodal emotion data is subjected to context-aware fusion analysis to generate preliminary emotion recognition results and corresponding first credibility. Obtain the user's historical emotional patterns, match and verify the preliminary emotional recognition results and the first credibility with the user's historical emotional patterns, and generate the final emotional state and the corresponding second credibility. Based on the second level of credibility, a graded response strategy corresponding to the final emotional state is determined; the graded response strategy defines multiple response levels from non-interference monitoring to proactive intervention. Based on the final emotional state and the hierarchical response strategy, personalized care control instructions for smart home devices are generated and executed.

[0006] The steps for performing context-aware fusion analysis on the multimodal emotion data based on the current context to generate preliminary emotion recognition results and corresponding first confidence levels include: Based on the current context, the dynamic weights of each modality of emotional data corresponding to the current context are determined from the preset context-modality weight mapping table. Analyze the changes in key facial feature points in the facial image to generate a facial expression feature vector; Analyze the acoustic feature changes of the speech audio to generate a speech emotion feature vector; Analyze the waveform and numerical changes of the physiological signals to generate a physiological state feature vector; Based on the dynamic weights, the facial expression feature vector, voice emotion feature vector, and physiological state feature vector are weighted, fused, and classified to generate the preliminary emotion recognition result, and the first confidence level is calculated based on the correlation between each feature vector.

[0007] The steps of obtaining the user's historical emotional patterns, matching and verifying the preliminary emotional recognition result and the first credibility with the user's historical emotional patterns, and generating the final emotional state and the corresponding second credibility include: Based on the current context and the user's historical emotional patterns, a reasonableness benchmark for the user's emotional state in the current context is determined. The preliminary emotion recognition results are compared with the reasonableness benchmark to generate a situation reasonableness score; Based on the contextual plausibility score, the first credibility is corrected to generate the second credibility, and the preliminary emotion recognition result is output as the final emotion state.

[0008] The steps for determining the rationality benchmark of the user's emotional state in the current context, based on the current situation and the user's historical emotional patterns, include: From the user's historical sentiment patterns, query historical context records that are the same as or similar to the current context; Extract the user's historical emotional states and their frequency of occurrence from the historical context records; Based on the frequency of occurrence, a probability distribution representing the likelihood of different emotional states occurring in the current context is constructed as the rationality benchmark; When the current situation is the first occurrence or there is insufficient historical data, the preset general situation-emotion common sense library is invoked as a supplementary rationality benchmark.

[0009] The step of comparing the preliminary emotion recognition result with the reasonableness benchmark to generate a situation reasonableness score includes: From the rationality benchmark, obtain the historical occurrence probability corresponding to the preliminary emotion recognition result; Calculate the degree of difference between the preliminary emotion recognition result and the emotion state with the highest probability in the reasonableness benchmark; By combining the historical occurrence probability with the degree of difference, the situation rationality score is calculated and generated using a predefined scoring function.

[0010] The steps for determining the graded response strategy corresponding to the final emotional state based on the second credibility include: Obtain a preset graded response threshold, wherein the graded response threshold defines the minimum confidence level corresponding to each response level; The second confidence level is compared with the graded response threshold to determine the corresponding response level; Based on the final emotional state and the response level, the hierarchical response strategy is queried and generated from a preset strategy database. The hierarchical response strategy includes the type, intensity, and time parameters of the response operation.

[0011] The hierarchical response strategy includes at least two levels of strategy combination: A monitoring and recording-level strategy is used to record only the final emotional state without generating control instructions when the second confidence level is lower than the first threshold, while simultaneously activating a delayed monitoring mechanism. An environmental cue-level strategy is used to generate control instructions within a preset minimum influence range when the second confidence level is between the first threshold and the second threshold. A moderate care level strategy is used to generate control instructions containing a revocable flag when the second confidence level is between the second threshold and the third threshold. An active intervention-level strategy is used to generate a sequence of control instructions when the second confidence level is higher than the third threshold, and can trigger a notification mechanism based on the type of the final emotional state.

[0012] Before the step of generating and executing personalized care control instructions for smart home devices based on the final emotional state and the graded response strategy, the method further includes: Starting from the emotion recognition process, the occurrence and changes of the final emotion state are continuously monitored, and the duration of the corresponding emotion state is recorded; The duration of the emotional state is compared with a preset care initiation delay threshold to determine whether the time condition for initiating care operation has been met, and a delay condition satisfaction judgment is generated. The duration of the emotional state is compared with a preset emotional fluctuation threshold to determine whether the current emotional state is a brief emotional fluctuation, and a non-brief fluctuation judgment is generated. Based on the judgment that the delay condition is met and the judgment that it is not a transient fluctuation, the response level determined according to the second credibility is dynamically adjusted, and a graded response strategy corrected by the time dimension is generated based on the adjusted response level.

[0013] Dynamically adjusting the response level determined based on the second confidence level specifically includes: Based on the duration range of the emotional state, the corresponding response level adjustment direction and adjustment range are determined from the predefined duration-response level correction mapping relationship; Based on the direction of the response level adjustment, the response level is upgraded or downgraded to obtain a preliminary adjusted response level. Based on the adjustment range, the initially adjusted response level is quantitatively corrected to generate the final response level after time dimension correction. Based on the final response level, the preset strategy database is queried again to obtain the corresponding response strategy parameters and generate the time-dimension corrected hierarchical response strategy.

[0014] The emotion-based smart home control system includes: a memory, a processor, and an emotion-based smart home control program stored in the memory and executable on the processor. The emotion-based smart home control program is configured to implement the steps of the emotion-based smart home control method.

[0015] The smart home control method and system based on emotion recognition proposed in this application acquires multimodal emotion data and performs context-aware fusion analysis, combined with historical pattern verification, and determines a hierarchical response strategy based on credibility, thereby accurately controlling smart home devices. This improves the accuracy of emotion recognition, implements a hierarchical response strategy, avoids user misoperation, and enhances the user experience. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the emotion recognition-based smart home control method of this application; Figure 2 For this application Figure 1 A detailed flowchart of step S200; Figure 3 For this application Figure 1 Detailed flowchart of step S300; Figure 4 For this application Figure 3 A detailed flowchart of step S310; Figure 5 For this application Figure 3 A detailed flowchart of step S320; Figure 6 For this application Figure 1 Detailed flowchart of step S430; Figure 7 This is a flowchart illustrating another embodiment of the emotion recognition-based smart home control method of this application; Figure 8 For this application Figure 7 A detailed flowchart of step S640; Figure 9 This is a schematic diagram of a structure provided for an embodiment of the emotion recognition-based smart home control system of this application.

[0019] Explanation of icon numbers: 10. Memory; 20. Processor.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be understood that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] In existing technologies, smart home systems based on emotion recognition suffer from a high misjudgment rate when identifying user emotions. When the system misjudges a user's emotion, it triggers inappropriate home control commands, interfering with user activities and degrading the user experience. This is because existing solutions lack a mechanism to evaluate the reliability of emotion recognition results and fail to adopt differentiated response strategies based on reliability, leading to interference or untimely responses to needs when misjudged.

[0024] Based on this, embodiments of this application provide a smart home control method based on emotion recognition, referring to... Figure 1 The aforementioned smart home control method based on emotion recognition includes steps S100 to S500, wherein: Step S100: Obtain the user's current context and multimodal emotion data; the multimodal emotion data includes facial images, voice audio, and physiological signals; Step S200: Based on the current context, perform context-aware fusion analysis on the multimodal emotion data to generate preliminary emotion recognition results and corresponding first credibility. Step S300: Obtain the user's historical emotional pattern, match and verify the preliminary emotional recognition result, the first credibility, and the user's historical emotional pattern, and generate the final emotional state and the corresponding second credibility. Step S400: Based on the second credibility, determine the graded response strategy corresponding to the final emotional state; the graded response strategy defines multiple response levels from non-interference monitoring to active intervention; Step S500: Based on the final emotional state and the graded response strategy, generate and execute personalized care control instructions for smart home devices.

[0025] In this embodiment, the current context refers to the user's environmental state, including but not limited to information such as time, location, weather, ongoing activities, and surrounding people. This information can be acquired through various sensors in the smart home system or through user input, providing contextual information for emotion recognition. Multimodal emotion data refers to user emotion-related data collected from different perceptual dimensions. Facial images are used to capture changes in the user's facial expressions; voice audio is used to analyze the user's pitch, speech rate, volume, and other acoustic features; physiological signals are used to reflect the user's heart rate, skin conductance, brain waves, and other physiological indicators. These data collectively provide information input for emotion recognition. The preliminary emotion recognition result refers to the user's emotion category initially determined by the system after fusing and analyzing the multimodal emotion data, considering the current context.

[0026] The first credibility refers to the reliability of the initial sentiment recognition results. This credibility reflects the consistency between the data from different modalities and the confidence level of the recognition algorithm in the results during the multimodal sentiment data fusion analysis process. User historical sentiment patterns refer to the patterns of user sentiment expression in different situations that the system has recorded and learned over a long period. This pattern can include the user's frequently occurring emotional states, duration of emotions, and responses to specific stimuli in specific situations, used for personalized verification and correction of sentiment recognition results. The final sentiment state refers to the user's final sentiment category determined by the system after verification through matching the user's historical sentiment patterns. This state is a sentiment judgment that conforms to the user's actual situation after being corrected by context and historical data.

[0027] The second credibility level refers to the reliability of the final emotional state. This credibility level is a modified value obtained by matching and verifying the user's historical emotional patterns based on the first credibility level, reflecting the confidence level of the final emotional state. The tiered response strategy refers to the differentiated control schemes the system develops for smart home devices based on the type of the final emotional state and the level of the second credibility level. This strategy defines response levels ranging from non-intrusive monitoring to proactive intervention to ensure caring measures are taken at different credibility levels. Personalized care control commands refer to the operational instructions issued by the system to smart home devices based on the final emotional state and the tiered response strategy, aiming to provide users with personalized care that matches their emotional state.

[0028] The emotion recognition-based smart home control method first acquires the user's current context and multimodal emotion data. Current context information can be manually input by the user or periodically collected by environmental sensors within the smart home system. Multimodal emotion data includes facial images, voice audio, and physiological signals. Facial images can be captured by cameras installed indoors; voice audio can be recorded by microphones; and physiological signals can be collected by health monitoring devices worn by the user. This data is then transmitted to a processing unit for further analysis.

[0029] Based on the current context, the system performs context-aware fusion analysis on the multimodal emotion data to generate a preliminary emotion recognition result and a corresponding first confidence level. Upon receiving the multimodal emotion data, the system can extract emotion features from each modality, such as facial expression features from facial images, intonation features from audio speech, and heart rate change features from physiological signals. These extracted features are then input into an emotion classification model, which weights or averages the features from each modality to generate an emotion category judgment. Simultaneously, the model outputs a numerical value representing the reliability of the preliminary recognition result, i.e., the first confidence level, based on its internal classification confidence.

[0030] Furthermore, the system acquires the user's historical emotional patterns and matches and verifies the preliminary emotional recognition result and the first level of credibility with these patterns to generate a final emotional state and a corresponding second level of credibility. The user's historical emotional patterns can be a database stored in the system, recording the emotional states identified in the user at different times and in different situations in the past. After obtaining the preliminary emotional recognition result and the first level of credibility, the system can query this historical database to find historical records corresponding to the current situation. For example, if the preliminary recognition result is "irritable," the system will check whether the user frequently exhibits "irritable" emotions in similar situations. If the historical records show that the user is usually "calm" in that situation, the system will consider the current "irritable" recognition result to be potentially biased. Through this historical data comparison, the system can verify the preliminary emotional recognition result and correct the first level of credibility, thereby obtaining a final emotional state and a corresponding second level of credibility that conforms to the actual situation.

[0031] Based on the second level of credibility, a tiered response strategy corresponding to the final emotional state is determined. This tiered response strategy defines response levels ranging from non-intrusive monitoring to proactive intervention. The system can preset thresholds, such as low, medium, and high thresholds. When the second level of credibility is below the low threshold, the system may select the "no response" level; when the second level of credibility is between the low and medium thresholds, the system may select the "cue" level; when the second level of credibility is above the medium threshold, the system may select the "regulation" level. For example, if the final emotional state is "relaxed" and the second level of credibility is high, the system can determine a "regulation" level response strategy, which might include adjusting the lights, playing music, etc.

[0032] Finally, based on the final emotional state and the tiered response strategy, personalized care control instructions for smart home devices are generated and executed. Once the final emotional state and the corresponding tiered response strategy are determined, the system directly generates the corresponding control instructions based on the strategy content. For example, if the strategy is "adjust the lights to a soft mode and play music," the system will send instructions to the smart lighting and audio devices to perform the corresponding operations. The execution of these instructions aims to provide users with immediate home environment care that matches their emotional state.

[0033] By employing context-aware multimodal emotion fusion analysis and combining it with users' historical emotion patterns to verify the credibility of the recognition results, the risk of misjudgment in emotion recognition is reduced. By determining a tiered response strategy based on the corrected credibility, the system can avoid intervention under low credibility conditions, reducing interference from inappropriate control on user activities. Simultaneously, under high credibility conditions, the system can provide personalized care control in real time, thereby enhancing the user experience of the smart home system.

[0034] In one feasible implementation, refer to Figure 2 Step S200 includes steps S210 to S250, wherein: Step S210: Based on the current context, determine the dynamic weights of each modality of emotional data corresponding to the current context from the preset context-modality weight mapping table; Step S220: Analyze the changes in key facial feature points in the facial image to generate a facial expression feature vector; Step S230: Analyze the acoustic feature changes of the speech audio and generate a speech emotion feature vector; Step S240: Analyze the waveform and numerical changes of the physiological signal to generate a physiological state feature vector; Step S250: Based on the dynamic weights, the facial expression feature vector, voice emotion feature vector, and physiological state feature vector are weighted, fused, and classified to generate the preliminary emotion recognition result, and the first confidence level is calculated based on the correlation between each feature vector.

[0035] In this embodiment, based on the current context, the dynamic weights of each modality of emotional data corresponding to the current context are determined from a preset context-modality weight mapping table. This context-modality weight mapping table pre-stores the relative importance or reliability of facial images, audio recordings, and physiological signals for emotion recognition in different contexts. For example, in dimly lit environments, the reliability of facial images may be lower, and their weight will be correspondingly reduced; while when users are engaging in voice communication, the weight of audio recordings may be increased. These dynamic weights can be trained and optimized based on expert experience, historical data analysis, or machine learning models to ensure that, in specific contexts, the system prioritizes the most informative and reliable emotional modalities.

[0036] The changes in key facial feature points in the facial image are analyzed to generate a facial expression feature vector. This process typically involves preprocessing the facial image, such as face detection and alignment, and then identifying and tracking key feature points in areas such as the eyes, eyebrows, and mouth. By quantifying the changes in the position, shape, motion trajectory, and interrelationships of these feature points, for example by calculating the intensity of Action Units (AUs), a numerical vector that can characterize the user's facial expression can be constructed, i.e., the facial expression feature vector.

[0037] The acoustic feature changes of the spoken audio are analyzed to generate a speech emotion feature vector. This includes extracting a series of acoustic parameters from the user's spoken audio, such as pitch, intensity, speech rate, formant frequencies, Mel-frequency cepstral coefficients (MFCCs), jitter, and shimmer. These acoustic features reflect the prosody, timbre, and spectral characteristics of speech, and are then integrated to form the speech emotion feature vector to capture the emotional information contained in the speech.

[0038] The waveform and numerical changes of the physiological signals are analyzed to generate a physiological state feature vector. Physiological signals may include heart rate, heart rate variability, skin conductance, blood volume pulse, respiratory rate, etc. Through in-depth analysis of the waveform characteristics (such as periodicity and amplitude) and numerical changes (such as mean and standard deviation) of these signals, features reflecting the user's physiological arousal level, tension level, or emotional state can be extracted and combined into a physiological state feature vector.

[0039] Based on the dynamic weights, the facial expression feature vector, voice emotion feature vector, and physiological state feature vector are weighted, fused, and classified to generate the preliminary emotion recognition result. Specifically, the three modal feature vectors extracted earlier are multiplied by their corresponding dynamic weights, and then the weighted feature vectors are fused (e.g., through concatenation or a more complex fusion network). The fused feature representation is input into a pre-trained emotion classification model, which maps the fused features to predefined emotion categories (such as happiness, sadness, anger, calmness, etc.), thereby outputting the preliminary emotion recognition result.

[0040] Based on this, the first confidence level is calculated based on the correlation between the feature vectors. This confidence level aims to assess the reliability of the preliminary sentiment recognition results. For example, it can be obtained by calculating the degree of consistency between the sentiment categories or sentiment dimensions indicated by the feature vectors of different modalities. If all modalities strongly point to the same sentiment, the correlation is high, and the first confidence level is also high; conversely, if there is conflict or uncertainty between the modalities, the correlation is low, and the first confidence level is also low. This can be achieved through statistical methods such as entropy, analysis of variance, or voting consistency of prediction results among modalities.

[0041] By introducing a context-aware dynamic weighting mechanism, the system can intelligently adjust the contribution of each modality's data according to the current context, thereby avoiding misjudgments of emotion due to insufficient or interference from a single modality and improving the accuracy and robustness of the initial emotion recognition results. Simultaneously, by calculating the correlation between the feature vectors of each modality, the confidence level of the recognition results can be quantitatively assessed, i.e., a first level of confidence is generated. This provides reliable input for subsequent verification of user historical emotion patterns, ensuring that the smart home control system can generate personalized care control commands based on more accurate and reliable emotion judgments, thus achieving more refined and humanized smart home services.

[0042] In one feasible implementation, refer to Figure 3 Step S300 includes steps S310 to S330, wherein: Step S310: Based on the current context and the user's historical emotional patterns, determine the rationality benchmark of the user's emotional state in the current context; Step S320: Compare the preliminary emotion recognition result with the rationality benchmark to generate a situation rationality score; Step S330: Based on the situational rationality score, the first credibility is corrected to generate the second credibility, and the preliminary emotion recognition result is output as the final emotion state.

[0043] In this embodiment, acquiring user historical emotional patterns refers to the long-term accumulation of data on the emotional states and patterns exhibited by users in different contexts. These patterns can include information such as the type, intensity, and duration of emotions frequently expressed by the user at specific times, places, activities, or interactions with specific people. This data can be collected and stored through continuous monitoring of the user's multimodal emotional data, user feedback, or historical records of interactions with smart home devices. For example, the system can record that a user typically exhibits "relaxed" or "pleasant" emotions in a "watching a movie" context, while in a "working" context, they may exhibit "focused" or "slightly stressed" emotions. These patterns can be stored in a database and represented in the form of structured data (such as time series, probability distributions, rule sets, etc.).

[0044] Based on the current context and the user's historical emotional patterns, a reasonable benchmark for the user's emotional state in the current context is determined. The aim is to incorporate personalized and contextual considerations to assess whether the initial emotional recognition results conform to reference standards for the user's typical emotional expression in a specific context. Besides retrieving historical emotional patterns that are identical or similar to the current context and extracting emotional states and their frequencies to construct a probability distribution, other methods can be used. For example, machine learning models (such as Hidden Markov Models, Recurrent Neural Networks, etc.) can be used to train the user's historical emotional patterns, learning the dynamic evolution of the user's emotional state in different contexts, thereby predicting the reasonable range or most likely emotional state of the user in the current context. Furthermore, an expert knowledge base or a pre-defined user profile can be combined to set an initial reasonable range or weight for the emotional state in a specific context.

[0045] The preliminary emotion recognition result is compared with the rationality benchmark to generate a contextual rationality score. This score is a quantitative indicator that measures the degree of match between the preliminary emotion recognition result and the user's historical emotion patterns in the current context. The score reflects the "rationality" or "probability" of the currently identified emotion state in the current context. Besides combining historical occurrence probabilities and differences through a predefined scoring function, various comparison and scoring methods can be used. For example, the KL divergence (Kullback-Leibler divergence) or JS divergence (Jensen-Shannon divergence) of the preliminary emotion recognition result and the distribution of each emotion state in the rationality benchmark can be calculated to quantify the difference between the two. Alternatively, based on fuzzy logic, the preliminary emotion recognition result can be mapped to a fuzzy set in the rationality benchmark, and its membership degree can be calculated as the rationality score. Furthermore, a series of rules can be set to provide a binary or multi-valued score based on whether the preliminary emotion recognition result falls within the "normal" emotion range defined by the rationality benchmark.

[0046] Based on the contextual plausibility score, the first credibility is revised to generate a second credibility. The purpose is to incorporate the contextual plausibility dimension into the final sentiment recognition credibility, making it more personalized and context-adaptive. The revision process can employ various mathematical models or algorithms. For example, the contextual plausibility score can be used as a weighting factor, weighted and averaged with the first credibility, or combined using multiplication, addition, or other methods. Specifically, a higher contextual plausibility score indicates a high degree of consistency between the initial sentiment recognition result and the user's historical patterns, allowing for an appropriate increase in the first credibility; conversely, a lower score indicates a potential significant deviation between the initial sentiment recognition result and the user's historical patterns, allowing for an appropriate decrease in the first credibility. This revision mechanism ensures that the final credibility considers not only the fusion analysis results of multimodal data but also the user's personalized contextual sentiment patterns.

[0047] The initial emotion recognition result is then output as the final emotion state. This means that after matching and verifying against historical emotion patterns, the system typically does not change the initially identified emotion type (e.g., if the initial identification is "happy," the final emotion state remains "happy"). Instead, it primarily adjusts the credibility score to reflect the rationality of the emotion in the current context. This approach avoids misjudgments of the emotion type that may result from over-adjustment, while simultaneously providing a more refined and reliable decision-making basis for subsequent smart home control through credibility adjustments. In some extreme cases, if the contextual rationality score is extremely low, the system may choose to mark the emotion state as "uncertain" or trigger a manual confirmation mechanism, but normally, the initially identified emotion type is retained.

[0048] By leveraging users' historical emotional patterns, the system can determine the rationality benchmark of a user's emotional state based on the current context and the user's personalized emotional habits. Subsequently, the preliminary emotional recognition result is compared with this rationality benchmark to generate a contextual rationality score, which quantifies the degree of matching between the current recognition result and the user's historical patterns. Based on this, the system corrects the initial credibility of the recognition according to the contextual rationality score, thereby generating a more personalized and context-adaptive second credibility. This correction mechanism effectively solves the problem that emotional recognition results may not match the user's true emotional state if relying solely on multimodal data fusion analysis, improving the accuracy and reliability of emotional recognition. By using the preliminary emotional recognition result as the final emotional state output, supplemented by the corrected second credibility, the system ensures that subsequent smart home control commands can more accurately respond to the user's personalized needs, avoiding interference or discomfort caused by misjudgment, thus greatly improving the user experience and care effect of the smart home system.

[0049] In one feasible implementation, refer to Figure 4 Step S310 includes steps S311 to S314, wherein: Step S311: Query historical situation records that are the same as or similar to the current situation from the user's historical sentiment patterns; Step S312: Extract the user's historical emotional states and their frequency of occurrence from the historical context records. Step S313: Based on the frequency of occurrence, construct a probability distribution representing the likelihood of different emotional states occurring in the current situation, as the rationality benchmark; Step S314: When the current situation is the first occurrence or the historical record is insufficient, the preset general situation-emotion common sense library is invoked as a supplementary rationality benchmark.

[0050] In this embodiment, when determining the rationality benchmark, the system first queries the user's historical emotional pattern database for historical situation records that have the same or similar characteristics as the current situation. For example, if the current situation is "evening, living room, watching a movie," the system will retrieve the user's past records in situations such as "evening, living room, entertainment" or "evening, living room, leisure." This query can be implemented through a situational feature matching algorithm, which comprehensively considers multiple situational dimensions such as time, location, activity type, and companions, calculates the similarity between historical records and the current situation, and filters out records with similarity reaching a preset threshold.

[0051] The system will analyze and extract the various emotional states that the user exhibited in those historical contexts and their corresponding frequencies from these retrieved records. For example, in similar contexts such as "evening, living room, watching a movie," the user might exhibit "relaxation" 80% of the time, "pleasure" 15% of the time, and "calmness" 5% of the time. These statistics quantify the likelihood of the user expressing different emotions in a specific context.

[0052] Based on the frequency of occurrence obtained from the above statistics, the system constructs a probability distribution representing the likelihood of different emotional states occurring in the current context. This probability distribution serves as a benchmark for evaluating the reasonableness of the initial emotion recognition results. For example, statistical models such as frequency distributions, Bayesian networks, or Markov chains can be used to transform the frequency of emotional states into probability values, forming a dynamic and personalized emotion probability model.

[0053] To address the cold start problem or the issue of sparse historical data—specifically, when the current situation is a first occurrence or user history is insufficient to construct a reliable probability distribution—the system invokes a pre-built general context-emotion common sense base. This base is a pre-constructed context-emotion mapping rule library based on extensive user data or expert knowledge. It provides the likely emotional states that ordinary users might exhibit in various typical situations, along with their approximate probabilities. For example, in the context of "early morning, kitchen, preparing breakfast," the common sense base might pre-determine a higher probability of "positive, calm." In this case, the general context-emotion common sense base serves as a supplementary rationality benchmark, fused with a small amount of user historical data (if available) to ensure the comprehensiveness and robustness of the rationality benchmark.

[0054] By querying historical records related to or similar to the current context and extracting the user's emotional state and frequency in that context, a personalized probability distribution can be formed, thus more accurately reflecting the user's emotional tendency in a specific situation. Furthermore, to address the cold start problem—where the current context is the first occurrence or historical records are insufficient—a general context-emotion common sense base is introduced as a supplement, effectively compensating for the deficiencies caused by data sparsity and ensuring the comprehensiveness and robustness of the reasonable benchmark. This makes subsequent verification of the preliminary emotion recognition results more reliable, improving the accuracy and credibility of the final emotion state recognition, avoiding misjudgments caused by benchmark bias, and thus enabling the generation of personalized care and control instructions that better meet the user's actual needs and context.

[0055] In one feasible implementation, refer to Figure 5 Step S320 includes steps S321 to S323, wherein: Step S321: Obtain the historical occurrence probability corresponding to the preliminary emotion recognition result from the rationality benchmark; Step S322: Calculate the degree of difference between the preliminary emotion recognition result and the emotion state with the highest probability in the rationality benchmark; Step S323: Combining the historical occurrence probability and the degree of difference, the situation rationality score is calculated and generated using a predefined scoring function.

[0056] In this embodiment, obtaining the historical occurrence probability corresponding to the preliminary emotion recognition result from the rationality benchmark refers to the system querying a pre-constructed rationality benchmark based on the current context and the preliminary emotion recognition result. This rationality benchmark is formed based on the user's historical emotion patterns, recording the frequency or probability of various emotion states occurring in specific contexts. For example, if the preliminary emotion recognition result is "happy" and the current context is "watching a comedy," the system will retrieve the historical probability of the user experiencing the emotion of "happy" in the context of "watching a comedy" from the rationality benchmark. This step aims to quantify the "commonness" or "expectancy" of the current preliminary emotion state in history, providing basic data for subsequent rationality assessment.

[0057] Building upon this, this application further proposes calculating the degree of discrepancy between the preliminary emotion recognition result and the most probable emotion state in the rationality benchmark. This step aims to measure the degree of deviation between the current preliminary emotion recognition result and the emotion state that best fits the historical pattern in this context. For example, if in a given context, the rationality benchmark shows "calm" as the most probable emotion state, while the preliminary emotion recognition result is "anger," then the degree of discrepancy between the two would be relatively high. The degree of discrepancy can be calculated in various ways. For instance, if the emotion state is discretely classified, it can be based on a lookup table using a predefined emotion distance matrix; if the emotion state is mapped to a continuous emotion dimension space (such as a valence-arousal model), then metrics such as Euclidean distance or cosine similarity can be used. This degree of discrepancy provides a quantitative indicator for assessing the "abnormality" or "unexpectedness" of the current emotion state.

[0058] Furthermore, this application generates a contextual rationality score by combining the aforementioned historical occurrence probability and dissimilarity using a predefined scoring function. This scoring function is a mathematical model or algorithm that takes historical occurrence probability and dissimilarity as input and outputs a comprehensive numerical value. For example, the scoring function can be designed as: Contextual Rationality Score = w1 × Historical Occurrence Probability - w2 × Dissimilarity, where w1 and w2 are adjustable weighting coefficients used to balance the influence of historical predictability and current anomaly on the score. In this way, the contextual rationality score can comprehensively reflect the rationality of the initial sentiment recognition result in the current context; a higher score indicates greater rationality, and a lower score indicates less rationality. The generation of this score is a key basis for subsequent correction of the initial credibility.

[0059] By acquiring the historical occurrence probability corresponding to the initial emotion recognition result, the system can determine the frequency of the emotion state in history. Simultaneously, by calculating the difference between the initial emotion recognition result and the most likely historical emotion state, the system can quantify the "abnormality" of the current emotion. Combining these two aspects of information, a situational reasonableness score is generated through a predefined scoring function. This transforms the reasonableness assessment of the initial emotion recognition result from a simple binary judgment into a continuous and quantifiable process. This provides a solid foundation for subsequent precise correction of the first credibility score, thereby ensuring the accuracy of the final emotion state and the reliability of the second credibility score. It avoids misjudgments caused by brief or abnormal emotional fluctuations, improving the accuracy of smart home control and the user experience.

[0060] In one feasible implementation, refer to Figure 6 Step S400 includes steps S410 to S430, wherein: Step S410: Obtain a preset graded response threshold, wherein the graded response threshold defines the minimum confidence level corresponding to each response level; Step S420: Compare the second confidence level with the graded response threshold to determine the corresponding response level; Step S430: Based on the final emotional state and the response level, query and generate the graded response strategy from the preset strategy database. The graded response strategy includes the type, intensity, and time parameters of the response operation.

[0061] In this embodiment, the system acquires preset graded response thresholds. These thresholds are a set of pre-defined values ​​used to define the boundaries between different response levels. Each response level corresponds to a minimum confidence value; when the second confidence level of emotion recognition reaches or exceeds this value, the system can consider adopting the corresponding response level. These thresholds are typically trained and set based on a large amount of user behavior data, expert experience, and the system's requirements for the accuracy of emotion recognition. For example, multiple thresholds can be set, such as 0.6, 0.75, and 0.9, corresponding to intervals such as "low confidence," "medium confidence," and "high confidence," respectively, and thus mapped to different response levels.

[0062] The second confidence level is compared with the tiered response threshold to determine the corresponding response level. This step aims to compare the second confidence level currently identified by the system with the preset tiered response threshold, thereby clarifying the confidence interval of the current emotion recognition result and determining the appropriate response level accordingly. For example, if the second confidence level is 0.8, and the thresholds are set to 0.6, 0.75, and 0.9, then 0.8 falls between 0.75 and 0.9, and the system can determine that the current response level should be "medium confidence." This comparison can be a simple numerical comparison or a logical operation based on interval judgment.

[0063] Based on this, and using the final emotional state and the response level, the system queries and generates the tiered response strategy from a pre-defined strategy database. The strategy database is a knowledge base storing various predefined smart home control strategies. Each strategy is associated with a specific final emotional state and response level, and specifies in detail the type of response operation (e.g., light adjustment, music playback, temperature control), intensity (e.g., light brightness, music volume, temperature fluctuation), and time parameters (e.g., duration, delay start time). Once the user's final emotional state and the appropriate response level are determined, the system retrieves the most matching strategy from the database and outputs it as the tiered response strategy. For example, if the final emotional state is "anxiety" and the response level is "moderate care," the system might retrieve a strategy: play soothing music (type), medium volume (intensity), and last for 30 minutes (time parameter).

[0064] By using preset tiered response thresholds, the system can accurately map the abstract second level of trust to specific response levels, avoiding arbitrariness in response strategies. Based on this, and combining the final emotional state and the determined response level, the system queries the strategy database and generates detailed tiered response strategies containing parameters such as operation type, intensity, and time. This ensures that smart home devices can provide appropriate, personalized, and controllable care services based on the trustworthiness of the user's emotional state. This not only improves the accuracy and rationality of the smart home system's response but also allows users to receive a care experience that better matches their current emotional needs and is risk-controlled, effectively solving the problem of inconsistent or inappropriate care measures under different levels of trust.

[0065] In one feasible implementation, the tiered response strategy includes a combination of at least two levels of strategies: a monitoring and recording level strategy, used to record only the final emotional state without generating control instructions when the second confidence level is lower than the first threshold, while simultaneously activating a delayed monitoring mechanism; an environmental cue level strategy, used to generate control instructions within a preset minimum impact range when the second confidence level is between the first and second thresholds; a moderate care level strategy, used to generate control instructions containing a revocable flag when the second confidence level is between the second and third thresholds; and a proactive intervention level strategy, used to generate a sequence of control instructions when the second confidence level is higher than the third threshold, and may trigger a notification mechanism based on the type of the final emotional state.

[0066] In this embodiment, when the second confidence level is lower than the first threshold, the system will adopt a monitoring and recording-level strategy. This strategy aims to address situations where the system's confidence in recognizing the user's emotional state is low, avoiding unnecessary or potentially erroneous interventions when the system lacks confidence, thereby protecting user privacy and reducing the negative impact of misjudgments. Under this strategy, the system only records the final recognized emotional state without generating control commands, while simultaneously initiating a delayed monitoring mechanism. For example, the system internally maintains a log module to record the final recognized emotional state, the corresponding second confidence level, and the time of occurrence. The delayed monitoring mechanism can be implemented by setting a timer or counter to continuously collect user data within a certain time window, aiming to obtain more accurate emotional recognition results in subsequent monitoring.

[0067] When the second level of confidence falls between the first and second thresholds, the system will employ an environmental cue-based strategy. This strategy is suitable when the system has a certain level of confidence in recognizing the user's emotional state, but has not yet reached a high level of confidence. It indirectly influences the user's emotions by generating control commands within a preset minimum impact range, subtly and non-intrusively adjusting the user's environment. This strategy aims to provide gentle care or guidance while minimizing interference with the user's normal activities. For example, control commands within the minimum impact range could include adjusting the color temperature or brightness of indoor lighting, playing soft background music, or slightly adjusting the air conditioning temperature. These commands are typically reversible, have a limited range of influence, and do not cause a direct perceptual impact on the user.

[0068] When the second level of confidence falls between the second and third thresholds, the system will adopt a moderate level of care strategy. This strategy is suitable for situations where the system has a high degree of confidence in recognizing the user's emotional state, but still requires careful handling. It generates control commands containing a revocable flag, providing more direct care while granting the user the right to revoke or modify system operations. This ensures the effectiveness of the care while respecting the user's autonomy and avoiding forced intervention. For example, the revocable flag can be implemented by embedding specific parameters in the control command or by providing "undo" or "restore default" buttons on the smart home control interface.

[0069] When the second confidence level exceeds the third threshold, the system will employ a proactive intervention strategy. This strategy is suitable for situations where the system has extremely high confidence in recognizing the user's emotional state and determines that the user may require immediate or significant care. It provides comprehensive and timely intervention and support by generating a sequence of control instructions and triggering a notification mechanism based on the type of the final emotional state. For example, the sequence of control instructions could be a series of coordinated actions; when the system detects that the user is in extreme sadness, it could sequentially execute: playing comforting music preferred by the user, adjusting the indoor lighting to a warm tone, activating an aromatherapy diffuser, and sending a notification to a pre-set emergency contact. The notification mechanism can select different notification recipients and content based on the emotional type.

[0070] By combining the aforementioned tiered response strategies, this application enables refined and differentiated management of care control commands for smart home devices based on different ranges of second confidence. When the second confidence is low, a monitoring and recording-level strategy is adopted to avoid unnecessary intervention when the system lacks confidence, effectively reducing the risk of misjudgment, protecting user privacy, and providing a time window for more accurate subsequent judgments. As the second confidence increases, environmental cue-level and moderate care-level strategies are gradually introduced, providing care in a progressive, non-intrusive, or revocable manner. This not only positively guides the user's emotions but also fully respects the user's autonomy, avoiding discomfort caused by excessive intervention. When the second confidence is extremely high, the proactive intervention-level strategy can provide a timely and comprehensive sequence of care commands, and even trigger a notification mechanism, ensuring effective support when the user urgently needs help. This multi-level, multi-dimensional strategy combination enables the smart home system to respond to user needs more intelligently and humanely based on emotion recognition, improving user experience and the accuracy and effectiveness of care, while balancing the timeliness of system intervention with user acceptance.

[0071] In one feasible implementation, refer to Figure 7 Before step S500, the method further includes steps S610 to S640, wherein: Step S610: Starting from the emotion recognition process, continuously monitor the occurrence and changes of the final emotion state, and record the duration of the corresponding emotion state; Step S620: Compare the duration of the emotional state with a preset care initiation delay threshold to determine whether the time condition for initiating care operation has been met, and generate a delay condition satisfaction judgment. Step S630: Compare the duration of the emotional state with a preset emotional fluctuation threshold to determine whether the current emotional state is a brief emotional fluctuation and generate a non-brief fluctuation judgment. Step S640: Based on the judgment that the delay condition is met and the judgment that it is not a transient fluctuation, the response level determined according to the second credibility is dynamically adjusted, and a graded response strategy corrected by the time dimension is generated based on the adjusted response level.

[0072] In this embodiment, the system continuously monitors the appearance and changes of the final emotional state from the start of the emotion recognition process, and records the duration of the corresponding emotional state. Continuous monitoring means that after recognizing the user's final emotional state, the system does not immediately stop paying attention to the emotional state, but continues to track whether the emotional state persists, whether it changes, and the trend of change. For example, the system can repeat the emotion recognition process at a preset sampling frequency (such as once per second or once every few seconds), or trigger re-recognition when new emotional input is detected (such as the user speaking again, or a significant change in facial expression). Recording the duration of the emotional state means that the system quantifies and records the time from the first recognition of a specific final emotional state to its disappearance or replacement by another emotional state. This can be achieved by maintaining a timer or timestamp database. For example, when the emotion "sadness" is recognized, the timer is started; when "calm" or "anger" is recognized, the timer for "sadness" is stopped and its duration is recorded. This aims to provide a temporal basis for subsequent judgments on the stability and authenticity of emotions.

[0073] Based on this, the system compares the duration of the emotional state with a preset care initiation delay threshold to determine whether the time condition for initiating care operations has been met, and generates a delay condition satisfaction judgment. The care initiation delay threshold is a pre-set time length used to prevent the system from immediately responding to brief emotional fluctuations in the user. For example, this threshold can be set to 30 seconds, 1 minute, or longer, and the specific value can be configured according to user preferences, situation type, or experience data. The system compares the recorded duration of the emotional state with this threshold. If the duration does not reach the threshold, it is considered that the time condition for initiating care operations has not been met, and a judgment result of "delay condition not satisfied" is generated; conversely, if the duration reaches or exceeds the threshold, a judgment result of "delay condition satisfied" is generated. This judgment aims to ensure that the system only considers initiating care after the user's emotional state has lasted for a certain period, thereby avoiding overreaction to momentary emotions.

[0074] The system also compares the duration of the emotional state with a preset emotional fluctuation threshold to determine whether the current emotional state is a brief emotional fluctuation and generates a non-brief fluctuation judgment. The emotional fluctuation threshold is another preset time parameter used to distinguish whether a user's emotion is a brief, non-continuous fluctuation or a relatively stable, continuous emotional state. For example, if the duration of the emotional state is less than the emotional fluctuation threshold (e.g., 5 seconds, 10 seconds), it may be judged as a brief emotional fluctuation; if the duration is greater than or equal to the threshold, the emotional state is considered not a brief fluctuation. The system compares the duration of the emotional state with the emotional fluctuation threshold. If the duration is less than the threshold, a "brief fluctuation" judgment is generated; conversely, if the duration is greater than or equal to the threshold, a "non-brief fluctuation" judgment is generated. This judgment helps the system filter out transient emotional changes that are not worth immediate response to, focusing on more stable and authentic user emotions.

[0075] Finally, based on the judgment that the delay condition is met and the judgment that the emotion is not a transient fluctuation, the system dynamically adjusts the response level determined according to the second credibility, and generates a graded response strategy corrected for the time dimension based on the adjusted response level. This step is the core of integrating the aforementioned time dimension judgments. The system comprehensively considers the results of the "delay condition met judgment" and the "non-transient fluctuation judgment". For example, the system will only consider the current emotional state to be stable and worthy of response if both the "delay condition met judgment" are true (i.e., the duration of the emotion has reached the minimum requirement for initiating care) and the "non-transient fluctuation judgment" are true (i.e., the emotion is not a transient fluctuation). On this basis, the system will dynamically adjust the response level originally determined according to the second credibility. This adjustment can be an upgrade (e.g., from "monitoring and recording level" to "environmental cues level" or "appropriate care level"), a downgrade (e.g., if judged to be a transient fluctuation, the original "proactive intervention level" is downgraded to "monitoring and recording level"), or even remain unchanged. The adjusted response level will serve as the new basis for querying and generating the final, time-adjusted tiered response strategy from the preset strategy database.

[0076] Through the aforementioned technical solution, this application introduces a time dimension consideration of the user's final emotional state before generating and executing personalized care control commands for smart home devices. By continuously monitoring the occurrence and changes in emotional states and recording their duration, the system can effectively distinguish between short-lived emotional fluctuations and persistent emotional states. By comparing the duration of the emotional state with preset care initiation delay thresholds and emotional fluctuation thresholds, a judgment of delay condition fulfillment and non-short-lived fluctuation can be generated. Based on these two judgments, the system can dynamically adjust the response level previously determined solely based on second credibility. For example, for emotions with short durations and judged as short-lived fluctuations, even if their second credibility is high, the system may downgrade their response level to avoid unnecessary immediate intervention; while for emotions with long durations and stability, the system may maintain or even upgrade the response level to ensure timely and effective care. This time-dimensional correction mechanism improves the robustness and intelligence of the smart home system's response, avoids excessive or inappropriate intervention caused by momentary emotional fluctuations, and makes the generation of personalized care control commands more accurate and humanized, thereby effectively improving the user experience and reducing user resistance to smart home system intervention.

[0077] In one feasible implementation, refer to Figure 8 Step S640 includes steps S641 to S644, wherein: Step S641: Based on the duration interval of the emotional state, determine the corresponding response level adjustment direction and adjustment range from the predefined duration-response level correction mapping relationship; Step S642: Based on the response level adjustment direction, upgrade or downgrade the response level to obtain a preliminarily adjusted response level; Step S643: Based on the adjustment range, the initially adjusted response level is quantitatively corrected to generate the final response level after time dimension correction; Step S644: Based on the final response level, re-query the preset strategy database to obtain the corresponding response strategy parameters and generate the time-dimension corrected hierarchical response strategy.

[0078] In this embodiment, based on the duration range of the emotional state, the corresponding response level adjustment direction and magnitude are determined from a predefined duration-response level correction mapping relationship. This aims to more precisely determine how to adjust the smart home's response level according to the duration of the emotional state. This step recognizes that emotional states of different durations may require different levels of response. Specifically, the duration of the emotional state can be divided into multiple preset ranges. For example, 0-5 minutes can be defined as a "short" range, 5-15 minutes as a "medium" range, and more than 15 minutes as a "continuous" range. These duration ranges can be defined and optimized based on a large amount of user behavior data, psychological research findings, or expert experience. The predefined duration-response level correction mapping relationship is a lookup table or rule set stored in the system that associates different duration ranges with specific response level adjustment directions (such as "upgrade," "downgrade," "maintain") and adjustment magnitudes (such as "small," "medium," "large," or specific level steps). For example, when the duration of an emotional state is in the "short" range, the mapping relationship might indicate an adjustment direction of "downgrading" with a "small" adjustment magnitude; while when the duration is in the "long" range, it might indicate an adjustment direction of "upgrading" with a "medium" or "large" adjustment magnitude. By querying this mapping relationship, the system can obtain the adjustment strategy corresponding to the duration of the current emotional state.

[0079] Based on the adjustment direction of the response level, the response level is upgraded or downgraded to obtain a preliminary adjusted response level. This step involves a preliminary, qualitative adjustment to the current response level based on the adjustment direction determined in the previous step. For example, if the determined adjustment direction is "upgrade," the system will raise the current response level by one or more preset levels, such as from "monitoring and recording level" to "environmental cues level." Conversely, if the adjustment direction is "downgrade," the current response level will be lowered by one or more levels. This upgrade or downgrade is based on a preset response level sequence, such as monitoring and recording level, environmental cues level, appropriate care level, and proactive intervention level.

[0080] Based on the adjustment magnitude, the initially adjusted response level is quantified and corrected to generate the final response level after time dimension correction. This step further refines the quantification based on the adjustment magnitude to obtain a final, more accurate response level. The adjustment magnitude can be a specific numerical value, such as +0.5 or -1, or a more refined description, such as "small magnitude" corresponding to 0.5 level steps, "medium magnitude" to 1 level step, and "large magnitude" to 1.5 level steps. This quantization correction introduces continuity or finer-grained division between discrete response levels, allowing for a smoother transition between response levels. For example, if the response level is 1 to 4 and the adjustment magnitude is 0.5, intermediate levels such as 1.5 and 2.5 can be obtained, which can correspond to more refined policy parameters.

[0081] Based on the final response level, the system re-queries the preset strategy database to obtain the corresponding response strategy parameters and generates a time-adjusted hierarchical response strategy. This step involves retrieving specific control command parameters from the strategy database based on the final response level after fine-tuning the time dimension, thus generating the final, personalized hierarchical response strategy. The strategy database stores the specific response operation types, intensities, and time parameters corresponding to different response levels (including intermediate levels after quantification correction). The system uses the final response level as the query key to retrieve matching strategy parameters from the strategy database, such as the type of response operation (e.g., playing soft music, adjusting light color, turning on aromatherapy, sending care messages), intensity (e.g., music volume, light brightness / color temperature, aromatherapy concentration), and time parameters (e.g., music playback duration, light fading speed, aromatherapy duration). By obtaining these parameters, the system can generate a complete and refined hierarchical response strategy to guide smart home devices in executing corresponding care control commands.

[0082] By determining the direction and magnitude of response level adjustments based on the duration of the emotional state within a predefined duration-response level correction mapping relationship, and then quantitatively correcting the initially adjusted response level, this application enables a more refined and adaptive response to the user's emotional state. This time-based quantitative correction allows the smart home system to more accurately capture the dynamics of user emotional changes, avoiding excessive intervention in short-term emotional fluctuations and ensuring timely and appropriate care for persistent emotional states. For example, for prolonged negative emotions, the system can gradually and quantitatively upgrade the response level based on its duration, from subtle environmental cues to more proactive care, thereby providing more personalized services that meet the user's actual needs and emotional changes, improving the user experience and care effectiveness of the smart home system.

[0083] In the embodiments of this application, the smart home control method based on emotion recognition acquires multimodal emotion data and performs context-aware fusion analysis, combines historical pattern verification, and determines a graded response strategy based on credibility, thereby accurately controlling smart home devices. This improves the accuracy of emotion recognition, implements a graded response strategy, avoids user misoperation, and enhances the user experience.

[0084] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the smart home control method based on emotion recognition in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0085] This application also provides a smart home control system based on emotion recognition, see reference. Figure 9 The emotion-based smart home control system includes: a memory 10, a processor 20, and an emotion-based smart home control program stored on the memory 10 and executable on the processor 20. The emotion-based smart home control program is configured to implement the steps of the emotion-based smart home control method.

[0086] The emotion-recognition-based smart home control system provided in this application, employing the emotion-recognition-based smart home control method described in the above embodiments, can improve the accuracy of emotion recognition, implement a hierarchical response strategy, and avoid user misoperation. Compared with the prior art, the beneficial effects of the emotion-recognition-based smart home control system provided in this application are the same as those of the emotion-recognition-based smart home control method provided in the above embodiments, and other technical features in the emotion-recognition-based smart home control system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0087] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. All equivalent structural transformations made under the technical concept of this application using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of this application.

Claims

1. A smart home control method based on emotion recognition, characterized in that, The method includes: Acquire the user's current context and multimodal emotional data; the multimodal emotional data includes facial images, speech audio, and physiological signals; Based on the current context, the multimodal emotion data is subjected to context-aware fusion analysis to generate preliminary emotion recognition results and corresponding first credibility. Obtain the user's historical emotional patterns, match and verify the preliminary emotional recognition results and the first credibility with the user's historical emotional patterns, and generate the final emotional state and the corresponding second credibility. Based on the second level of credibility, a graded response strategy corresponding to the final emotional state is determined; the graded response strategy defines multiple response levels from non-interference monitoring to proactive intervention. Based on the final emotional state and the hierarchical response strategy, personalized care control instructions for smart home devices are generated and executed.

2. The smart home control method based on emotion recognition as described in claim 1, characterized in that, The steps for performing context-aware fusion analysis on the multimodal emotion data based on the current context to generate preliminary emotion recognition results and corresponding first confidence levels include: Based on the current context, the dynamic weights of each modality of emotional data corresponding to the current context are determined from the preset context-modality weight mapping table. Analyze the changes in key facial feature points in the facial image to generate a facial expression feature vector; Analyze the acoustic feature changes of the speech audio to generate a speech emotion feature vector; Analyze the waveform and numerical changes of the physiological signals to generate a physiological state feature vector; Based on the dynamic weights, the facial expression feature vector, voice emotion feature vector, and physiological state feature vector are weighted, fused, and classified to generate the preliminary emotion recognition result, and the first confidence level is calculated based on the correlation between each feature vector.

3. The smart home control method based on emotion recognition as described in claim 1, characterized in that, The steps of obtaining the user's historical emotional patterns, matching and verifying the preliminary emotional recognition result and the first credibility with the user's historical emotional patterns, and generating the final emotional state and the corresponding second credibility include: Based on the current context and the user's historical emotional patterns, a reasonableness benchmark for the user's emotional state in the current context is determined. The preliminary emotion recognition results are compared with the reasonableness benchmark to generate a situation reasonableness score; Based on the contextual plausibility score, the first credibility is corrected to generate the second credibility, and the preliminary emotion recognition result is output as the final emotion state.

4. The smart home control method based on emotion recognition as described in claim 3, characterized in that, The steps for determining the rationality benchmark of the user's emotional state in the current context, based on the current situation and the user's historical emotional patterns, include: From the user's historical sentiment patterns, query historical context records that are the same as or similar to the current context; Extract the user's historical emotional states and their frequency of occurrence from the historical context records; Based on the frequency of occurrence, a probability distribution representing the likelihood of different emotional states occurring in the current context is constructed as the rationality benchmark; When the current situation is the first occurrence or there is insufficient historical data, the preset general situation-emotion common sense library is invoked as a supplementary rationality benchmark.

5. The smart home control method based on emotion recognition as described in claim 3, characterized in that, The step of comparing the preliminary emotion recognition result with the reasonableness benchmark to generate a situation reasonableness score includes: From the rationality benchmark, obtain the historical occurrence probability corresponding to the preliminary emotion recognition result; Calculate the degree of difference between the preliminary emotion recognition result and the emotion state with the highest probability in the reasonableness benchmark; By combining the historical occurrence probability with the degree of difference, the situation rationality score is calculated and generated using a predefined scoring function.

6. The smart home control method based on emotion recognition as described in claim 1, characterized in that, The steps for determining the graded response strategy corresponding to the final emotional state based on the second credibility include: Obtain a preset graded response threshold, wherein the graded response threshold defines the minimum confidence level corresponding to each response level; The second confidence level is compared with the graded response threshold to determine the corresponding response level; Based on the final emotional state and the response level, the hierarchical response strategy is queried and generated from a preset strategy database. The hierarchical response strategy includes the type, intensity, and time parameters of the response operation.

7. The smart home control method based on emotion recognition as described in claim 6, characterized in that, The hierarchical response strategy includes at least two levels of strategy combination: A monitoring and recording-level strategy is used to record only the final emotional state without generating control instructions when the second confidence level is lower than the first threshold, while simultaneously activating a delayed monitoring mechanism. An environmental cue-level strategy is used to generate control instructions within a preset minimum influence range when the second confidence level is between the first threshold and the second threshold. A moderate care level strategy is used to generate control instructions containing a revocable flag when the second confidence level is between the second threshold and the third threshold. An active intervention-level strategy is used to generate a sequence of control instructions when the second confidence level is higher than the third threshold, and can trigger a notification mechanism based on the type of the final emotional state.

8. The smart home control method based on emotion recognition as described in claim 1, characterized in that, Before the step of generating and executing personalized care control instructions for smart home devices based on the final emotional state and the graded response strategy, the method further includes: Starting from the emotion recognition process, the occurrence and changes of the final emotion state are continuously monitored, and the duration of the corresponding emotion state is recorded; The duration of the emotional state is compared with a preset care initiation delay threshold to determine whether the time condition for initiating care operation has been met, and a delay condition satisfaction judgment is generated. The duration of the emotional state is compared with a preset emotional fluctuation threshold to determine whether the current emotional state is a brief emotional fluctuation, and a non-brief fluctuation judgment is generated. Based on the judgment that the delay condition is met and the judgment that it is not a transient fluctuation, the response level determined according to the second credibility is dynamically adjusted, and a graded response strategy corrected by the time dimension is generated based on the adjusted response level.

9. The smart home control method based on emotion recognition as described in claim 8, characterized in that, Dynamically adjusting the response level determined based on the second confidence level specifically includes: Based on the duration range of the emotional state, the corresponding response level adjustment direction and adjustment range are determined from the predefined duration-response level correction mapping relationship; Based on the direction of the response level adjustment, the response level is upgraded or downgraded to obtain a preliminary adjusted response level. Based on the adjustment range, the initially adjusted response level is quantitatively corrected to generate the final response level after time dimension correction. Based on the final response level, the preset strategy database is queried again to obtain the corresponding response strategy parameters and generate the time-dimension corrected hierarchical response strategy.

10. A smart home control system based on emotion recognition, characterized in that, The emotion-based smart home control system includes: a memory, a processor, and an emotion-based smart home control program stored in the memory and executable on the processor, wherein the emotion-based smart home control program is configured to implement the steps of the emotion-based smart home control method as described in any one of claims 1 to 9.

Images