Smart home control method, readable storage medium and smart home system

Abstract

The invention provides a smart home control method, a readable storage medium and a smart home system. The method comprises the steps that character characteristics of target home equipment are set at least according to equipment functions of the target home equipment, the character characteristics comprise character dimensions, and the character dimensions comprise at least one of gentle, cold, humor and fine; according to at least part of the voice data, the behavior data and the indoor environment data of the user, the emotional state of the user is determined, and the behavior data comprises use condition data of the user for the multiple intelligent devices; analyzing the emotional state and the character traits by using a large model, and at least determining an interaction strategy of the target home device, the interaction strategy comprising a response style and response content; and under the condition that the emotional interaction function of the target home equipment is started, at least controlling the target home equipment to interact with the user according to the interaction strategy. According to the invention, the personification and intelligence level of the smart home equipment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of smart home, in particular to a smart home control method, a computer readable storage medium and a smart home system. BACKGROUND

[0002] In the current field of smart home, the control and feedback of devices mainly rely on preset programs or simple voice commands, lacking personalized interactive experience. Such single and mechanical instruction execution mode makes the device appear rigid when dealing with the diversified needs of users, and the degree of intelligence is insufficient. SUMMARY

[0003] The main purpose of the present application is to provide a smart home control method, a computer readable storage medium and a smart home system to at least solve the problem of single response mechanism and insufficient intelligence of smart home devices in the prior art.

[0004] In order to achieve the above purpose, according to one aspect of the present application, a smart home control method is provided, comprising: setting personality traits of a target home device according to at least device functions of the target home device, the personality traits including personality dimensions, the personality dimensions including at least one of the following: gentleness, calmness, humor, subtlety; determining emotional state of a user according to at least part of voice data, behavior data and indoor environment data of the user, the behavior data including usage data of the user on multiple smart devices; analyzing the emotional state and the personality traits by using a large model to determine at least interaction strategy of the target home device, the interaction strategy including response style and response content; and controlling the target home device to interact with the user in the interaction strategy under the condition that emotionalized interaction function of the target home device is turned on.

[0005] Optionally, setting personality traits of a target home device according to at least device functions of the target home device comprises: determining device role of the target home device and score of multiple personality trait parameters corresponding to the device role according to device functions of the target home device, multiple personality trait parameters including gentleness, calmness, humor and subtlety; adjusting scores of multiple personality trait parameters according to self-defined setting instruction of the user under the condition that the self-defined setting instruction is received; determining the personality dimensions and dimension levels of the target home device according to scores of multiple personality trait parameters to obtain the personality traits including the device role, the personality dimensions and the dimension levels, the dimension levels representing degrees of the personality dimensions.

[0006] Optionally, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least the following: intonation, speaking speed, volume, text shape and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait.

[0007] Optionally, there are multiple target home devices, and the emotional state and the personality traits are analyzed using a large model to at least determine the interaction strategy of the target home devices, including: using the large model to analyze the emotional state and the personality traits to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scenarios of multiple target home devices to determine the interaction priority of multiple target home devices; using the large model to analyze the emotional state and the device roles of multiple target home devices to determine the control strategy of each target home device.

[0008] Optionally, at least controlling the target home devices to interact with the user according to the interaction strategy includes: controlling multiple target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; and controlling the operation of the corresponding target home devices according to each control strategy.

[0009] Optionally, the emotional state of the user is determined based on at least part of the user's voice data, behavior data and indoor environment data, including: when the voice data is received, the voice data is recognized to obtain the voice features and text features of the user; the voice features, the text features, the behavior data and the indoor environment data are input into a first multimodal emotion recognition model to obtain the emotional state, or, the voice features and the text features are feature fused to obtain the emotional state, the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical voice features, historical text features, historical behavior data, historical indoor environment data and corresponding historical emotional states; when the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state, the second multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional states.

[0010] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and the text features to obtain the emotional state.

[0011] Optionally, before inputting the sound features, the text features, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: using the OCEAN model to extract features from the behavior data to obtain behavior features; and performing feature extraction and feature conversion on the indoor environment data to obtain environmental factors.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0013] According to another aspect of the present application, a smart home system is provided, comprising: a home appliance; a controller of the home appliance, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0014] By applying the technical solution of the present application, the personality traits of smart home devices are set according to their device functions, and the user's emotional state and the personality traits of the devices are analyzed through a large model to obtain an interaction strategy that matches the user's current emotional state. The smart home devices are then controlled to interact with the user using the interaction strategy, so that the interaction between smart home devices and users is not limited to simple command issuance. Smart home devices can also communicate with users more deeply emotionally, achieving a more natural and humane interaction between devices and users, and ensuring that the smart home devices are highly intelligent and personified. This not only increases the fun of interaction, but also makes users feel closer and more comfortable, ensuring a high user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a smart home control method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of a flow chart of a smart home control method provided according to an embodiment of the present application is shown;

[0018] Figure 3 A control flow chart of a smart home provided according to an embodiment of the present application is shown.

[0019] The above drawings include the following reference numerals:

[0020] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0021] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] As introduced in the background technology, the response mechanism of smart home devices in the prior art is single and the degree of intelligence is insufficient. In order to solve the above technical problems, the embodiments of the present application provide a smart home control method, a computer-readable storage medium and a smart home system.

[0025] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure diagram of a mobile terminal of a smart home control method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the smart home control method in the embodiment of the present invention. The processor 102 executes the computer program stored in the memory 104 to execute various functional applications and data processing, thereby implementing the method described. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of such networks may include a wireless network provided by the mobile terminal's telecommunications provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] In this embodiment, a method for controlling a smart home running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 2 This is a flow chart of a smart home control method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0030] Step S201, setting a personality trait of the target home appliance based at least on the device function of the target home appliance, wherein the personality trait includes a personality dimension, and the personality dimension includes at least one of the following: gentleness, calmness, humor, and delicacy;

[0031] Specifically, the "gentleness" characterizes the softness of the target home appliance's tone when responding or the use of emoticons in its text replies; the "calmness" characterizes the directness and logic of the target home appliance's responses; the "humor" characterizes the automatic insertion of jokes or segments into the target home appliance's responses; and the "delicacy" characterizes the target home appliance's attention to and foresight of the user's potential needs. The target home appliance can have only one personality trait or multiple personality traits.

[0032] Step S202: determining the user's emotional state based on at least part of the user's voice data, behavior data, and indoor environment data, wherein the behavior data includes usage data of the user on multiple smart devices;

[0033] Specifically, the emotional state includes, but is not limited to, anger, joy, fatigue, irritability, and excitement. The voice data can be a device wake-up voice or a control command voice. The multiple smart devices include, but are not limited to, mobile phones, smart watches, smart bracelets, and various smart home devices such as air conditioners, televisions, washing machines, lighting equipment, curtains, etc.

[0034] Step S203: Analyze the emotional state and the personality traits using a large model to at least determine an interaction strategy for the target home appliance, wherein the interaction strategy includes a response style and a response content;

[0035] Specifically, the response style and the response content are consistent with the personality dimensions of the target home appliance. The response style may include a gentle response, a calm response, a humorous response, and a delicate response, etc.

[0036] Step S204: When the emotional interaction function of the target home appliance is turned on, at least the target home appliance is controlled to interact with the user using the interaction strategy.

[0037] Specifically, the emotional interaction function refers to the function of the target home appliance to interact with the user using its set personality traits.

[0038] According to the embodiment, first, personality traits are configured for the target home device based on at least the device function of the target home device; then, the user's emotional state is determined based on at least part of the user's voice data, behavior data, and indoor environment data; then, a large model is used to analyze the emotional state and personality traits, thereby at least determining an interaction strategy for the target home device; and finally, when the emotional interaction function of the target home device is turned on, at least the target home device is controlled to interact with the user according to the determined interaction strategy. This application sets the personality traits of the smart home device based on the device function, analyzes the user's emotional state and the personality traits of the device through a large model, obtains an interaction strategy that matches the user's current emotional state, and then controls the smart home device to interact with the user using the interaction strategy, so that the interaction between the smart home device and the user is not limited to simple command issuance, but the smart home device can also communicate with the user more deeply emotionally, achieving a more natural and humanized interaction between the device and the user, ensuring that the smart home device is more intelligent and personified, which not only increases the fun of interaction, but also makes the user feel closer and more comfortable, ensuring a higher user experience.

[0039] In addition to the aforementioned gentleness, calmness, humor, and delicacy, the personality dimensions can also include enthusiasm, romance, child-friendliness, elderly-friendliness, and personalized responses. Among them, enthusiasm represents that the target home appliance responds in a romantic language style, romance represents that the target home appliance responds in a simple, friendly, and childlike language style, elderly-friendliness represents that the target home appliance responds in a clear, slow, and non-complex language style, and personalized responses represent that the response style is determined each time based on the user's personal preference style. Each personality dimension is designed to adapt to different user needs and situations, providing users with a richer, more appropriate, and more humane interactive experience. By flexibly adjusting the personality dimensions of smart home devices, they can better integrate into the user's life and become an indispensable partner in the user's life.

[0040] The large model in this application can be an AI large model.

[0041] In an optional solution, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the scores of multiple personality trait parameters corresponding to the device role according to the device function of the target home device, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor and delicacy; upon receiving a customized setting instruction from the user, adjusting the scores of the multiple personality trait parameters according to the customized setting instruction; determining the personality dimension and dimension level of the target home device according to the scores of the multiple personality trait parameters, and obtaining the personality trait including the device role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

[0042] In the described embodiment, the role of the smart home device is closely integrated with its function. The device role and the score of its personality trait parameters are determined by the device function, which can ensure that the device displays the most appropriate personality characteristics in different application scenarios; and the score adjustment mechanism of the personality trait parameters allows the device to self-learn and optimize based on user feedback and usage habits, which means that the device can gradually "understand" the user, thereby providing the user with more personalized and accurate services. By giving smart home devices specific personality traits including device roles, personality dimensions and their dimension levels, the interaction between the device and the user becomes more like human communication, and is no longer a simple command execution. This kind of personified interaction increases the emotional warmth of the home environment, allowing users to feel companionship and care when using the device, thereby improving overall user satisfaction and loyalty.

[0043] Specifically, the dimension level refers to the gentleness, calmness, humor, and delicacy of the device. For example, when the personality dimension of the target home device includes delicacy, its corresponding dimension level can be general delicacy, relatively delicacy, or very delicacy. General delicacy refers to the basic level of delicacy exhibited by the device. In this case, the device's responses and behaviors will reflect moderate care and consideration, but will not be overly proactive. Relatively delicacy indicates a level of delicacy that goes beyond the basic level. In this case, the device will show more patience and careful attention in daily interactions, and may provide extra care and advice when it detects possible discomfort or needs of the user. For example, if the user's voice sounds like symptoms of a cold, the device may remind the user to rest and provide suggestions on adjusting the temperature and humidity to promote physical recovery. Very delicacy is the highest level of delicacy, which means that the device will be extremely considerate and caring when interacting with the user, almost like a caring friend or family member. In this case, the device will not only respond to the needs directly expressed by the user, but also identify potential needs or emotional changes based on deep learning and emotional intelligence, and proactively make caring responses and actions. For example, after a busy day at work, the user may say, "Thank you for your hard work. The day's fatigue is about to end. I have prepared a comfortable environment to help you relax."

[0044] In actual application, the scores of multiple personality trait parameters can be weighted and summed to obtain a sum value, and the personality dimension and dimension level can be determined based on the sum value and the range of multiple personality trait parameters; the scores of individual personality trait parameters and the size of each score relative to a predetermined threshold can also be compared, and the personality dimension can be determined based on the comparison results, and then the dimension level corresponding to the personality dimension can be determined based on the score corresponding to the personality dimension and the dimension level range.

[0045] In the embodiment, the device role is determined specifically according to the function of the device. For example, the device role corresponding to the air conditioner can be a refrigeration butler, the device role corresponding to the water heater can be a hot water guard, the device role corresponding to the camera can be a detective, the device role corresponding to the gas alarm can be a guard, and the device role corresponding to the sweeping robot can be a cleaning nanny.

[0046] In another optional scheme, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the corresponding personality dimensions according to the device function of the target home device to obtain the personality traits; and upon receiving the user's customized setting instructions, adjusting the personality dimensions according to the customized setting instructions to obtain the adjusted personality traits.

[0047] In some embodiments, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least part of the following: intonation, speaking speed, volume, text shape, and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait. In this embodiment, the large model first determines the correlation between the user's real-time emotional state and the device's personality dimensions. If the correlation is low, it indicates that the user's emotional state does not affect the device's existing personality traits. In this case, the device will respond based on the existing personality traits. This ensures that the device's behavior conforms to its "role" setting and maintains consistency without the need for special adjustments. If the correlation is high, it indicates that the user's current emotional state may not match the device's existing personality traits. In this case, the device will adjust the personality dimensions and / or dimension levels based on the user's current emotional state, so that the device's response can better adapt to the user's emotional changes. The large model ensures that the device can make corresponding personality adjustments based on the user's real-time emotional state. This personalized and contextualized response strategy can significantly improve user satisfaction.

[0048] In the embodiment, based on the user's emotional state and the personality dimensions of the device, the response style elements such as intonation, speaking speed, volume, text font and text symbols are carefully determined and adjusted through a large model, so that the device's response can be closer to the habits of human communication, making the user feel like he is communicating with a "person" who understands his emotions, rather than a machine.

[0049] For example, when the existing personality dimensions of the air conditioner are temperature and delicacy, and the user's current emotional state is detected to be fatigue, and the large model analysis determines that the two are highly correlated, the dimension levels of temperature and delicacy can be increased, and an interaction strategy can be obtained and the device can be controlled to respond to the user according to this interaction strategy, making the device's response more considerate and gentle, which helps to relieve the user's fatigue.

[0050] In specific applications, before using a large-scale model to analyze the correlation between emotional state and personality dimensions, it is first necessary to collect a large amount of user interaction data, including user voice data, text data, behavioral data, and environmental data. This data needs to be labeled with the user's emotional state. This can be achieved through direct user feedback, behavioral pattern analysis, or third-party emotion recognition services. Features related to emotional state and personality dimensions are then extracted from the collected data. A large-scale model capable of processing multimodal information (voice, text, behavior, and environmental data) is then constructed, including but not limited to a Transformer model, a multimodal fusion network, or a joint learning model for emotion recognition and personality analysis. The model is then trained using the labeled dataset, with the goal of optimizing model parameters so that it can accurately predict the user's emotional state and the personality dimensions that the device should adopt given the given features. This results in the large-scale model, which analyzes the correlation between the user's current emotional state and personality dimensions. If the correlation is less than a threshold, the device responds according to the preset personality traits. If the correlation is greater than or equal to the threshold, the device adjusts the scores of the personality dimensions and / or specific personality dimensions to better adapt to the user's emotional state and provide more personalized responses and services.

[0051] In order to determine the interaction strategy of the target home appliance more efficiently and quickly, in other embodiments, a large model is used to analyze the emotional state and the personality traits to at least determine the interaction strategy of the target home appliance, including: using the large model to analyze the emotional state, determining the response content that is consistent with the emotional state, and using the large model to analyze the personality traits, determining the response style corresponding to the personality traits, and obtaining the interaction strategy including the response content and the response style.

[0052] According to some other embodiments of the present application, the target home device is multiple, and the large model is used to analyze the emotional state and the personality trait to determine at least the interaction strategy of the target home device, comprising: using the large model to analyze the emotional state and the personality trait to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scene where the multiple target home devices are located to determine the interaction priority of the multiple target home devices; using the large model to analyze the emotional state and the device role of the multiple target home devices to determine the control strategy of each target home device. When there are multiple smart home devices in the family, the technical solution ensures that the devices can respond to user needs in an orderly and efficient manner through the intelligent coordination of the large model, and solves the problems of device response conflict and low efficiency in a multi-device environment through the comprehensive analysis of device functions, user emotions and scene understanding by the large model, so that the smart home system can actively and coordinately manage the home environment like a wise family butler.

[0053] For example, the home includes a smart curtain, a smart bulb and a smart sound, and in the scenario where the user is preparing to sleep, the large model is used to determine the interaction strategy and control strategy of the smart curtain, the smart bulb and the smart sound, and then according to the emotional state of the user and the current scene, the smart curtain is preferentially caused to respond to the user (gently broadcasting that the curtain is being pulled up for you) and execute the corresponding control strategy (close), then the smart bulb is caused to respond to the user (gently broadcasting that the light brightness is being lowered for you, wishing you a good dream, and delicately asking whether the user needs to completely turn off the bulb) and execute the corresponding control strategy (gradually reduce the brightness), and finally the smart sound is caused to respond to the user (gently broadcasting that the current is playing sleep-aiding music for you, and the music will stop after half an hour) and execute the corresponding control strategy (play soft music), the whole process is smooth and natural, without the need for the user to operate one by one, greatly improving the convenience and comfort of life.

[0054] Optionally, at least controlling the target home device to interact with the user in the interaction strategy comprises: controlling the multiple target home devices to sequentially interact with the user in the interaction priority and the interaction strategy; and controlling the corresponding target home device to operate according to the control strategy. The technical solution further realizes seamless cooperation between multiple devices by setting the interaction priority and control strategy between devices, thereby further improving the overall performance of the smart home system, while further ensuring the coordination between devices, avoiding unnecessary interference and waiting time, and further solving the problems of uncoordinated device response and poor user experience in a multi-device environment, so that the smart home system can provide continuous, efficient and personalized services to users.

[0055] In other embodiments, in the context of multi-target home devices, each device is endowed with certain "roles" and "personalities", and when these devices are performing their role tasks, their control demands can conflict with each other. To resolve such conflicts, the application also introduces a "priority voting mechanism". First, conflicts in control demands between devices are identified, which can be achieved by monitoring the requests of devices and the state of the environment. When a conflict occurs, the devices involved in the conflict will submit their control requests and the priority reasons behind them. Based on the requests and priority reasons of the devices, a virtual "voting" process is organized. Here, "voting" actually means that according to the importance of the roles of the devices, the user's set preferences, the current environmental state, and other factors, the requests of the devices are comprehensively evaluated to determine which device's request should be executed first.

[0056] An important component of the priority voting mechanism is the "user preference learning module". This module continuously learns and remembers the user's choices in similar situations in the past and their preferences for different devices through machine learning techniques. Based on the results of user preference learning, the collaboration strategy between devices can be automatically adjusted, reducing the need for human intervention. This means that when similar conflicts occur, decisions can be made automatically based on the user's historical preferences. One of the key goals throughout the mechanism is to maintain the dynamic balance of the personalities of the entire smart home device group. This not only involves resolving direct conflicts between devices, but also considers the balance between the consistency of device personalities and the efficiency of the entire system, ensuring that the behavior of each device is carried out in the context of respecting the user's overall needs.

[0057] In some example schemes, determining the emotional state of the user according to at least part of the voice data, the behavior data, and the indoor environment data includes: in a case where the voice data is received, identifying the voice data to obtain voice features and text features of the user; inputting the voice features, the text features, the behavior data, and the indoor environment data into a first multi-modal emotion recognition model to obtain the emotional state, or performing feature fusion on the voice features and the text features to obtain the emotional state, the first multi-modal emotion recognition model being trained by machine learning using a plurality of sets of data, each set of data in the plurality of sets of data including historical voice features, historical text features, historical behavior data, historical indoor environment data, and a corresponding historical emotional state; in a case where the voice data is not received, inputting the behavior data and the indoor environment data into a second multi-modal emotion recognition model to obtain the emotional state, the second multi-modal emotion recognition model being trained by machine learning using a plurality of sets of data, each set of data in the plurality of sets of data including historical behavior data, historical indoor environment data, and a corresponding historical emotional state.

[0058] In the described embodiment, when user voice data is received, a multimodal emotion recognition model is used to capture multidimensional information about the user's emotions from different data sources (voice, behavior, and environment), thereby providing a more accurate and comprehensive assessment of the user's emotional state. Alternatively, the user's emotional state is assessed through the acoustic and textual features of the voice, thereby efficiently and simply obtaining the user's emotional state. In the absence of user voice data, a multimodal emotion recognition model is used to capture information about the user's emotions from behavioral and environmental data, so that the assessment of the user's emotional state is not limited to the user's active voice expression. In this way, even without direct voice input, smart home devices can accurately predict the user's emotional state, thereby providing more natural and personalized services. This not only enhances the user experience but also reflects the intelligence and humanized design of smart home devices.

[0059] In addition, a large amount of historical data is utilized in the model training process, including historical sound features, historical text features, historical behavior data, historical indoor environment data, and corresponding historical emotional states, which enables the model to learn the complex relationship between emotional states and these features, thereby improving the recognition accuracy and generalization ability.

[0060] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed, and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and text features to obtain the emotional state. This embodiment achieves accurate recognition of the user's emotional state by combining the Wav2Vec2 and BERT models. The Wav2Vec2 model can extract rich voice features from the user's voice, such as pitch, speaking speed, and volume. These features are often closely related to people's emotions, while the BERT model focuses on understanding text content and can parse the semantics and emotional tendencies in the user's speech. The fusion of the two makes it possible not only to understand what the user says, but also to perceive how the user says it, thereby more comprehensively grasping the user's emotional state.

[0061] In some further embodiments of the present application, before inputting the sound features, the text features, the behavioral data, and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavioral data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: extracting features from the behavioral data using the OCEAN model to obtain behavioral features; and extracting and transforming features from the indoor environment data to obtain environmental factors. This technical solution further enriches the recognition dimensions of the user's emotional state by introducing the OCEAN model to identify the user's behavioral features and extracting environmental factors through environmental features, thereby further ensuring that the obtained user emotional state is relatively accurate.

[0062] In addition, this application also adds a multimodal interface that integrates VoiceXML and Emotion API during speech classification to identify the user's emotional state.

[0063] The solution of this application promotes the transformation of smart homes from simple functionality to emotional intelligence scenarios, such as emotional companionship and mood regulation.

[0064] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the smart home control method of the present application will be described in detail below with reference to specific embodiments.

[0065] Example 1

[0066] This embodiment relates to a specific smart home control method, such as Figure 3 As shown, the following steps are included:

[0067] Step S1: Set the device roles and personality dimensions of the smart home devices, and input the device roles and personality dimensions into the speech model. For example, the device role allocation can be shown in Table 1, and the personality dimension system can be shown in Table 2.

[0068] Table 1

[0069]

[0070] Table 2

[0071] Dimension Type Parameter range Example performance Gentleness 1-100 Use a softer tone and emojis Calmness 1-100 Strong logic and direct answers sense of humor 1-100 Automatically insert jokes / paragraphs Fineness 1-100 Pay attention to potential user needs (such as automatically increasing the air conditioning humidity when coughing)

[0072] After the personality dimension is set for the device, the device's emotional interaction function is set to be turned on or off in the large model, so that the device turns on or off the function.

[0073] Step S2: adjusting the response mechanism of the relevant smart home devices input into the large model according to user needs and settings;

[0074] Step S3: Receive user voice commands and perform multimodal emotion recognition. Exemplarily, the multimodal recognition process includes:

[0075] Speech sentiment analysis: Using the Wav2Vec2+BERT model, we can identify the user's emotional state (e.g., anger, happiness, fatigue). For example, if a user says, "xxxx (wake-up word), I'm so tired," we can identify the user's tiredness based on the tone and text.

[0076] Behavioral pattern analysis: Inferring user psychological needs through device usage data. For example, identifying a user's nervousness by frequently checking the camera late at night can provide prompts and comfort based on the user's current status.

[0077] Environmental perception fusion: Indoor temperature and humidity, light intensity, and noise level are combined to generate environmental emotional factors. Based on the corresponding environmental impact, this is used as a reference for evaluating emotional state. For example, if the indoor humidity is too high, people will feel irritable. The dehumidifier device can promptly ask the user and turn on the dehumidification device.

[0078] Step S4: Use the voice big model to analyze the input user emotional state, device role and device personality dimensions, and generate the device's interaction strategy so that the device responds according to the personality dimensions, device role and user emotional state.

[0079] For example, Gree+APP selects a device to set the device personality dimension: living room air conditioner (gentleness +88), role setting: cooling butler, personality traits: gentle and delicate, function implementation: voice reply, coordination of device conflicts (such as automatically dimming the lights when the oven is detected in use); interactive design: gentle voice style: "We have made an appointment for you to turn on the air conditioner cooling mode at 19:00 today"; visual interface: Gree+ voice APP assistant, the air conditioner is the device representative (icon) reply.

[0080] Example 2

[0081] Based on Example 1, a smart home control method considering a multi-device collaboration scenario includes the following steps:

[0082] Step S1': When multiple devices respond simultaneously, the master and slave roles are automatically assigned according to the scenario. For example, in a security scenario, the camera acts as the "detective" (master) and the speaker acts as the "guard" (slave);

[0083] Step S2': In the event of conflicting response strategies for multiple devices, a priority voting mechanism is used to determine the devices that are subject to limited execution. For example, the air conditioner acts as a "housekeeper" that provides a comfortable temperature, while the plant light is a "gardener" dedicated to providing optimal lighting conditions for plants. When these devices are performing their roles, their control requirements may conflict with each other. For example, on a hot day, the air conditioner may need to close the curtains to lower the indoor temperature, while the plant light requires the curtains to remain open in order to maintain the light required for plant growth. Based on the device's request and priority reasons, the requests of each device are comprehensively evaluated according to factors such as the importance of the device role, the user's set preferences, and the current environmental status to determine which device's request should be executed first. When it is found that the user tends to sacrifice temperature comfort to maintain indoor brightness when the sun is strong, the plant light's request will tend to be supported in future conflicts.

[0084] Step S3': Control the natural transition of the personality traits of multiple devices during interaction. Each device collects voice in real time through a microphone, uses a pre-trained acoustic model to extract fundamental frequency, energy and other features, combines the NLP model to analyze text emotions, and generates emotional labels (such as "anger" and "joy"). The device selects a matching strategy from the response library based on the role configuration (such as children's watches prioritize soothing, speakers focus on entertainment, and air conditioners cool down to solve the stuffy and hot environment). For example, when the user is detected to be anxious, the bedroom lights are adjusted to a warm color, the background music host plays the sound of natural stream water, and the air conditioner is adjusted to a quiet working wind.

[0085] For example, in the morning wake-up scenario (6:00-6:30) with multi-role collaboration:

[0086] The cooling manager activates "Fresh Mode": fresh air + ultrasonic humidifier work together, and gently announces: "Morning light has entered the bedroom, today's UV index is level 3";

[0087] The water dispenser (drinking water guard) announces: "There are 2 tasks to do today. It is recommended to drink 300ml of warm water first."

[0088] The sweeping robot (hygiene housekeeper) announces: "Cleaning has started for you and will be finished in half an hour."

[0089] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0090] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is run, the device where the computer-readable storage medium is located is controlled to execute the smart home control method.

[0091] Specifically, the control method of the smart home includes:

[0092] Step S201, setting a personality trait of the target home appliance based at least on the device function of the target home appliance, wherein the personality trait includes a personality dimension, and the personality dimension includes at least one of the following: gentleness, calmness, humor, and delicacy;

[0093] Specifically, the "gentleness" characterizes the softness of the target home appliance's tone when responding or the use of emoticons in its text replies; the "calmness" characterizes the directness and logic of the target home appliance's responses; the "humor" characterizes the automatic insertion of jokes or segments into the target home appliance's responses; and the "delicacy" characterizes the target home appliance's attention to and foresight of the user's potential needs. The target home appliance can have only one personality trait or multiple personality traits.

[0094] Step S202: determining the user's emotional state based on at least part of the user's voice data, behavior data, and indoor environment data, wherein the behavior data includes usage data of the user on multiple smart devices;

[0095] Specifically, the emotional state includes, but is not limited to, anger, joy, fatigue, irritability, and excitement. The voice data can be a device wake-up voice or a control command voice. The multiple smart devices include, but are not limited to, mobile phones, smart watches, smart bracelets, and various smart home devices such as air conditioners, televisions, washing machines, lighting equipment, curtains, etc.

[0096] Step S203: Analyze the emotional state and the personality traits using a large model to at least determine an interaction strategy for the target home appliance, wherein the interaction strategy includes a response style and a response content;

[0097] Specifically, the response style and the response content are consistent with the personality dimensions of the target home appliance. The response style may include a gentle response, a calm response, a humorous response, and a delicate response, etc.

[0098] Step S204: When the emotional interaction function of the target home appliance is turned on, at least the target home appliance is controlled to interact with the user using the interaction strategy.

[0099] Specifically, the emotional interaction function refers to the function of the target home appliance to interact with the user using its set personality traits.

[0100] Optionally, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the scores of multiple personality trait parameters corresponding to the device role according to the device function of the target home device, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor and delicacy; upon receiving the user's customized setting instruction, adjusting the scores of the multiple personality trait parameters according to the customized setting instruction; determining the personality dimension and dimension level of the target home device according to the scores of the multiple personality trait parameters, and obtaining the personality trait including the device role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

[0101] Optionally, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least the following: intonation, speaking speed, volume, text shape and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait.

[0102] Optionally, there are multiple target home devices, and the emotional state and the personality traits are analyzed using a large model to at least determine the interaction strategy of the target home devices, including: using the large model to analyze the emotional state and the personality traits to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scenarios of multiple target home devices to determine the interaction priority of multiple target home devices; using the large model to analyze the emotional state and the device roles of multiple target home devices to determine the control strategy of each target home device.

[0103] Optionally, at least controlling the target home devices to interact with the user according to the interaction strategy includes: controlling multiple target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; and controlling the operation of the corresponding target home devices according to each control strategy.

[0104] Optionally, the emotional state of the user is determined based on at least part of the user's voice data, behavior data and indoor environment data, including: when the voice data is received, the voice data is recognized to obtain the voice features and text features of the user; the voice features, the text features, the behavior data and the indoor environment data are input into a first multimodal emotion recognition model to obtain the emotional state, or, the voice features and the text features are feature fused to obtain the emotional state, the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical voice features, historical text features, historical behavior data, historical indoor environment data and corresponding historical emotional states; when the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state, the second multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional states.

[0105] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and the text features to obtain the emotional state.

[0106] Optionally, before inputting the sound features, the text features, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: using the OCEAN model to extract features from the behavior data to obtain behavior features; and performing feature extraction and feature conversion on the indoor environment data to obtain environmental factors.

[0107] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the smart home control method when running.

[0108] Specifically, the control method of the smart home includes:

[0109] Step S201, setting a personality trait of the target home appliance based at least on the device function of the target home appliance, wherein the personality trait includes a personality dimension, and the personality dimension includes at least one of the following: gentleness, calmness, humor, and delicacy;

[0110] Specifically, the gentleness represents the gentleness of the tone of the target home device in response or the use of emoticons in the text reply; the calmness represents the directness and logic of the reply of the target home device; the humor represents the behavior of the target home device automatically inserting jokes or monologues in the response process; and the subtlety represents the attention and predictability of the target home device to the potential needs of the user. The target home device can have only one personality trait, or can have multiple personality traits.

[0111] In step S202, a user's emotional state is determined according to at least part of voice data, behavior data and indoor environment data of the user, and the behavior data includes usage data of the user on multiple smart devices.

[0112] Specifically, the emotional state includes but is not limited to anger, joy, fatigue, irritability and excitement, etc. The voice data can be device wake-up voice or control instruction voice. The multiple smart devices include but are not limited to mobile phones, smart watches, smart bracelets, air conditioners, televisions, washing machines, lighting devices, curtains and various smart home devices.

[0113] In step S203, a large model is used to analyze the emotional state and the personality trait, and at least an interaction strategy of the target home device is determined, the interaction strategy including a response style and a response content.

[0114] Specifically, the response style and the response content are consistent with the personality dimensions possessed by the target home device, and the response style can include gentle response, calm response, humorous response and subtle response, etc.

[0115] In step S204, at least the target home device is controlled to interact with the user in the interaction strategy when the emotional interaction function of the target home device is turned on.

[0116] Specifically, the emotional interaction function refers to the function of the target home device interacting with the user by using the set personality trait.

[0117] Optionally, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the scores of multiple personality trait parameters corresponding to the device role according to the device function of the target home device, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor and delicacy; upon receiving the user's customized setting instruction, adjusting the scores of the multiple personality trait parameters according to the customized setting instruction; determining the personality dimension and dimension level of the target home device according to the scores of the multiple personality trait parameters, and obtaining the personality trait including the device role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

[0118] Optionally, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least the following: intonation, speaking speed, volume, text shape and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait.

[0119] Optionally, there are multiple target home devices, and the emotional state and the personality traits are analyzed using a large model to at least determine the interaction strategy of the target home devices, including: using the large model to analyze the emotional state and the personality traits to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scenarios of multiple target home devices to determine the interaction priority of multiple target home devices; using the large model to analyze the emotional state and the device roles of multiple target home devices to determine the control strategy of each target home device.

[0120] Optionally, at least controlling the target home devices to interact with the user according to the interaction strategy includes: controlling multiple target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; and controlling the operation of the corresponding target home devices according to each control strategy.

[0121] Optionally, the emotional state of the user is determined based on at least part of the user's voice data, behavior data and indoor environment data, including: when the voice data is received, the voice data is recognized to obtain the voice features and text features of the user; the voice features, the text features, the behavior data and the indoor environment data are input into a first multimodal emotion recognition model to obtain the emotional state, or, the voice features and the text features are feature fused to obtain the emotional state, the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical voice features, historical text features, historical behavior data, historical indoor environment data and corresponding historical emotional states; when the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state, the second multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional states.

[0122] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and the text features to obtain the emotional state.

[0123] Optionally, before inputting the sound features, the text features, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: using the OCEAN model to extract features from the behavior data to obtain behavior features; and performing feature extraction and feature conversion on the indoor environment data to obtain environmental factors.

[0124] An embodiment of the present invention provides a smart home system, comprising: home appliances; a controller for the home appliances, wherein the controller comprises a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are performed:

[0125] Step S201, setting a personality trait of the target home appliance based at least on the device function of the target home appliance, wherein the personality trait includes a personality dimension, and the personality dimension includes at least one of the following: gentleness, calmness, humor, and delicacy;

[0126] Specifically, the "gentleness" characterizes the softness of the target home appliance's tone when responding or the use of emoticons in its text replies; the "calmness" characterizes the directness and logic of the target home appliance's responses; the "humor" characterizes the automatic insertion of jokes or segments into the target home appliance's responses; and the "delicacy" characterizes the target home appliance's attention to and foresight of the user's potential needs. The target home appliance can have only one personality trait or multiple personality traits.

[0127] Step S202: determining the user's emotional state based on at least part of the user's voice data, behavior data, and indoor environment data, wherein the behavior data includes usage data of the user on multiple smart devices;

[0128] Specifically, the emotional state includes, but is not limited to, anger, joy, fatigue, irritability, and excitement. The voice data can be a device wake-up voice or a control command voice. The multiple smart devices include, but are not limited to, mobile phones, smart watches, smart bracelets, and various smart home devices such as air conditioners, televisions, washing machines, lighting equipment, curtains, etc.

[0129] Step S203: Analyze the emotional state and the personality traits using a large model to at least determine an interaction strategy for the target home appliance, wherein the interaction strategy includes a response style and a response content;

[0130] Specifically, the response style and the response content are consistent with the personality dimensions of the target home appliance. The response style may include a gentle response, a calm response, a humorous response, and a delicate response, etc.

[0131] Step S204: When the emotional interaction function of the target home appliance is turned on, at least the target home appliance is controlled to interact with the user using the interaction strategy.

[0132] Specifically, the emotional interaction function refers to the function of the target home appliance to interact with the user using its set personality traits.

[0133] The controller in this article can be a PC, PAD, mobile phone, etc.

[0134] Optionally, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the scores of multiple personality trait parameters corresponding to the device role according to the device function of the target home device, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor and delicacy; upon receiving the user's customized setting instruction, adjusting the scores of the multiple personality trait parameters according to the customized setting instruction; determining the personality dimension and dimension level of the target home device according to the scores of the multiple personality trait parameters, and obtaining the personality trait including the device role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

[0135] Optionally, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least the following: intonation, speaking speed, volume, text shape and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait.

[0136] Optionally, there are multiple target home devices, and the emotional state and the personality traits are analyzed using a large model to at least determine the interaction strategy of the target home devices, including: using the large model to analyze the emotional state and the personality traits to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scenarios of multiple target home devices to determine the interaction priority of multiple target home devices; using the large model to analyze the emotional state and the device roles of multiple target home devices to determine the control strategy of each target home device.

[0137] Optionally, at least controlling the target home devices to interact with the user according to the interaction strategy includes: controlling multiple target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; and controlling the operation of the corresponding target home devices according to each control strategy.

[0138] Optionally, the emotional state of the user is determined based on at least part of the user's voice data, behavior data and indoor environment data, including: when the voice data is received, the voice data is recognized to obtain the voice features and text features of the user; the voice features, the text features, the behavior data and the indoor environment data are input into a first multimodal emotion recognition model to obtain the emotional state, or, the voice features and the text features are feature fused to obtain the emotional state, the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical voice features, historical text features, historical behavior data, historical indoor environment data and corresponding historical emotional states; when the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state, the second multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional states.

[0139] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and the text features to obtain the emotional state.

[0140] Optionally, before inputting the sound features, the text features, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: using the OCEAN model to extract features from the behavior data to obtain behavior features; and performing feature extraction and feature conversion on the indoor environment data to obtain environmental factors.

[0141] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and by adjusting kernel parameters, the problem of the single response mechanism and insufficient intelligence level of smart home devices in the prior art can be at least solved.

[0142] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0143] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0144] Step S201, setting a personality trait of the target home appliance based at least on the device function of the target home appliance, wherein the personality trait includes a personality dimension, and the personality dimension includes at least one of the following: gentleness, calmness, humor, and delicacy;

[0145] Step S202: determining the user's emotional state based on at least part of the user's voice data, behavior data, and indoor environment data, wherein the behavior data includes usage data of the user on multiple smart devices;

[0146] Step S203: Analyze the emotional state and the personality traits using a large model to at least determine an interaction strategy for the target home appliance, wherein the interaction strategy includes a response style and a response content;

[0147] Step S204: When the emotional interaction function of the target home appliance is turned on, at least the target home appliance is controlled to interact with the user using the interaction strategy.

[0148] Optionally, the personality traits of the target home device are set at least according to the device function of the target home device, including: determining the device role of the target home device and the scores of multiple personality trait parameters corresponding to the device role according to the device function of the target home device, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor and delicacy; upon receiving the user's customized setting instruction, adjusting the scores of the multiple personality trait parameters according to the customized setting instruction; determining the personality dimension and dimension level of the target home device according to the scores of the multiple personality trait parameters, and obtaining the personality trait including the device role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

[0149] Optionally, a large model is used to analyze the emotional state and the personality trait to at least determine the interaction strategy of the target home appliance, including: using a large model to analyze the correlation between the emotional state and the personality dimension; when the correlation is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, and the response style includes at least the following: intonation, speaking speed, volume, text shape and text symbols; when the correlation is greater than or equal to the threshold, adjusting the personality dimension and at least one of the dimension levels corresponding to the personality dimension according to the emotional state to obtain the adjusted personality trait, and determining that the interaction strategy includes the response content and the response style corresponding to the adjusted personality trait.

[0150] Optionally, there are multiple target home devices, and the emotional state and the personality traits are analyzed using a large model to at least determine the interaction strategy of the target home devices, including: using the large model to analyze the emotional state and the personality traits to determine the interaction strategy of each target home device; using the large model to analyze the emotional state and the application scenarios of multiple target home devices to determine the interaction priority of multiple target home devices; using the large model to analyze the emotional state and the device roles of multiple target home devices to determine the control strategy of each target home device.

[0151] Optionally, at least controlling the target home devices to interact with the user according to the interaction strategy includes: controlling multiple target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; and controlling the operation of the corresponding target home devices according to each control strategy.

[0152] Optionally, the emotional state of the user is determined based on at least part of the user's voice data, behavior data and indoor environment data, including: when the voice data is received, the voice data is recognized to obtain the voice features and text features of the user; the voice features, the text features, the behavior data and the indoor environment data are input into a first multimodal emotion recognition model to obtain the emotional state, or, the voice features and the text features are feature fused to obtain the emotional state, the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical voice features, historical text features, historical behavior data, historical indoor environment data and corresponding historical emotional states; when the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state, the second multimodal emotion recognition model is trained by machine learning using multiple sets of data, each of the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional states.

[0153] Optionally, the voice data is recognized to obtain the voice features and text features of the user, including: using a Wav2Vec2 model to recognize the voice features corresponding to the voice data, the voice features including pitch, speaking speed and volume; using a BERT model to recognize the text features corresponding to the voice data; and fusing the voice features and the text features to obtain the emotional state.

[0154] Optionally, before inputting the sound features, the text features, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: using the OCEAN model to extract features from the behavior data to obtain behavior features; and performing feature extraction and feature conversion on the indoor environment data to obtain environmental factors.

[0155] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0156] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0160] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0162] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0163] The technical features of the above-described embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0165] From the above description, it can be seen that the embodiments described in this application achieve the following technical effects:

[0166] In the smart home control method of the present application, first, personality traits are configured for the target home device based on at least the device function of the target home device; then, the user's emotional state is determined based on at least part of the user's voice data, behavior data, and indoor environment data; then, a large model is used to analyze the emotional state and personality traits, thereby at least determining an interaction strategy for the target home device; and finally, when the emotional interaction function of the target home device is enabled, at least the target home device is controlled to interact with the user according to the determined interaction strategy. The present application sets the personality traits of the smart home device based on the device function, analyzes the user's emotional state and the personality traits of the device through a large model, obtains an interaction strategy that matches the user's current emotional state, and then controls the smart home device to interact with the user using the interaction strategy, so that the interaction between the smart home device and the user is not limited to simple command issuance, but the smart home device can also communicate with the user more deeply emotionally, achieving a more natural and more humane interaction between the device and the user, ensuring that the smart home device has a high degree of intelligence and anthropomorphism, which not only increases the fun of interaction, but also makes the user feel closer and more comfortable, ensuring a high user experience.

[0167] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A control method for a smart home, characterized in that: include: Setting a personality trait of the target home appliance based at least on a function of the target home appliance, the personality trait comprising a personality dimension, the personality dimension comprising at least one of the following: gentleness, calmness, humor, and delicacy; determining an emotional state of the user based on at least part of the user's voice data, behavior data, and indoor environment data, wherein the behavior data includes usage data of the user on multiple smart devices; Analyzing the emotional state and the personality traits using a large model to at least determine an interaction strategy for the target home appliance, the interaction strategy including a response style and a response content; When the emotional interaction function of the target home appliance is turned on, at least the target home appliance is controlled to interact with the user according to the interaction strategy.

2. The method according to claim 1, characterized in that Setting the personality traits of the target home device based at least on the device function of the target home device includes: Determining, based on the device function of the target home device, a device role of the target home device and scores of multiple personality trait parameters corresponding to the device role, wherein the multiple personality trait parameters include gentleness, calmness, sense of humor, and delicacy; Upon receiving the user's custom setting instruction, adjusting the scores of the plurality of personality trait parameters according to the custom setting instruction; The personality dimension and dimension level of the target home appliance are determined based on the scores of the plurality of personality trait parameters, to obtain the personality trait including the appliance role, the personality dimension and the dimension level, wherein the dimension level represents the degree of the personality dimension.

3. The method according to claim 2, characterized in that The emotional state and the personality traits are analyzed using a large model to at least determine an interaction strategy for the target home appliance, including: Using a large model to analyze the correlation between the emotional state and the personality dimension; When the correlation degree is less than a threshold, determining that the interaction strategy includes the response content and the response style corresponding to the personality trait, wherein the response style includes at least the following: intonation, speech speed, volume, text font, and text symbols; When the degree of correlation is greater than or equal to the threshold, the personality dimension and at least one of the dimension levels corresponding to the personality dimension are adjusted according to the emotional state to obtain an adjusted personality trait, and the interaction strategy is determined to include the response content and the response style corresponding to the adjusted personality trait.

4. The method according to claim 2, characterized in that There are multiple target home devices, and a large model is used to analyze the emotional state and the personality traits to at least determine the interaction strategy of the target home devices, including: Analyzing the emotional state and the personality traits using the large model to determine the interaction strategy for each of the target home appliances; Using the large model to analyze the emotional state and the application scenarios of the multiple target home appliances, and determine the interaction priorities of the multiple target home appliances; The large model is used to analyze the emotional state and the device roles of the plurality of target home devices, and a control strategy for each of the target home devices is determined.

5. The method according to claim 4, characterized in that At least controlling the target home device to interact with the user using the interaction strategy includes: Controlling the plurality of target home devices to interact with the user in sequence according to the interaction priority and the interaction strategy; The operation of the corresponding target home appliance is controlled according to each control strategy.

6. The method according to claim 1, characterized in that Determining the user's emotional state based on at least part of the user's voice data, behavior data, and indoor environment data includes: When the voice data is received, the voice data is recognized to obtain the voice characteristics and text characteristics of the user; Inputting the sound features, the text features, the behavior data, and the indoor environment data into a first multimodal emotion recognition model to obtain the emotional state, or performing feature fusion on the sound features and the text features to obtain the emotional state, wherein the first multimodal emotion recognition model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data including: historical sound features, historical text features, historical behavior data, historical indoor environment data, and corresponding historical emotional states; In the case where the voice data is not received, the behavior data and the indoor environment data are input into a second multimodal emotion recognition model to obtain the emotional state. The second multimodal emotion recognition model is trained through machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: historical behavior data, historical indoor environment data and corresponding historical emotional state.

7. The method according to claim 6, characterized in that Recognizing the voice data to obtain the user's voice features and text features includes: Using a Wav2Vec2 model to identify the sound features corresponding to the voice data, the sound features including pitch, speaking speed, and volume; Using the BERT model to identify the text features corresponding to the voice data; The sound feature and the text feature are fused to obtain the emotional state.

8. The method according to claim 6, characterized in that Before inputting the sound feature, the text feature, the behavior data and the indoor environment data into the first multimodal emotion recognition model, or, Before inputting the behavior data and the indoor environment data into the second multimodal emotion recognition model, the method further includes: Using the OCEAN model to extract features from the behavior data to obtain behavior features; Feature extraction and feature conversion are performed on the indoor environment data to obtain environmental factors.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

10. A smart home system, characterized in that: include: Home appliances; The controller of the home device includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 8.

Images