Multi-mode-based smart home system, method, device, equipment and medium

Abstract

The invention provides a multi-mode-based smart home system, method, device and equipment and a medium. The smart home system comprises a multi-mode data acquisition module, a multi-mode intention recognition module, an active energy-saving control module and a plurality of smart home devices. Wherein the multi-modal data acquisition module is used for acquiring target user data and target environment data of different modals; the multi-mode intention recognition module is used for determining a target user intention according to the target user data and the target environment data of different modes; and the active energy-saving control module is used for determining a target smart home device from the plurality of smart home devices according to the target user intention, and performing energy-saving control on the target smart home device in combination with the target environment data and the target user intention. Through the embodiment of the invention, the problems of energy waste and the like caused by mismatching of equipment operation and user requirements can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of smart homes, and in particular to a control method for smart home devices based on multimodality, a control device for smart home devices based on multimodality, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularity of smart home devices, more and more smart devices are connected to home networks. However, traditional smart home devices usually only operate according to preset conditions or simple sensor information, and cannot fully understand the user's true intentions, which may lead to energy waste. Summary of the invention

[0003] In view of the above problems, a control method of a multi-modal smart home device, a control device of a multi-modal smart home device, an electronic device and a computer-readable storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A multimodal smart home system, comprising: a multimodal data acquisition module, a multimodal intent recognition module, an active energy-saving control module, and a plurality of smart home devices; wherein:

[0005] The multimodal data acquisition module is used to collect target user data and target environment data in different modalities;

[0006] The multimodal intention recognition module is used to determine the target user intention based on the target user data and the target environment data in different modalities;

[0007] The active energy-saving control module is used to determine a target smart home device from multiple smart home devices according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention.

[0008] Optionally, the smart home system further includes:

[0009] The multimodal data preprocessing module is used to preprocess the target user data and the target environment data of different modalities, and send the preprocessed target user data and the target environment data of different modalities to the multimodal intention recognition module for intention recognition.

[0010] Optionally, the preprocessing includes at least one of the following processing methods:

[0011] Noise reduction, feature extraction, and data normalization.

[0012] Optionally, the smart home system further includes:

[0013] The user feedback and self-learning module is used to receive user feedback information after the active energy-saving control module performs energy-saving control on the target smart home device; and optimize the multimodal intention recognition module according to the user feedback information.

[0014] Optionally, the active energy-saving control module is used to determine a target energy-saving adjustment strategy based on the target environment data; determine a target energy-saving adjustment strategy based on the target environment data and the target user intention; and perform energy-saving control on the target smart home device according to the target energy-saving adjustment strategy.

[0015] Optionally, the active energy-saving control module is used to shut down at least one smart home device among the multiple smart home devices according to the target environment data and a preset energy-saving strategy when no one is present.

[0016] Optionally, the target user data includes at least one of the following data:

[0017] Image data, voice data.

[0018] Optionally, the target environment data includes at least one of the following data:

[0019] Temperature data, humidity data, light intensity data.

[0020] The embodiment of the present invention further provides a control method of a multi-modal smart home device, which is applied to the smart home system as described above; the method comprises:

[0021] Determine the target user intent of the user; the target user intent is determined based on target user data and target environment data of different modalities;

[0022] According to the target user intention, a target smart home device is determined from a plurality of smart home devices, and energy-saving control is performed on the target smart home device in combination with the target environment data and the target user intention.

[0023] The embodiment of the present invention further provides a control device for a multi-modal smart home device, characterized in that it is applied to the smart home system as described above; the device comprises:

[0024] A determination module, used to determine the target user intention of the user; the target user intention is determined based on target user data and target environment data of different modalities;

[0025] The control module is used to determine a target smart home device from a plurality of smart home devices according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention.

[0026] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the control method of the multi-modal smart home device is implemented as described above.

[0027] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the control method of a multi-modal smart home device is implemented as described above.

[0028] The embodiments of the present invention have the following advantages:

[0029] In an embodiment of the present invention, the smart home system includes: a multimodal data acquisition module, a multimodal intention recognition module, an active energy-saving control module, and a plurality of smart home devices; wherein: the multimodal data acquisition module is used to collect target user data and target environment data of different modalities; the multimodal intention recognition module is used to determine the target user intention according to the target user data and target environment data of different modalities; the active energy-saving control module is used to determine the target smart home device from a plurality of smart home devices according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention. Through the embodiment of the present invention, the user data and environment data of different modalities can be combined to predict the user's intention, and the smart home device can be energy-saving controlled based on the user's intention. Through the embodiment of the present invention, the problem of energy waste caused by the mismatch between device operation and user demand can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0031] Figure 1 is a structural schematic diagram of a multi-modal smart home system according to an embodiment of the present invention;

[0032] Figure 2 is a structural diagram of another multi-modal smart home system according to an embodiment of the present invention;

[0033] Figure 3 is a flowchart of a control method of a multi-modal smart home device according to an embodiment of the present invention;

[0034] Figure 4 It is a structural block diagram of a control device of a multi-modal smart home device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Traditional smart home systems usually rely on single sensor information or preset scenarios for control, and cannot fully understand the user's true intentions, and cannot take proactive energy-saving measures based on user behavior and environmental changes, resulting in a mismatch between device operation and user needs, which results in energy waste. In addition, the interaction methods of existing smart home systems are usually limited to mobile phone APPs (Applications) or simple voice control, which are cumbersome and not intuitive enough.

[0037] For example, when a user leaves a room and forgets to turn off the lights or air conditioner, the system cannot respond in time. Existing intelligent control systems still have a lot of room for improvement in energy efficiency optimization and human-computer interaction. Therefore, how to accurately identify user intentions through multimodal information and take proactive energy-saving measures has become an important research topic in the current smart home field.

[0038] In view of this, the embodiment of the present invention proposes a multi-modal smart home system, which can integrate user data and environmental data of different modalities to predict the user's intention, and perform energy-saving control on smart home devices based on the user's intention. Through the embodiment of the present invention, problems such as energy waste caused by the mismatch between device operation and user needs can be avoided.

[0039] For details, please refer to Figure 1 , shows a schematic diagram of the structure of a multi-modal smart home system according to an embodiment of the present invention. Figure 1 As shown, the multi-modal smart home system 10 may include:

[0040] A multimodal data acquisition module 110, a multimodal intention recognition module 120, an active energy-saving control module 130, and a plurality of smart home devices 140; wherein:

[0041] A multimodal data collection module 110, used to collect target user data and target environment data in different modalities;

[0042] A multimodal intention recognition module 120, for determining the target user intention based on target user data and target environment data in different modalities;

[0043] The active energy-saving control module 130 is used to determine a target smart home device from a plurality of smart home devices 140 according to the target user's intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user's intention.

[0044] In an embodiment of the present invention, a multimodal smart home system 10 may be composed of a multimodal data acquisition module 110, a multimodal intention recognition module 120, an active energy-saving control module 130, and a plurality of smart home devices 140; wherein the multimodal data acquisition module 110 may be used to collect target user data and target environment data of different modalities so as to predict the user's intention from different dimensions.

[0045] In an embodiment of the present invention, the target user data may include at least one of the following data: image data and voice data.

[0046] Exemplarily, the target user data may include image data collected from the user, and voice data collected from the user; the user may interact with the smart home system 10 through voice, vision, expression, etc.

[0047] Among them, image data can be collected by image acquisition devices such as cameras deployed in the smart home environment; voice data can be collected by voice acquisition devices such as microphones deployed in the smart home environment.

[0048] The multimodal data acquisition module 110 may include devices such as cameras and microphones deployed in a smart home environment, which is not limited in this embodiment of the present invention.

[0049] In an embodiment of the present invention, the target environment data may include at least one of the following data: temperature data, humidity data, and light intensity data.

[0050] Exemplarily, the target environment data may include temperature data, humidity data, light intensity data, etc. collected for the smart home environment deployed by the smart home system 10, and the embodiment of the present invention is not limited to this.

[0051] Among them, temperature data can be collected through temperature collection devices such as thermometers deployed in smart home environments; humidity data can be collected through humidity collection devices such as hygrometers deployed in smart home environments; light intensity data can be collected through light intensity collection devices such as light sensors deployed in smart home environments.

[0052] The multimodal data acquisition module 110 may also include a thermometer, a hygrometer, a temperature-humidity meter, a light sensor, etc. deployed in a smart home environment, and the embodiment of the present invention is not limited to this.

[0053] After collecting target user data and target environment data in different modalities, the multimodal data collection module 110 may send the target user data and target environment data to the multimodal intention recognition module 120 .

[0054] After obtaining the target user data and target environment data of different modalities, the multimodal intention recognition module 120 can use a deep learning model and a multimodal information fusion algorithm to perform a comprehensive analysis of multimodal information such as the target user data and the target environment data to predict the user's current target user intention.

[0055] Among them, deep learning model:

[0056] Convolutional Neural Network (CNN): Commonly used to process vision-related data, such as user facial expressions and gesture image information collected from cameras. CNN can automatically extract features from images through structures such as convolutional layers and pooling layers. For example, when recognizing gestures, the convolutional layer can detect basic features such as edges and lines at different positions in the gesture image, and the pooling layer compresses and refines these features, reducing the amount of data while retaining key features. After multiple layers of such operations, a high-level semantic feature vector that can represent the gesture is finally obtained for subsequent intent judgment. For example, when recognizing a dimming gesture of raising the hand, CNN can accurately extract key visual features such as the gesture contour to assist in judging the intent.

[0057] Recurrent Neural Network (RNN) and its variants (such as LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit)): very suitable for processing sequence data. Voice data is a typical sequence data that changes over time. Taking the long short-term memory network (LSTM) as an example, it can handle the semantic association between the front and back of the voice signal very well, avoiding the gradient vanishing problem of traditional neural networks when processing long sequences. For example, for the sentence "a little dark" said by the user, LSTM can understand the semantics of each word according to the order of speech, and accurately extract key semantic information based on the context to provide a basis for intent recognition. Moreover, when faced with longer voice commands, it can still effectively capture the core meaning.

[0058] Deep Neural Network (DNN): It can be used as a general model structure to further perform nonlinear transformation and feature combination on processed multimodal data. For example, features extracted from speech, features extracted from gesture images, and features after normalization of environmental data can be integrated through DNN to explore the hidden correlation between different modal features and finally output results related to user intentions.

[0059] Multimodal information fusion algorithm:

[0060] Early fusion (also called feature layer fusion): After the data preprocessing stage is completed, the data features from different modalities (such as speech, vision, environment, etc.) are directly spliced ​​together to form a new comprehensive feature vector, which is then sent to the deep learning model for training and intent recognition. For example, the vector representing the speech content features extracted from the speech signal, the gesture feature vector extracted from the gesture image, and the feature vectors such as ambient temperature and humidity, and light intensity are spliced ​​in sequence, allowing the model to simultaneously learn the combination pattern of these multimodal features to judge intent. The advantage of this method is that it can make full use of the original feature information, but it may cause difficulties in fusion due to differences in the dimensions and scales of different modal data.

[0061] Late fusion (also called decision-level fusion): The data of each modality is processed and analyzed by an independent deep learning model to obtain their own preliminary judgment results on the user's intention (for example, the voice modality model determines that the user may want to adjust a certain device, and the visual modality model also reaches a similar conclusion, etc.), and then the decision results of these different modalities are fused through some fusion strategies (such as voting method, weighted average method, etc.) to obtain the final user intention judgment. For example, for the user's intention to increase the brightness of the light, the voice model determines that there is a high probability that the light is to be brightened based on the voice command, and the gesture model also determines that it is related to adjusting the light based on the gesture action. The final intention is determined by weighted averaging based on the confidence of the two. This method is relatively flexible, but it may lose some information on the early interaction and fusion between modalities.

[0062] Fusion algorithm based on attention mechanism: The attention mechanism can dynamically assign different weights to information of different modalities, focusing on more critical modal information according to the focus of the current task. For example, when recognizing the user's intention to adjust the brightness of the light, if the voice command is clear and unambiguous, the attention mechanism will give a higher weight to the features corresponding to the voice modality, and will also refer to other modal information such as gestures, but with a slightly lower relative weight. It can adaptively fuse multimodal information according to the specific situation, and better explore the differences in the importance of different modalities for intent recognition in different scenarios.

[0063] For example, if the voice data is "a bit dark" and the external environment is daytime, it can be predicted that the user wants to open the curtains; for another example, if the voice data is "a bit dark" and the external environment is cloudy, it can be predicted that the user wants to turn on the lights.

[0064] After determining the target user intention of the user, the multimodal intention recognition module 120 may send the target user intention to the active energy saving control module 130 .

[0065] After receiving the target user's intention, the active energy-saving control module 130 may first determine the target smart home device that the user currently wants to control from the plurality of smart home devices 140 according to the target user's intention.

[0066] In addition, the active energy-saving control module 130 can also obtain target environment data; then, the active energy-saving control module 130 can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0067] For example: if the target user intends to increase the brightness of the light, the active energy-saving control module 130 can automatically brighten the light; in addition, when brightening the light, the active energy-saving control module 130 can also adjust the light to a more energy-saving lighting state according to the ambient light conditions; that is, while satisfying the user's target user intention, the target smart home device is controlled to operate in energy-saving mode.

[0068] In an embodiment of the present invention, the smart home system 10 includes: a multimodal data acquisition module 110, a multimodal intention recognition module 120, an active energy-saving control module 130, and a plurality of smart home devices 140; wherein: the multimodal data acquisition module 110 is used to collect target user data and target environment data of different modes; the multimodal intention recognition module 120 is used to determine the target user intention according to the target user data and target environment data of different modes; the active energy-saving control module 130 is used to determine the target smart home device from the plurality of smart home devices 140 according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention. Through the embodiment of the present invention, the user's intention can be predicted by combining the user data and environment data of different modes, and the smart home device 140 can be energy-saving controlled based on the user's intention. Through the embodiment of the present invention, the problems such as energy waste caused by the mismatch between device operation and user demand can be avoided.

[0069] In one embodiment of the present invention, Figure 2 As shown, the smart home system 10 also includes:

[0070] The multimodal data preprocessing module 150 is used to preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module 120 for intention recognition.

[0071] In some feasible embodiments, the smart home system 10 may further include a multimodal data preprocessing module 150 ; the multimodal data preprocessing module 150 may be used to preprocess target user data and target environment data of different modalities.

[0072] In one embodiment of the present invention, the preprocessing includes at least one of the following processing methods: noise reduction, feature extraction, and data standardization.

[0073] Specifically, the multimodal data preprocessing module 150 can be used to perform noise reduction processing on the speech data.

[0074] The multimodal data preprocessing module 150 may also be used to extract features from image data; for example, if the image data is collected based on user gestures, gesture features may be extracted from the image data.

[0075] The multimodal data preprocessing module 150 may also be used to perform data standardization on the target user data and target environment data of different modalities collected by the multimodal data collection module 110 to ensure the accuracy and validity of the data.

[0076] Exemplarily, the multimodal data preprocessing module 150 can perform noise reduction on the collected voice and extract the keyword "a bit dark"; perform image processing on the image data captured by the camera and identify the user's dimming gestures, etc. The embodiment of the present invention is not limited to this.

[0077] In some feasible embodiments, the multimodal data collection module 110 may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0078] After collecting target user data and target environment data of different modalities, the multimodal data collection module 110 may send the target user data and target environment data to the multimodal data preprocessing module 150 .

[0079] Then, the multimodal data preprocessing module 150 can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module 120 for intention recognition.

[0080] After obtaining the pre-processed target user data and target environment data of different modalities, the multimodal intention recognition module 120 can use a deep learning model and a multimodal information fusion algorithm to perform a comprehensive analysis of multimodal information such as the pre-processed target user data and target environment data to predict the user's current target user intention.

[0081] After determining the target user intention of the user, the multimodal intention recognition module 120 may send the target user intention to the active energy saving control module 130 .

[0082] After receiving the target user's intention, the active energy-saving control module 130 may first determine the target smart home device that the user currently wants to control from the plurality of smart home devices 140 according to the target user's intention.

[0083] In addition, the active energy-saving control module 130 can also obtain target environment data; then, the active energy-saving control module 130 can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0084] In one embodiment of the present invention, the smart home system 10 further includes:

[0085] The user feedback and self-learning module 160 is used to receive user feedback information after the active energy-saving control module 130 performs energy-saving control on the target smart home device; and optimize the multimodal intention recognition module 120 according to the user feedback information.

[0086] In some feasible embodiments, the smart home system 10 may further include a user feedback and self-learning module 160; the user feedback and self-learning module 160 may be used to inquire about the user's satisfaction with the results of the energy-saving control after the active energy-saving control module 130 performs energy-saving control on the target smart home device.

[0087] The user can provide feedback on the inquiry through voice or interface; at this time, the user feedback and self-learning module 160 can receive the user feedback information. Then, the user feedback and self-learning module 160 can optimize the multimodal intention recognition module 120 according to the user feedback information.

[0088] Specifically, the user feedback and self-learning module 160 can record the user's preferences based on the user feedback information, and use the machine learning algorithm to continuously optimize the intention recognition and control strategy.

[0089] For example, when the user is satisfied with the adjusted light brightness, the smart home system 10 will record this preference and give priority to adopting this strategy in similar scenarios.

[0090] Exemplarily, the multimodal data preprocessing module 150 can perform noise reduction on the collected voice and extract the keyword "a bit dark"; perform image processing on the image data captured by the camera and identify the user's dimming gestures, etc. The embodiment of the present invention is not limited to this.

[0091] In some feasible embodiments, the multimodal data collection module 110 may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0092] After collecting target user data and target environment data of different modalities, the multimodal data collection module 110 may send the target user data and target environment data to the multimodal data preprocessing module 150 .

[0093] Then, the multimodal data preprocessing module 150 can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module 120 for intention recognition.

[0094] After obtaining the pre-processed target user data and target environment data of different modalities, the multimodal intention recognition module 120 can use a deep learning model and a multimodal information fusion algorithm to perform a comprehensive analysis of multimodal information such as the pre-processed target user data and target environment data to predict the user's current target user intention.

[0095] After determining the target user intention of the user, the multimodal intention recognition module 120 may send the target user intention to the active energy saving control module 130 .

[0096] After receiving the target user's intention, the active energy-saving control module 130 may first determine the target smart home device that the user currently wants to control from the plurality of smart home devices 140 according to the target user's intention.

[0097] In addition, the active energy-saving control module 130 can also obtain target environment data; then, the active energy-saving control module 130 can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0098] After the active energy-saving control module 130 performs energy-saving control on the target smart home device, the user feedback and self-learning module 160 may inquire about the user's satisfaction with the result of the energy-saving control.

[0099] The user can provide feedback on the inquiry through voice or interface; at this time, the user feedback and self-learning module 160 can receive the user feedback information. Then, the user feedback and self-learning module 160 can optimize the multimodal intention recognition module 120 according to the user feedback information.

[0100] In one embodiment of the present invention, the active energy-saving control module 130 is used to determine a target energy-saving adjustment strategy according to target environment data and target user intention; and perform energy-saving control on target smart home devices according to the target energy-saving adjustment strategy.

[0101] In some feasible embodiments, after receiving the target user's intention, the active energy-saving control module 130 may determine the target smart home device that the user wants to control.

[0102] In addition, on this basis, the active energy-saving control module 130 can also determine a target energy-saving adjustment strategy for the target smart home device according to the target environment data and the target user intention.

[0103] For example: the target smart home device is a lamp, and the user expects the control strategy for the target smart home device to be to increase the brightness; then the active energy-saving control module 130 can, based on this, determine the specific adjustment brightness (i.e., the target energy-saving adjustment strategy) according to the target environment data (here is the light intensity data); the specific adjustment brightness can meet the user's needs while avoiding the problem of resource waste due to excessive brightness.

[0104] After determining the target energy-saving adjustment strategy, the active energy-saving control module 130 can perform energy-saving control on the target smart home device according to the target energy-saving adjustment strategy. Continuing with the above example, the brightness of the lamp can be adjusted to a brightness that meets the user's needs and does not cause resource waste. The embodiment of the present invention does not limit this.

[0105] Exemplarily, the multimodal data collection module 110 may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0106] After collecting target user data and target environment data of different modalities, the multimodal data collection module 110 may send the target user data and target environment data to the multimodal data preprocessing module 150 .

[0107] Then, the multimodal data preprocessing module 150 can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module 120 for intention recognition.

[0108] After obtaining the pre-processed target user data and target environment data of different modalities, the multimodal intention recognition module 120 can use a deep learning model and a multimodal information fusion algorithm to perform a comprehensive analysis of multimodal information such as the pre-processed target user data and target environment data to predict the user's current target user intention.

[0109] After determining the target user intention of the user, the multimodal intention recognition module 120 may send the target user intention to the active energy saving control module 130 .

[0110] After receiving the target user's intention, the active energy-saving control module 130 can determine the target smart home device that the user wants to control.

[0111] In addition, on this basis, the active energy-saving control module 130 can also determine a target energy-saving adjustment strategy for the target smart home device according to the target environment data and the target user intention.

[0112] After determining the target energy-saving adjustment strategy, the active energy-saving control module 130 can perform energy-saving control on the target smart home device according to the target energy-saving adjustment strategy. Continuing with the above example, the brightness of the lamp can be adjusted to a brightness that meets the user's needs and does not cause resource waste. The embodiment of the present invention does not limit this.

[0113] In one embodiment of the present invention, the active energy-saving control module 130 is used to shut down at least one smart home device 140 among the multiple smart home devices 140 according to target environment data and a preset energy-saving strategy when no one is present.

[0114] In some feasible embodiments, the active energy-saving control module 130 may also shut down at least one smart home device 140 among the multiple smart home devices 140 according to the target environment data and a pre-set preset energy-saving strategy when the smart home environment is in an unmanned state.

[0115] Specifically, when the smart home environment is in an unmanned state, the active energy-saving control module 130 can first determine at least one smart home device 140 that is not currently running from multiple smart home devices 140 according to the target environment data and the preset energy-saving strategy. The preset energy-saving strategy can be a default strategy for controlling the smart home device 140, or a strategy set by the user; it is mainly aimed at the situation where the smart home environment is in an unmanned state, and under what circumstances, what kind of smart home device 140 is turned off, and the embodiment of the present invention does not limit it.

[0116] Then, the active energy-saving control module 130 may shut down the at least one smart home device 140 that is currently not running unnecessarily to avoid energy waste.

[0117] For example: when the smart home environment is unmanned, the active energy-saving control module 130 detects that the temperature corresponding to the target environment data is 25 degrees Celsius, and the pre-set preset energy-saving strategy is "turn off the air conditioner when the temperature is lower than 26 degrees when no one is there"; then at this time, the active energy-saving control module 130 can turn off the air conditioner, and the embodiments of the present invention are not limited to this.

[0118] For example, when no one is around, the smart home system 10 can use environmental sensors (such as temperature and humidity sensors, light sensors, etc.) to continuously monitor the environmental status and combine time information. For example, when the light intensity reaches a certain value during the day and there is no sign of human activity for a long time (such as several hours) (which can be judged by whether there is sound, whether human movement is detected, etc.), the system will automatically turn off the lighting equipment according to the preset logic to determine that lighting is not needed at this time; similarly, for air-conditioning equipment, if the ambient temperature is within a suitable range and no one is around for a long time, the air-conditioning will also be turned off to save energy.

[0119] The smart home system 10 will record the usage of past devices and the usual status patterns of devices in different time periods and different environmental scenarios. For example, there are usually few people using electrical appliances in a room during the day on weekdays. After the system learns this pattern, it will actively turn off unnecessary appliances in the room during the corresponding time period when it detects that there is no one. In addition, it can also be based on usage habits in different seasons, such as some electric heating equipment automatically turning off in winter when no one is there and the indoor temperature is suitable. These are all based on pre-set scene modes and learned usage habits to perform energy-saving controls. They do not identify real-time user intentions, but operate from the perspective of overall energy saving and regular usage patterns.

[0120] In the embodiment of the present invention, by integrating multimodal information such as voice, vision, gesture, and environment, the smart home system can accurately identify the user's intention and actively perform energy-saving operations; and the user can interact with the smart home system in various ways such as voice, gesture, and expression.

[0121] The smart home system can proactively judge and execute energy-saving operations based on changes in the environment and user behavior.

[0122] The smart home system has self-learning and adaptive functions, and can continuously optimize its intention recognition and energy-saving strategies based on user feedback and usage habits to achieve personalized energy-saving management.

[0123] Based on the above-mentioned smart home system, the embodiment of the present invention also provides a control method of smart home devices based on multi-mode, which can be referred to Figure 3 , shows a step flow chart of a control method for a multi-modal smart home device according to an embodiment of the present invention.

[0124] like Figure 3 As shown, the control method of the multi-modal smart home device may include the following steps:

[0125] Step 301: determine the target user intent of the user; the target user intent is determined based on target user data and target environment data of different modalities.

[0126] In an embodiment of the present invention, a multimodal data acquisition module in a multimodal smart home system can be used to collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0127] After collecting target user data and target environment data in different modalities, the multimodal data acquisition module can send the target user data and target environment data to the multimodal intention recognition module.

[0128] After obtaining the target user data and target environment data of different modalities, the multimodal intention recognition module can use the deep learning model and multimodal information fusion algorithm to conduct a comprehensive analysis of multimodal information such as the target user data and target environment data to predict the user's current target user intention.

[0129] After determining the target user intention of the user, the multimodal intention recognition module can send the target user intention to the active energy saving control module.

[0130] Step 302: According to the target user intention, determine the target smart home device from multiple smart home devices, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention.

[0131] After receiving the target user's intention, the active energy-saving control module can first determine the target smart home device that the user currently wants to control from multiple smart home devices according to the target user's intention.

[0132] In addition, the active energy-saving control module can also obtain target environment data; then, the active energy-saving control module can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0133] For example: if the target user's intention is to increase the brightness of the light, the active energy-saving control module can automatically brighten the light; in addition, when brightening the light, the active energy-saving control module can also adjust the light to a more energy-saving lighting state according to the ambient light conditions; that is, while satisfying the user's target user intention, the target smart home device is controlled to operate in energy-saving mode.

[0134] In some feasible embodiments, the smart home system may further include a multimodal data preprocessing module; the multimodal data preprocessing module may be used to preprocess target user data and target environment data of different modalities.

[0135] In one embodiment of the present invention, the multimodal data preprocessing module can be used to perform noise reduction processing on speech data.

[0136] The multimodal data preprocessing module can also be used to extract features from image data; for example, if the image data is collected based on user gestures, gesture features can be extracted from the image data.

[0137] The multimodal data preprocessing module can also be used to standardize the target user data and target environment data of different modalities collected by the multimodal data collection module to ensure the accuracy and validity of the data.

[0138] Exemplarily, the multimodal data preprocessing module can perform noise reduction on the collected voice and extract the keyword "a bit dark"; perform image processing on the image data captured by the camera and identify the user's dimming gestures, etc. The embodiment of the present invention is not limited to this.

[0139] In some feasible embodiments, the multimodal data collection module may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0140] After collecting target user data and target environment data of different modalities, the multimodal data collection module may send the target user data and target environment data to the multimodal data preprocessing module.

[0141] Then, the multimodal data preprocessing module can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module for intention recognition.

[0142] After obtaining the preprocessed target user data and target environment data of different modalities, the multimodal intention recognition module can use the deep learning model and multimodal information fusion algorithm to conduct a comprehensive analysis of multimodal information such as the preprocessed target user data and target environment data to predict the user's current target user intention.

[0143] After determining the target user intention of the user, the multimodal intention recognition module can send the target user intention to the active energy saving control module.

[0144] After receiving the target user's intention, the active energy-saving control module can first determine the target smart home device that the user currently wants to control from multiple smart home devices according to the target user's intention.

[0145] In addition, the active energy-saving control module can also obtain target environment data; then, the active energy-saving control module can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0146] In some feasible embodiments, the smart home system may further include a user feedback and self-learning module; the user feedback and self-learning module may be used to inquire about the user's satisfaction with the results of the energy-saving control after the active energy-saving control module performs energy-saving control on the target smart home device.

[0147] The user can provide feedback on the inquiry through voice or interface; at this time, the user feedback and self-learning module can receive the user feedback information. Then, the user feedback and self-learning module can optimize the multimodal intent recognition module according to the user feedback information.

[0148] Specifically, the user feedback and self-learning module can record user preferences based on user feedback information and use machine learning algorithms to continuously optimize intent recognition and control strategies.

[0149] Exemplarily, the multimodal data preprocessing module can perform noise reduction on the collected voice and extract the keyword "a bit dark"; perform image processing on the image data captured by the camera and identify the user's dimming gestures, etc. The embodiment of the present invention is not limited to this.

[0150] In some feasible embodiments, the multimodal data collection module may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0151] After collecting target user data and target environment data of different modalities, the multimodal data collection module may send the target user data and target environment data to the multimodal data preprocessing module.

[0152] Then, the multimodal data preprocessing module can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module for intention recognition.

[0153] After obtaining the preprocessed target user data and target environment data of different modalities, the multimodal intention recognition module can use the deep learning model and multimodal information fusion algorithm to conduct a comprehensive analysis of multimodal information such as the preprocessed target user data and target environment data to predict the user's current target user intention.

[0154] After determining the target user intention of the user, the multimodal intention recognition module can send the target user intention to the active energy saving control module.

[0155] After receiving the target user's intention, the active energy-saving control module can first determine the target smart home device that the user currently wants to control from multiple smart home devices according to the target user's intention.

[0156] In addition, the active energy-saving control module can also obtain target environment data; then, the active energy-saving control module can integrate the target environment data and the target user intention to perform energy-saving control on the target smart home device.

[0157] After the active energy-saving control module performs energy-saving control on the target smart home device, the user feedback and self-learning module can ask the user about his / her satisfaction with the result of the energy-saving control.

[0158] The user can provide feedback on the inquiry through voice or interface; at this time, the user feedback and self-learning module can receive the user feedback information. Then, the user feedback and self-learning module can optimize the multimodal intent recognition module according to the user feedback information.

[0159] In some feasible embodiments, after receiving the target user's intention, the active energy-saving control module can determine the target smart home device that the user wants to control.

[0160] In addition, on this basis, the active energy-saving control module can also determine the target energy-saving adjustment strategy for the target smart home device according to the target environment data and the target user intention.

[0161] For example: the target smart home device is a lamp, and the user expects the control strategy for the target smart home device to be to increase the brightness; then the active energy-saving control module can, based on this, determine the specific adjustment brightness (i.e., the target energy-saving adjustment strategy) according to the target environment data (here is the light intensity data); the specific adjustment brightness can meet the user's needs while avoiding the problem of resource waste due to excessive brightness.

[0162] After determining the target energy-saving adjustment strategy, the active energy-saving control module can perform energy-saving control on the target smart home device according to the target energy-saving adjustment strategy. Continuing with the above example, the brightness of the lamp can be adjusted to a brightness that meets the user's needs and does not cause resource waste. The embodiment of the present invention does not limit this.

[0163] Exemplarily, the multimodal data collection module may first collect target user data and target environment data in different modalities so as to predict the user's intention from different dimensions.

[0164] After collecting target user data and target environment data of different modalities, the multimodal data collection module may send the target user data and target environment data to the multimodal data preprocessing module.

[0165] Then, the multimodal data preprocessing module can preprocess the target user data and target environment data of different modalities, and send the preprocessed target user data and target environment data of different modalities to the multimodal intention recognition module for intention recognition.

[0166] After obtaining the preprocessed target user data and target environment data of different modalities, the multimodal intention recognition module can use the deep learning model and multimodal information fusion algorithm to conduct a comprehensive analysis of multimodal information such as the preprocessed target user data and target environment data to predict the user's current target user intention.

[0167] After determining the target user intention of the user, the multimodal intention recognition module can send the target user intention to the active energy saving control module.

[0168] After receiving the target user's intention, the active energy-saving control module can determine the target smart home device that the user wants to control.

[0169] In addition, on this basis, the active energy-saving control module can also determine the target energy-saving adjustment strategy for the target smart home device according to the target environment data and the target user intention.

[0170] After determining the target energy-saving adjustment strategy, the active energy-saving control module can perform energy-saving control on the target smart home device according to the target energy-saving adjustment strategy. Continuing with the above example, the brightness of the lamp can be adjusted to a brightness that meets the user's needs and does not cause resource waste. The embodiment of the present invention does not limit this.

[0171] In some feasible embodiments, the active energy-saving control module can also shut down at least one smart home device among multiple smart home devices according to target environment data and a pre-set preset energy-saving strategy when the smart home environment is unmanned.

[0172] Specifically, when the smart home environment is in an unmanned state, the active energy-saving control module can first determine at least one smart home device that is not currently running from multiple smart home devices based on the target environment data and a preset energy-saving strategy. The preset energy-saving strategy can be a default strategy for controlling smart home devices, or a strategy set by the user. It is mainly aimed at the situation where the smart home environment is in an unmanned state. Under what circumstances, what kind of smart home device is turned off, and the embodiment of the present invention does not limit it.

[0173] Then, the active energy-saving control module can shut down the at least one smart home device that is currently unnecessarily running to avoid energy waste.

[0174] In an embodiment of the present invention, the target user intention of the user is determined; the target user intention is determined based on target user data and target environment data of different modalities; based on the target user intention, the target smart home device is determined from multiple smart home devices, and the target smart home device is energy-saving controlled in combination with the target environment data and the target user intention. Through the embodiment of the present invention, the user's intention can be predicted by combining user data and environment data of different modalities, and the smart home device can be energy-saving controlled based on the user's intention. Through the embodiment of the present invention, problems such as energy waste caused by the mismatch between device operation and user needs can be avoided.

[0175] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0176] Reference Figure 4 , shows a schematic structural diagram of a control device for a multi-modal smart home device according to an embodiment of the present invention, which is applied to the smart home system mentioned in the above embodiment.

[0177] like Figure 4 As shown, the control device based on multi-modal smart home equipment may include the following modules:

[0178] The determination module 401 is used to determine the target user intention of the user; the target user intention is determined based on target user data and target environment data of different modalities;

[0179] The control module 402 is used to determine a target smart home device from multiple smart home devices according to the target user's intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user's intention.

[0180] In an embodiment of the present invention, the target user intention of the user is determined; the target user intention is determined based on target user data and target environment data of different modalities; based on the target user intention, the target smart home device is determined from multiple smart home devices, and the target smart home device is energy-saving controlled in combination with the target environment data and the target user intention. Through the embodiment of the present invention, the user's intention can be predicted by combining user data and environment data of different modalities, and the smart home device can be energy-saving controlled based on the user's intention. Through the embodiment of the present invention, problems such as energy waste caused by the mismatch between device operation and user needs can be avoided.

[0181] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above-mentioned multi-modal-based smart home device control method is implemented.

[0182] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the control method of a multi-modal smart home device as described above is implemented.

[0183] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0185] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0186] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0190] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0191] The above provides a detailed introduction to a control method for a smart home device based on multimodality, a control device for a smart home device based on multimodality, an electronic device and a computer-readable storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-modal smart home system, characterized in that: The smart home system includes: a multimodal data acquisition module, a multimodal intention recognition module, an active energy-saving control module, and a plurality of smart home devices; wherein: The multimodal data acquisition module is used to collect target user data and target environment data in different modalities; The multimodal intention recognition module is used to determine the target user intention based on the target user data and the target environment data in different modalities; The active energy-saving control module is used to determine a target smart home device from multiple smart home devices according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention.

2. The smart home system according to claim 1, characterized in that: The smart home system also includes: The multimodal data preprocessing module is used to preprocess the target user data and the target environment data of different modalities, and send the preprocessed target user data and the target environment data of different modalities to the multimodal intention recognition module for intention recognition.

3. The smart home system according to claim 2, characterized in that: The pre-processing includes at least one of the following processing methods: Noise reduction, feature extraction, and data normalization.

4. The smart home system according to claim 1, characterized in that: The smart home system also includes: The user feedback and self-learning module is used to receive user feedback information after the active energy-saving control module performs energy-saving control on the target smart home device; and optimize the multimodal intention recognition module according to the user feedback information.

5. The smart home system according to claim 1, characterized in that: The active energy-saving control module is used to determine a target energy-saving adjustment strategy based on the target environment data; determine a target energy-saving adjustment strategy based on the target environment data and the target user's intention; and perform energy-saving control on the target smart home device based on the target energy-saving adjustment strategy.

6. The smart home system according to claim 1, characterized in that: The active energy-saving control module is used to shut down at least one smart home device among the multiple smart home devices according to the target environment data and a preset energy-saving strategy when no one is present.

7. The smart home system according to claim 1, characterized in that: The target user data includes at least one of the following data: Image data, voice data.

8. The smart home system according to claim 1, characterized in that: The target environment data includes at least one of the following data: Temperature data, humidity data, light intensity data.

9. A control method for a multi-modal smart home device, characterized in that: Applied in a smart home system as claimed in any one of claims 1 to 8; the method comprises: Determine the target user intent of the user; the target user intent is determined based on target user data and target environment data of different modalities; According to the target user intention, a target smart home device is determined from a plurality of smart home devices, and energy-saving control is performed on the target smart home device in combination with the target environment data and the target user intention.

10. A control device for a multi-modal smart home device, characterized in that: Applicable to the smart home system according to any one of claims 1 to 8; the device comprises: A determination module, used to determine the target user intention of the user; the target user intention is determined based on target user data and target environment data of different modalities; The control module is used to determine a target smart home device from a plurality of smart home devices according to the target user intention, and perform energy-saving control on the target smart home device in combination with the target environment data and the target user intention.

11. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the control method of the multi-modal smart home device as claimed in claim 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the control method of the multi-modal smart home device as claimed in claim 9 is implemented.

Images