Smart home control method and system, and computer readable storage medium
By using infrared thermal sensing modules and convolutional neural networks to identify user emotions and behaviors, and dynamically control smart home devices, this solves the problem that existing devices cannot deeply perceive the user's state, enabling personalized feedback and security monitoring, while reducing hardware costs.
Patent Information
- Application Number
- CN202511801219.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-02-13
AI Technical Summary
Existing smart home devices lack the ability to deeply perceive users' psychological state and behavioral characteristics, making it impossible to achieve human-computer emotional interaction and personalized feedback, and the hardware cost is relatively high.
An infrared thermal sensing module is used to collect the user's thermal image data. Multimodal feature extraction and temporal modeling are performed through a convolutional neural network to identify the user's emotional and behavioral state, and dynamically generate lighting and fan control signals to achieve two-way feedback between emotions and the environment.
It enables more humanized, emotional, and secure control of smart home devices, reduces hardware costs, and caters to the needs of users of different ages.
Smart Images
Figure CN121523079A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent control technology, specifically to a control method and system for smart homes, and a computer-readable storage medium. Background Technology
[0002] With the development of artificial intelligence technology, smart devices in smart homes, such as smart lights and smart fans, are evolving towards multi-functionality and intelligence. For example, existing smart lights use voice control, light sensing, or infrared human body detection modules to achieve basic on / off control or brightness adjustment, and some products can automatically adjust based on ambient temperature and humidity.
[0003] However, existing smart devices can only sense external physical parameters and lack the ability to deeply perceive the user's psychological state or behavioral characteristics, thus failing to achieve human-computer emotional interaction and personalized feedback. Most existing infrared sensing systems are only used to detect the presence or movement trajectory of a human body, unable to distinguish different postures, movements, and behavioral characteristics, let alone recognize changes in the user's emotions. Even some smart fans and lights that support temperature sensing or timer control remain at the stage of simple trigger logic, making it difficult to dynamically match wind speed and light according to the user's state.
[0004] Furthermore, current lighting control systems lack the ability to learn from long-term user data and cannot adaptively optimize based on user habits, age, or health conditions. This results in a disconnect between the lighting experience and human needs, making it difficult to achieve true intelligent response and safety assurance in scenarios such as age-friendly care and emotional relaxation.
[0005] An existing multimodal fusion-based intelligent sensor switch control method includes: real-time acquisition of multimodal sensing data; feature extraction and fusion processing of the multimodal sensing data to obtain a temporal representation vector sequence of user activity patterns; semantic parsing of the temporal representation vector sequence of user activity patterns to accurately identify the user's behavioral and emotional states; dynamic generation of a lighting parameter configuration scheme based on a pre-built intelligent response decision engine, combining behavioral states, emotional states, and multimodal sensing data; and conversion of the lighting parameter configuration scheme into a lighting control command sequence, which is then sent to the lighting control device via a KNX bus. However, this method analyzes user activity patterns through multiple sensors to determine emotional or behavioral states, resulting in increased hardware costs due to the use of numerous sensors. Summary of the Invention
[0006] The primary objective of this invention is to provide a smart home control method using only one infrared thermal sensing module.
[0007] A second objective of this invention is to provide a control system for a smart home that implements the aforementioned smart home control method.
[0008] A third objective of this invention is to provide a computer-readable storage medium for implementing the above-described smart home control method.
[0009] To achieve the first objective of this invention, the present invention provides a smart home control method, which includes: acquiring a set of user thermal image data collected by an infrared thermal sensing module; inputting the thermal image data set into a multimodal feature extraction module, and outputting emotion classification information and behavior classification information; the multimodal feature extraction module includes a convolutional neural network, which includes a backbone network connected to a feature pyramid network, which is connected to a temporal modeling module, which is connected to an emotion classification module, and the temporal modeling module is also connected to a behavior classification module; outputting temperature change information of the facial region and user contour change information, the emotion classification module outputting emotion classification information based on the temperature change information of the facial region, and the behavior classification module outputting behavior classification information based on the user contour change information; querying smart home control information based on the emotion classification information and behavior classification information, and applying the smart home control information.
[0010] As can be seen from the above scheme, the heat map data set consists of multiple heat map data points containing time series data. The convolutional neural network identifies facial temperature information through the heat map data, and then identifies changes in emotion based on this facial temperature information. For example, if the forehead or neck shows large temperature fluctuations and uneven heat distribution on the face, the output is anxiety. By using only one infrared thermal sensing module to acquire emotion and behavior classification information, intelligent devices can be controlled, achieving two-way feedback between emotion and environment, thus realizing humanized, emotional, and secure control of smart homes. Because this invention uses only one infrared thermal sensing module, the hardware cost is relatively low.
[0011] In a further proposed solution, the steps for obtaining the user's thermal image data set collected by the infrared thermal sensing module include: acquiring the temperature matrix data of the infrared thermal sensing module; performing temperature compensation on the temperature matrix data based on the ambient temperature data to obtain temperature-compensated data; upsampling the temperature-compensated data using a bilinear interpolation algorithm to obtain interpolated temperature matrix data; normalizing the interpolated temperature matrix data to obtain normalized thermal image data; and stacking multiple time-series consecutive normalized thermal image data sets to obtain a thermal image data set.
[0012] Therefore, after acquiring the temperature matrix data from the infrared thermal sensing module, it is transformed into a thermal image data set through interpolation and normalization, thereby achieving high-precision thermal image data acquisition.
[0013] In a further proposed solution, smart home control information includes dimming information and fan movement control information.
[0014] In a further proposed solution, the steps for querying smart home control information based on emotion classification information and behavior classification information include: obtaining dimming mapping rules from a dimming table based on emotion classification information and behavior classification information, and generating dimming signals based on the dimming mapping rules; the steps for querying smart home control information based on emotion classification information and behavior classification information also include: obtaining fan mapping rules from a fan table based on emotion classification information and behavior classification information, and generating fan control signals based on the fan mapping rules.
[0015] This demonstrates that intelligent lighting and intelligent fans can achieve human-computer emotional interaction, personalized comfort adjustment, and safety warning functions through dynamic responses.
[0016] In a further proposed solution, after applying smart home control information, the following steps are performed: recording feature vector information, emotion classification information, behavior classification information, smart home control information, and user operation information of the heatmap data set to form user data; inputting the user data into a convolutional neural network for training to obtain a trained convolutional neural network; and acquiring the latest heatmap data set and processing the latest heatmap data set using the trained convolutional neural network.
[0017] This shows that dynamically updating the convolutional neural network is more in line with user habits.
[0018] In a further proposed solution, user profile data is obtained to determine whether the user is an elderly person. If so, the maximum brightness threshold of the dimming meter is reduced, the trigger threshold for falls in the behavior classification module is increased, and the determination time for stationary behavior in the behavior classification module is extended.
[0019] Therefore, this invention has aging safety monitoring capabilities and is suitable for users of all ages.
[0020] In a further proposed approach, a feature pyramid network is used to capture facial temperature information and user contour information.
[0021] This demonstrates that by capturing facial temperature and user contour information through a feature pyramid network, the temporal modeling module can more easily identify emotional changes and action information.
[0022] To achieve the second objective, the present invention provides a smart home control system comprising a processor, a memory, and a smart device. The processor includes an infrared thermal sensing module, a multimodal feature extraction module, a dimming module, a fan module, and an adjustment module. The infrared thermal sensing module is connected to the multimodal feature extraction module, the dimming module is connected to the multimodal feature extraction module, the fan module is connected to the multimodal feature extraction module, and the adjustment module is connected to the multimodal feature extraction module. The smart device is connected to the processor. The memory stores a computer program, and when the computer program is executed, it implements the aforementioned smart home control method.
[0023] To achieve the third objective, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the aforementioned smart home control method. Attached Figure Description
[0024] Figure 1 This is a system structure block diagram of an embodiment of the smart home control system of the present invention.
[0025] Figure 2 This is a flowchart of an embodiment of the smart home control method of the present invention.
[0026] Figure 3 This is a structural diagram of a convolutional neural network in an embodiment of the smart home control method of the present invention.
[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. Detailed Implementation
[0028] The smart home control method provided by this invention outputs emotion change information based on temperature change information and behavior change information based on behavior contour information through the temporal modeling module of a convolutional neural network. Based on emotion classification information and behavior classification information, it queries smart home control information to control the smart home, thereby realizing two-way feedback between emotion and environment and achieving humanized, emotional and safe control of the smart home.
[0029] Example of a smart home control system: The smart home control system of this embodiment includes a processor 1, a memory 2, and a smart device 3. The processor 1 includes an infrared thermal sensing module 11, a multimodal feature extraction module 12, a dimming module 13, a fan module 14, and an adjustment module 15. The infrared thermal sensing module 11 is connected to the multimodal feature extraction module 12, the dimming module 13 is connected to the multimodal feature extraction module 12, the fan module 14 is connected to the multimodal feature extraction module 12, the adjustment module 15 is connected to the multimodal feature extraction module 12, and the smart device 3 is connected to the processor 1.
[0030] The infrared thermal sensing module 11 collects user thermal image data. The multimodal feature extraction module 12, equipped with a convolutional neural network, inputs the thermal image data collected by the infrared thermal sensing module 11 into the convolutional neural network, outputting emotion classification information and behavior classification information. The dimming module 13 queries dimming mapping rules based on the emotion and behavior classification information to generate a dimming signal. The fan module 14 queries fan mapping rules based on the emotion and behavior classification information to generate a fan control signal. The adjustment module 15 records user data, inputs the user data into the convolutional neural network for training, and updates the convolutional neural network.
[0031] The smart device 3 includes a smart lamp and a smart fan. The smart lamp dims the light according to the dimming signal generated by the dimming module 13, and the smart fan operates according to the fan control signal output by the fan module 14.
[0032] Examples of smart home control methods: The smart home control method in this embodiment is implemented by the smart home control system described in the above embodiment. See also... Figure 2 First, step S1 is executed to acquire the user's thermal image data set collected by the infrared thermal sensing module. The infrared thermal sensing module includes an infrared thermal imaging sensor, which can be the MLX90640 model. The infrared thermal imaging sensor acquires the temperature matrix data from the infrared thermal sensing module at a preset refresh rate, which can be 8Hz to 64Hz. The temperature matrix data is a 32×24 two-dimensional temperature matrix, where the value of each pixel is the absolute temperature of that area. After acquiring the temperature matrix data from the infrared thermal sensing module, temperature compensation is performed on the temperature matrix data based on the ambient temperature data to obtain temperature-compensated data, ensuring measurement accuracy.
[0033] Then, the temperature compensation data with a resolution of 32×24 is upsampled using the bilinear interpolation algorithm to obtain the interpolated temperature matrix data with a resolution of 128×96. The resolution of the interpolated temperature matrix data can be higher than 128×96, which improves the accuracy of subsequent feature extraction.
[0034] Then, the interpolated temperature matrix data is normalized, mapping the temperature values to the interval [0,1]. The formula for normalizing the interpolated temperature matrix data is Pixel_norm = (Pixel_temp - T_min) / (T_max - T_min), where Pixel_norm is the normalized result, Pixel_temp is the original temperature value in the interpolated temperature matrix data, T_min is the minimum value of a preset fixed human body temperature range, such as 28°C, and T_max is the maximum value of a preset fixed human body temperature range, such as 40°C. T_min can also be the minimum value of the current interpolated temperature matrix data, and T_max can also be the maximum value of the current interpolated temperature matrix data. After normalizing the interpolated temperature matrix data, normalized heatmap data is obtained.
[0035] After obtaining the normalized heatmap data, multiple temporally consecutive normalized heatmap data are stacked to obtain a heatmap data group. These multiple normalized heatmap data are composed of multiple consecutive frames of normalized heatmap data. These multiple normalized heatmap data are then stacked to construct a heatmap data group of shape (N, Height, Width, 1), where N is the arrangement of the heatmap data group, Height is the height of the heatmap data, Width is the width of the heatmap data, and 1 is the number of heatmap data groups. This step can incorporate time-series information.
[0036] After acquiring the user's heat map data set collected by the infrared thermal sensing module, step S2 is executed to input the heat map data set into the multimodal feature extraction module, which outputs emotion classification information and behavior classification information.
[0037] See Figure 3 The multimodal feature extraction module includes a convolutional neural network (CNN), which comprises a backbone network connected to a feature pyramid network. The feature pyramid network is connected to a temporal modeling module, which in turn is connected to an emotion classification module and a behavior classification module. The backbone network is either a MobileNetV2 neural network or an EfficientNet-Lite neural network with the original classification head removed. The backbone network outputs one-quarter, one-eighth, and one-sixteenth feature maps. One-quarter of the feature maps are output to the first feature pyramid network, one-eighth to the second, and one-sixteenth to the third. The feature pyramid network (FPN module) is used to capture different feature maps output by the backbone network, as well as facial and user contour information. The feature pyramid network outputs a feature sequence. The one-sixteenth feature map is a low-resolution feature map. Facial information includes forehead, neck, and cheek region information.
[0038] After processing one-sixteenth of the feature map, the third feature pyramid network upsamples the output feature map to improve its spatial resolution, making the resolution of the upsampled feature map consistent with that of the feature map output by the second feature pyramid network.
[0039] The 1×1 convolutional layer fuses the feature map output from the second feature pyramid network with the upsampled feature map through channel processing, outputting a first feature map, a second feature map, and a third feature map. The first feature map is a high-resolution feature map, the second feature map is a medium-resolution feature map, and the third feature map is a low-resolution feature map.
[0040] Global average pooling is performed on the first, second, and third feature maps. These three feature maps are then concatenated to integrate their spatial information, resulting in a fused spatial feature. The concatenated feature maps are organized into a spatial feature vector sequence (N, D), where N represents the permutation of the heatmap data set, and D is the feature dimension.
[0041] The temporal modeling module is either a bidirectional single-layer Bi-LSTM model or a one-dimensional temporal convolutional network (1D-TCN). This module learns the output spatial feature sequence, injects temporal information, and outputs temperature change information for the facial region and user contour change information. The temperature change information represents the temperature changes of each facial region over time, such as the temperature changes in the forehead region. The user contour change information represents the changes in the user's contour over time.
[0042] The emotion classification module is a fully connected layer with a normalized exponential function (Softmax activation function), outputting the probability distribution of five emotion categories: relaxation, anxiety, pleasure, drowsiness, and neutrality. The behavior classification module is also a fully connected layer with a normalized exponential function (Softmax activation function), outputting the probability distribution of seven core behaviors: sitting, standing, walking, lying down, falling, waving, and hugging. The fully connected layer can weight and combine information based on the temperature changes of each facial region over time, scoring the temperature changes of each facial region and outputting the scores. The normalized exponential function outputs the probability distribution information based on the scores. For example, the behavior classification module sets a trigger threshold for falling or a judgment time for stationary behavior. When a falling behavior is detected, if the trigger threshold is met, it is judged as a fall; when stationary behavior is detected, if the judgment time for stationary behavior is met, it is judged as stationary.
[0043] Among them, the emotion classification module outputs the highest probability distribution value, which represents the emotion classification information, while the behavior classification module outputs the highest probability distribution value, which represents the behavior classification information.
[0044] To prevent result jitter, multiple heatmap data sets are continuously input into the convolutional neural network, which outputs multiple emotion classification information and multiple behavior classification information. The emotion classification information and the behavior classification information with the most outputs are used as the final emotion classification information and behavior classification information.
[0045] After outputting the emotion classification information and behavior classification information, step S3 is executed to query the smart home control information based on the emotion classification information and behavior classification information. The smart home control information includes dimming information and fan action control information.
[0046] The dimming module retrieves dimming mapping rules from the dimming table based on emotion and behavior classification information, and generates a dimming signal according to these rules. The fan module retrieves fan mapping rules from the fan table based on emotion and behavior classification information, and generates a fan control signal according to these rules.
[0047] The dimming mapping rule includes animation information and dimming parameter information. If the emotion and behavior classification information is anxiety and sitting, the dimming mapping rule can be a color temperature of 6500K, a blue hue, a brightness of 70%, and a slow breathing animation. If the emotion and behavior classification information is drowsiness and lying down, the dimming mapping rule can be a color temperature of 2700K, a warm white hue, a brightness of 30%, and no animation. If the emotion and behavior classification information is anxiety and falling, the dimming mapping rule can be a color temperature of 10000K, a blue hue, a brightness of 100%, and an emergency flashing animation (5Hz).
[0048] When generating a dimming signal, the animation script stored in the memory can be called to perform dimming. The refresh rate of each frame can be precisely controlled through the processor's hardware timer.
[0049] If the smart light fixture is an RGB LED array, the dimming signal can be sent via the SPI protocol, or the output can be simulated via PWM signal to control the smart light fixture.
[0050] The processor controls the fan module according to PWM or I2C protocols to achieve stepless speed regulation of the fan equipped with a DC brushless motor. If the emotion and behavior classification information is anxiety and sitting, the fan mapping rule can be high speed. If the emotion and behavior classification information is drowsy and lying down, the fan mapping rule can be low speed. If the emotion and behavior classification information is anxiety and falling, the fan speed is turned off.
[0051] Furthermore, when the output behavior classification information is "falling down", the processor enters the interrupt service routine, stops the fan from rotating, cuts off the power to the fan module, sets the dimming signal to an emergency flashing animation script, and sends an alarm message to the preset home cloud platform or mobile phone via the wireless module.
[0052] After applying smart home control information, feature vector information, emotion classification information, behavior classification information, smart home control information, and user operation information of the heatmap data set are recorded to form user data. This user data is then input into a convolutional neural network for training, resulting in a trained convolutional neural network. User operation information allows users to frequently manually adjust the effects of the system's automatic actions. For example, the system might determine "pleasure" and activate a colorful animation, but the user might repeatedly switch it back to warm white light.
[0053] The trained convolutional neural network is used to execute the next smart home control method. When the latest heat map data set is acquired, the trained convolutional neural network can be used to process the latest heat map data set.
[0054] The system retrieves user profile data to determine if a user is elderly. If so, it reduces the maximum brightness threshold of the dimming meter, increases the fall trigger threshold in the behavior classification module, and extends the judgment time for stationary behavior in the behavior classification module, thereby providing protection for the elderly.
[0055] Convolutional neural networks identify facial temperature information through thermal image data, and then use this temperature information to identify changes in emotion. For example, if the forehead or neck shows large temperature fluctuations and uneven heat distribution on the face, the output is anxiety. By using only one infrared thermal sensing module to acquire emotion and behavior classification information, this allows for the control of smart devices, achieving two-way feedback between emotion and environment, thus enabling humanized, emotional, and secure control of smart homes. Because this invention uses only one infrared thermal sensing module, its hardware cost is relatively low.
[0056] Examples of computer-readable storage media: The smart home control method in the smart home control system described in the above embodiments can be stored as a computer program in a computer-readable storage medium. When the computer program is executed by a processor, it can complete the steps of the smart home control method embodiments described above. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0057] The above are merely preferred embodiments of the present invention, but the design concept of the invention is not limited thereto. Without departing from the concept of the present invention, many other equivalent embodiments may be included. Those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the protection scope of the present invention.
Claims
1. A control method for smart homes, the method including: Acquire user thermal image data sets collected by the infrared thermal sensing module; Its features are: The heatmap data set is input into the multimodal feature extraction module, which outputs emotion classification information and behavior classification information. The multimodal feature extraction module includes a convolutional neural network, which includes a backbone network. The backbone network is connected to a feature pyramid network, which is connected to a temporal modeling module. The temporal modeling module is connected to an emotion classification module and also to a behavior classification module. The temporal modeling module outputs temperature change information and user contour change information for the facial region; the emotion classification module outputs emotion classification information based on the temperature change information for the facial region; and the behavior classification module outputs behavior classification information based on the user contour change information. Based on the emotion classification information and the behavior classification information, query the smart home control information and apply the smart home control information.
2. The smart home control method according to claim 1, characterized in that: The steps to obtain user thermal image data sets collected by the infrared thermal sensing module include: The temperature matrix data of the infrared thermal sensing module is obtained, and temperature compensation is performed on the temperature matrix data based on the ambient temperature data to obtain temperature compensation data. The temperature compensation data is upsampled using a bilinear interpolation algorithm to obtain interpolated temperature matrix data. The interpolated temperature matrix data is normalized to obtain normalized heat map data; Multiple normalized heatmap data that are consecutive in time are stacked to obtain a heatmap data set.
3. The smart home control method according to claim 1, characterized in that: The smart home control information includes dimming information and fan operation control information.
4. The smart home control method according to claim 3, characterized in that: The steps for querying smart home control information based on the emotion classification information and the behavior classification information include: Based on the emotion classification information and the behavior classification information, obtain the dimming mapping rules from the dimming table, and generate a dimming signal based on the dimming mapping rules; The step of querying smart home control information based on the emotion classification information and the behavior classification information further includes: Based on the emotion classification information and the behavior classification information, fan mapping rules are obtained from the fan table, and fan control signals are generated based on the fan mapping rules.
5. The smart home control method according to any one of claims 1 to 4, characterized in that: After applying the aforementioned smart home control information, the following also occurs: The feature vector information of the heatmap data group, the emotion classification information, the behavior classification information, the smart home control information, and the user operation information are recorded to form user data; The user data is input into the convolutional neural network for training to obtain the trained convolutional neural network; Obtain the latest heatmap data set and process it using the trained convolutional neural network.
6. The smart home control method according to claim 4, characterized in that: After applying the aforementioned smart home control information, the following also occurs: Obtain user profile data to determine if the user is an elderly person; If so, reduce the maximum brightness threshold of the light in the dimming meter, increase the trigger threshold for falling in the behavior classification module, and extend the determination time for stationary behavior in the behavior classification module.
7. The smart home control method according to any one of claims 1 to 4, characterized in that: The feature pyramid network is used to capture facial information and user contour information.
8. The smart home control method according to claim 4, characterized in that: The dimming mapping rules include animation information and dimming parameter information.
9. A control system for a smart home, comprising a processor, a memory, and smart devices, characterized in that, The processor includes an infrared thermal sensing module, a multimodal feature extraction module, a dimming module, a fan module, and an adjustment module. The infrared thermal sensing module is connected to the multimodal feature extraction module, the dimming module is connected to the multimodal feature extraction module, the fan module is connected to the multimodal feature extraction module, and the adjustment module is connected to the multimodal feature extraction module. The smart device is connected to the processor. The memory stores a computer program. When the computer program is executed, it implements the smart home control method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed, implements the smart home control method according to any one of claims 1 to 8.