Smart home control method and apparatus, electronic device, and computer readable medium

Abstract

The application relates to an intelligent home control method and device, electronic equipment and a computer readable medium. The method comprises the following steps: collecting multi-modal data of an intelligent home, wherein the multi-modal data comprises physical device data, environment data and user data; performing data fusion on the physical device data, the environment data and the user data in the multi-modal data to obtain a data fusion result; constructing a virtual model, wherein the virtual model comprises a physical model, an environment model and a user model, and a dynamic feedback mechanism is established among the physical model, the environment model and the user model; training the physical model, the environment model and the user model in the virtual model according to the data fusion result to obtain a digital twin model; and controlling the intelligent home based on the digital twin model and using a self-supervised learning algorithm according to real-time collected multi-modal data of the intelligent home. The application can improve the adaptability of the intelligent home device.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and in particular to smart home control methods, devices, electronic devices and computer-readable media. Background Technology

[0002] Smart home technology is widely used in home control, environmental sensing, and device linkage, enabling remote control and automated management of home devices. In recent years, through the integration of artificial intelligence and the Internet of Things, smart homes have entered the intelligent era.

[0003] However, existing smart home systems mainly rely on simple communication between sensors and devices. Most personalized functions predict user behavior based on pre-set rules and parameters to control and respond to devices. This lack of deep understanding of user behavior and inability to dynamically adapt to changes in user needs results in low adaptability of smart home devices. Summary of the Invention

[0004] This application provides smart home control methods, devices, electronic devices, and computer-readable media to address the technical problems in the prior art, such as the lack of deep understanding of user behavior and the inability to dynamically adapt to changes in user needs.

[0005] According to one aspect of an embodiment of this application, this application provides a smart home control method, the method comprising: collecting multimodal data of a smart home, the multimodal data including physical device data, environmental data, and user data; performing data fusion on the physical device data, environmental data, and user data in the multimodal data to obtain a data fusion result; constructing a virtual model, the virtual model including a physical model, an environmental model, and a user model, establishing a dynamic feedback mechanism among the physical model, the environmental model, and the user model; training the physical model, the environmental model, and the user model in the virtual model according to the data fusion result to obtain a digital twin model; and controlling the smart home based on the digital twin model using a self-supervised learning algorithm and according to the real-time collected smart home multimodal data.

[0006] Optionally, the construction of the virtual model includes: constructing a three-dimensional model of the physical device based on its physical characteristics to obtain a physical model; constructing a three-dimensional environmental model based on the environmental characteristics of the physical device to obtain an environmental model; constructing a three-dimensional user model based on user characteristics and user behavior characteristics to obtain a user model; and establishing a dynamic feedback mechanism among the physical model, the environmental model, and the user model to obtain the virtual model.

[0007] Optionally, the step of constructing a three-dimensional user model based on user characteristics and user behavior characteristics to obtain the user model includes: establishing a user profile model based on user characteristics; establishing a user behavior model of the interaction between the physical device and the user based on user behavior characteristics; and constructing a three-dimensional user model based on the user profile model and the user behavior model to obtain the user model.

[0008] Optionally, the step of fusing the physical device data, environmental data, and user data in the multimodal data to obtain a data fusion result includes: performing a first data fusion of the user data and the environmental data to obtain a first data fusion result; performing a second data fusion of the user data and the physical device data to obtain a second data fusion result; and performing a third data fusion of the physical device data and the environmental data to obtain a second data fusion result.

[0009] Optionally, training the physical model, environment model, and user model in the virtual model based on the data fusion result to obtain a digital twin model includes: inputting the first data fusion result into the user model and the environment model for training to enable the user model to interact with the environment model through virtual simulation; inputting the second data fusion result into the user model and the physical model for training to enable the user model to interact with the physical model through virtual simulation; inputting the third data fusion result into the physical model and the environment model for training to enable the physical model to interact with the environment model through virtual simulation; and integrating the trained user model, physical model, and environment model to obtain the digital twin model.

[0010] Optionally, after controlling the smart home based on the real-time collected multimodal data of the smart home using a self-supervised learning algorithm based on the digital twin model, the method further includes: performing speech recognition through a user interaction layer and dynamically adjusting scene settings in conjunction with the digital twin model; and performing model self-optimization through the visual feedback of the digital twin model.

[0011] Optionally, after training the physical model, environment model, and user model in the virtual model based on the data fusion result to obtain a digital twin model, the method further includes: deploying the digital twin model on a local server; and encrypting and protecting the multimodal data in the digital twin model using local edge computing technology and privacy computing technology.

[0012] According to another aspect of the embodiments of this application, this application provides a smart home control device, the device comprising: a data acquisition module for acquiring multimodal data of a smart home, the multimodal data including physical device data, environmental data, and user data; a data fusion module for fusing the physical device data, environmental data, and user data in the multimodal data to obtain a data fusion result; a construction module for constructing a virtual model, the virtual model including a physical model, an environmental model, and a user model, wherein a dynamic feedback mechanism is established between the physical model, the environmental model, and the user model; a training module for training the physical model, the environmental model, and the user model in the virtual model according to the data fusion result to obtain a digital twin model; and a control module for controlling the smart home based on the digital twin model using a self-supervised learning algorithm and according to the real-time acquired smart home multimodal data.

[0013] According to another aspect of the embodiments of this application, this application provides an electronic device, including a memory, a processor, a communication interface, and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of the smart home control method described above.

[0014] According to another aspect of the embodiments of this application, this application provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the steps of the smart home control method.

[0015] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0016] This application can be applied to smart home control scenarios. It collects multimodal data from smart homes, including physical device data, environmental data, and user data; fuses the physical device data, environmental data, and user data from the multimodal data to obtain a data fusion result; constructs a virtual model, including a physical model, an environmental model, and a user model, establishing a dynamic feedback mechanism among the physical model, environmental model, and user model; trains the physical model, environmental model, and user model in the virtual model based on the data fusion result to obtain a digital twin model; and uses a self-supervised learning algorithm based on the digital twin model to control the smart home system according to the real-time collected multimodal data. This allows for the full utilization of multimodal data to construct a digital twin model and, with the help of a self-supervised learning algorithm, achieves more intelligent, precise, and adaptive control of the smart home system, enabling smart homes to better serve people's lives and create a more comfortable, convenient, and safe home environment. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an optional smart home control method provided according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of another optional smart home control method provided according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of another optional smart home control method provided according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of another optional smart home control method provided according to an embodiment of this application;

[0023] Figure 5 This is a structural diagram of an optional smart home control device according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To address the problems mentioned in the background art, an embodiment of a smart home control method is provided according to one aspect of the embodiments of this application.

[0027] like Figure 1 As shown, Figure 1 A flowchart illustrating a smart home control method provided in an embodiment of the present invention. Taking the smart home control method being executed by a server as an example, a smart home control method includes the following steps:

[0028] Step S102: Collect multimodal data of smart home, including physical device data, environmental data and user data.

[0029] In this embodiment, the provided smart home control method is applicable to smart home control scenarios. During the smart home control process, the smart home control system can monitor data from all physical devices, environmental data, and user data involved in the entire process in real time, including real-time reading, judgment, and control of each data point. The physical devices refer to the various devices controlled in the smart home, such as smart appliances, smart lighting, smart security systems, and smart environmental control devices. Smart appliances include smart TVs, smart refrigerators, smart washing machines, etc.; smart lighting includes smart bulbs, smart light strips, and smart light belts; smart security systems include smart door locks, smart cameras, door and window sensors, etc.; and smart environmental control devices include smart air conditioners, smart air purifiers, and smart humidifiers / dehumidifiers, etc. It should be noted that this embodiment does not limit the type and quantity of the physical devices; these can be determined based on the actual physical devices used.

[0030] The aforementioned physical equipment data includes the size, shape, and materials of the physical equipment; its operating status, degree of damage, and normal maintenance schedule; and historical data (past maintenance records and production records). Various sensors installed within the physical equipment, along with manual records, are used to monitor the operating status, degree of damage, and normal maintenance schedule to acquire physical equipment data in real time.

[0031] The aforementioned environmental data includes both indoor and outdoor environmental data. Indoor environmental data includes air quality, light intensity, sound levels, temperature, humidity, and gas concentrations. Outdoor environmental data includes natural environment (weather conditions, air quality, light conditions), geographical and ecological factors (geographical location, surrounding vegetation, water bodies), and socio-cultural environment (noise levels, pollution sources, and surrounding activities). Environmental data can be collected by installing various sensors in the indoor environment where smart home devices are located, such as temperature sensors, humidity sensors, motion sensors, light sensors, locators, cameras, and microphones. Various data sensors can also be installed in the outdoor environment. It should be noted that various data from the environment in which the physical devices are located can be acquired in real time through these sensors. Then, the data from different sensors can be centralized on a unified data platform for format conversion and standardization to facilitate subsequent processing.

[0032] The aforementioned user data includes user characteristics and user behavioral characteristics. User characteristics include physiological and psychological features. Physiological characteristics can be acquired through sensors and wearable devices, focusing on the user's health and behavioral patterns. Psychological characteristics are inferred indirectly through voice, facial expressions, and behavioral habits, combined with AI algorithms to analyze user emotions, personality, and preferences. User behavioral characteristics include habitual switching of lights, temperature adjustments, and device operation. Device usage information for user behavioral characteristics is based on the most recent data to ensure accurate identification of user needs.

[0033] In this embodiment, when controlling a smart home, it is necessary to collect relevant data from the corresponding physical devices and user data in real time to facilitate the management and control of the smart home. Furthermore, by providing diverse data sources, more personalized analysis can be achieved in smart home control.

[0034] Step S104: Perform data fusion on the physical device data, environmental data, and user data in the multimodal data to obtain the data fusion result.

[0035] In some embodiments, physical features of physical equipment data are extracted, such as the size, shape, and material of the physical equipment, including the operating status, damage level, and predicted normal maintenance time of the physical equipment; indoor air quality, light intensity, sound level, temperature, humidity, and gas concentration are extracted, as well as outdoor natural environment (weather conditions, air quality, light conditions), geographical and ecological factors (geographical location, surrounding vegetation, water bodies), and socio-cultural environment (noise level, pollution sources, and surrounding activities) are extracted; user features, including physiological and psychological features, and user behavior features, including the usage of equipment, are extracted, with the most recent information as the basis, to ensure the accuracy of user demand conversion identification.

[0036] In some embodiments, the collected multimodal data, such as physical device data, environmental data, and user data, are cleaned to remove erroneous data, duplicate data, and noise. For example, sensors may generate abnormal data due to environmental interference or malfunctions, which need to be identified and processed by data cleaning algorithms. Simultaneously, standardizing and normalizing the data can improve the data quality of each modality, enabling different types of data to be analyzed and modeled at the same scale.

[0037] Specifically, after extracting effective features from physical device data, environmental data, and user data, the most representative and discriminative features are selected for fusion. Common methods include principal component analysis, linear discriminant analysis, and convolutional neural networks in deep learning. This approach reduces data volume, highlights key information, and achieves good fusion results. The fused data is then organized to have a clear structure and standardized format, thus obtaining the corresponding data fusion results, which facilitate subsequent analysis and use.

[0038] In this embodiment, each data source has limitations when used alone. Fusing data from different modalities can compensate for the deficiencies of single-modal data, leading to a more accurate and complete understanding of the surrounding environment. Of course, data fusion is also used in specific application scenarios such as abnormal behavior recognition (e.g., unexpected user behavior, abnormal device operation, etc.); through a dynamic perception model of multimodal data, more accurate predictions of user psychology and behavioral trends can be achieved.

[0039] Step S106: Construct a virtual model, which includes a physical model, an environment model, and a user model, and establish a dynamic feedback mechanism among the physical model, the environment model, and the user model.

[0040] In some embodiments, the virtual model described above can help designers quickly verify the feasibility of smart home control. Creating physical prototypes for smart homes is often costly and time-consuming. Virtual models can, to some extent, replace physical prototypes, reducing the number of production attempts.

[0041] In this embodiment, the physical model described above is a digital description of a physical entity (such as a device in a smart home), including the physical entity's geometry, mechanical properties, electrical properties, etc. For example, in a smart home system, the physical model of a smart refrigerator would include the refrigerator's external dimensions, internal structure, power and efficiency of the refrigeration system, and other physical attributes.

[0042] The environmental model described above is used to describe the surrounding environment of a physical entity (physical device). The surrounding environment can include indoor and outdoor environments. For example, the indoor environment includes environmental factors such as indoor temperature, humidity, lighting, and moving objects. The outdoor environment includes information such as natural environment (weather conditions, air quality, lighting conditions), geographical and ecological factors (geographical location, surrounding vegetation, water bodies), and socio-cultural environment (noise levels, pollution sources, and surrounding activities).

[0043] The user model described above is an abstract representation of user behavior, preferences, and intentions. In smart homes, a user model might include a user's preferred range of indoor temperature, preferred lighting brightness and color modes, and habits of using home appliances (such as the amount of time spent using the washing machine each day).

[0044] In this embodiment, after fusing physical device data, environmental data, and user data, a related virtual model can be constructed to simulate various user operations on the device, such as controlling the switching, brightness adjustment, and color change of lights through a mobile application, and checking whether the linkage function between devices (such as automatically turning on the living room lights when the door lock is opened) is normal, thereby discovering and resolving functional problems before the actual deployment of the system.

[0045] Of course, a dynamic feedback mechanism between the various models in the virtual model is also needed to adapt to and optimize the user experience. For example, the behavior of physical entities can be adjusted in real time according to changes in the environment and user needs to achieve better performance, higher efficiency, and services that better meet user expectations. For instance, in a smart home environment, when the ambient temperature rises and the user model shows that the user prefers a lower indoor temperature, the air conditioning equipment in the physical model can automatically adjust its cooling power through a dynamic feedback mechanism.

[0046] Step S108: Train the physical model, environment model, and user model in the virtual model based on the data fusion results to obtain a digital twin model.

[0047] In this embodiment, after obtaining the fused data of physical device data, environmental data, and user data, and constructing a virtual model, the data fusion results are used to train and integrate the sub-models in the virtual model. This enables the construction of a digital twin model that meets actual needs, has high accuracy and practicality, and provides strong support for intelligent applications in many fields.

[0048] Step S110: Based on the digital twin model, a self-supervised learning algorithm is used to control the smart home according to the real-time collected smart home multimodal data.

[0049] In some embodiments, a self-supervised learning algorithm is used to continuously collect new data to optimize and update the digital twin model, ensuring that the model can adapt to changing user needs and device operating status. A real-time feedback mechanism is established, enabling the virtual model to dynamically adjust based on real-time data, reflecting the latest user behavior and device status. This closed-loop process, based on a digital twin model and using a self-supervised learning algorithm to process real-time collected multimodal data from smart homes, enables more intelligent, precise, and adaptive control of the smart home system, providing users with a more comfortable, convenient, and secure home living experience.

[0050] In this embodiment of the invention, the present application can be applied to smart home control scenarios. The present application collects multimodal data from a smart home, including physical device data, environmental data, and user data; it then fuses the physical device data, environmental data, and user data from the multimodal data to obtain a data fusion result; it constructs a virtual model, including a physical model, an environmental model, and a user model, establishing a dynamic feedback mechanism among the physical model, environmental model, and user model; it trains the physical model, environmental model, and user model in the virtual model based on the data fusion result to obtain a digital twin model; and it uses a self-supervised learning algorithm based on the digital twin model to control the smart home system according to the real-time collected multimodal data. This allows for the full utilization of multimodal data to construct a digital twin model and, with the help of a self-supervised learning algorithm, achieves more intelligent, precise, and adaptive control of the smart home system, enabling smart homes to better serve people's lives and create a more comfortable, convenient, and safe home environment.

[0051] In an optional embodiment, such as Figure 2 As shown, step S102 specifically includes:

[0052] Step S202: Construct a three-dimensional model of the physical device based on its physical characteristics to obtain the physical model;

[0053] Step S204: Construct a three-dimensional environmental model based on the environmental characteristics of the physical device to obtain the environmental model;

[0054] Step S206: Construct a three-dimensional user model based on user characteristics and user behavior characteristics to obtain the user model;

[0055] Step S208: Establish a dynamic feedback mechanism among the physical model, the environment model, and the user model to obtain the virtual model.

[0056] In some embodiments, the physical characteristics of the aforementioned physical device include material properties (such as density, elastic modulus, thermal conductivity, etc.), mechanical parameters (such as mass, moment of inertia, friction, etc.), electrical parameters (such as resistance, capacitance, inductance, etc.), and other related characteristics. For example, in the physical model of a smart light bulb, the resistance of the filament is defined to simulate the process of converting electrical energy into light energy.

[0057] In this embodiment, computer-aided design (3ds Max) software is used to construct a three-dimensional model of the physical device based on its physical characteristics. For devices in a smart home, their appearance and internal structure are accurately depicted, such as the shape of the air conditioner's casing and the layout of its heat sinks, and the screen size and body proportions of a television, to ensure a high degree of spatial conformity with the actual physical entity, thereby obtaining the physical model corresponding to the physical device.

[0058] In some embodiments, key environmental factors related to smart home and application scenarios are identified. In a smart home environment, these primarily consider indoor and outdoor temperature, humidity, light intensity, air quality (such as PM2.5 concentration, CO2 content, etc.), and noise levels. For quantifiable and relatively stable environmental factors, such as topography, Geographic Information System (GIS) data and mathematical models can be used. For dynamically changing environmental factors, such as weather and air quality, a real-time data acquisition and update mechanism is established. This can be linked to data sources from meteorological departments or environmental monitoring agencies to obtain the latest environmental information. Time series analysis and stochastic process models are then used to simulate their changing trends, thereby constructing the corresponding environmental model.

[0059] In some embodiments, interfaces and mechanisms for how the environmental model affects the physical model can be defined. For example, in a smart home, ambient temperature and humidity affect the cooling or heating load of an air conditioner. By setting the heat transfer coefficient and humidity balance equation, ambient temperature and humidity data are input into the air conditioner's physical model, thereby affecting the air conditioner's operating status and energy consumption.

[0060] In this embodiment, data analysis and machine learning techniques are used to analyze the collected user features and user behavior features and establish corresponding three-dimensional user models, thereby obtaining the corresponding user models.

[0061] In some embodiments, an efficient data transmission channel is established to ensure that the physical model, environmental model, and user model can exchange data in real time or periodically. Feedback triggering conditions are set between the physical model, environmental model, and user model. For example, in the physical model, thresholds or event triggering conditions are set, such as equipment fault warnings (when the air conditioner compressor current abnormally increases or the motor temperature exceeds a safety threshold), performance index changes (such as the energy consumption of the intelligent lighting system exceeding a set range), etc. These situations will trigger feedback requests to the environmental model and user model. When environmental factors change significantly (such as a sudden drop in temperature or deterioration in air quality), the environmental model sends adjustment instructions to the physical model (such as requiring the air conditioner to switch to heating mode or the air purifier to increase its purification power) and notifies the user model to make user prompts or behavioral prediction adjustments. When user intentions change or abnormal behavior patterns occur, the user model provides feedback information to the physical model and environmental model.

[0062] Furthermore, based on the feedback triggering conditions, feedback control strategies are formulated among the various models. For feedback received by the physical model from the environmental and user models, corresponding control algorithms are designed to adjust the behavior of physical entities. For example, when the ambient temperature rises and the user model shows a lower preferred indoor temperature, the air conditioner in the physical model adjusts its cooling power and fan speed according to the feedback information to achieve the user's desired comfortable temperature. After the environmental model receives operational status feedback from the physical model, such as energy consumption data from smart devices, the environmental model can analyze the impact of environmental factors on energy consumption and propose suggestions for optimizing environmental settings (such as adjusting the opening and closing of curtains to optimize lighting and insulation). Based on the feedback from the physical and environmental models, the user model updates the user's behavior prediction and preference model, while providing personalized information feedback and operational suggestions to the user.

[0063] In this embodiment, a virtual model containing a physical model, an environmental model, and a user model, and equipped with a dynamic feedback mechanism, can be constructed. This model can effectively simulate and optimize the interaction between physical entities and the environment and users in different application scenarios, providing powerful decision support and system optimization tools for smart homes.

[0064] In an optional embodiment, combined with Figure 3 As shown, step S206 specifically includes:

[0065] Step 302: Establish a user profile model based on user characteristics;

[0066] Step 304: Establish a user behavior model of the interaction between the physical device and the user based on user behavior characteristics;

[0067] Step 306: Construct a three-dimensional user model based on the user profile model and the user behavior model to obtain the user model.

[0068] In some embodiments, user profiles can be modeled based on users' psychological and physiological characteristics, providing strong support and decision-making basis for personalized services in smart homes.

[0069] The aforementioned user behavior characteristics may include behavior frequency characteristics, behavior time characteristics, behavior pattern characteristics, and behavior preference characteristics.

[0070] Specifically, the frequency of specific user operations on physical devices is calculated. For example, the number of times a user uses a smart washing machine daily, or the frequency of adjusting the brightness of smart lights weekly, can be statistically analyzed to obtain corresponding frequency characteristics. These frequency characteristics can reflect the user's dependence on device functions and usage patterns.

[0071] Analyze the temporal distribution patterns of user operations on physical devices. This includes the distribution of usage time within a day (e.g., users typically use smart TVs between 7 PM and 10 PM), the distribution of usage days within a week (e.g., users mainly use smart robot vacuums on weekends), and the time intervals between different operations (e.g., how long after turning on a smart air conditioner will a user adjust the temperature setting). These temporal characteristics help understand users' daily routines and device usage scenarios, providing a basis for personalized device control and service recommendations.

[0072] Data mining can uncover patterns in how users interact with physical devices. For example, data mining algorithms can reveal a common user sequence when using a smart oven: first, set the temperature; then, select the baking mode; and finally, set the baking time. Identifying these patterns can help optimize the device's user interface design to better suit user habits. It can also provide insights for automated control and intelligent assistance functions, such as automatically recommending a suitable baking time after the user has set the temperature and baking mode.

[0073] Based on users' choices of different function options and parameter settings for physical devices, their preference characteristics can be determined. For example, analyzing users' preferences for frequently selected temperature ranges, fan speed modes, and whether to enable energy-saving modes on smart air conditioners can help. These preference characteristics can be used for personalized device configuration and operating strategy adjustments, such as automatically adjusting the preset temperature of the air conditioner according to the user's temperature preferences, or prioritizing the use of energy-saving modes when power supply is tight.

[0074] In this embodiment, the user model obtained by establishing a user profile model based on the aforementioned user characteristics and a user behavior model based on the aforementioned user behavior characteristics can effectively capture the patterns and characteristics of user interaction with physical devices, laying a solid foundation for achieving intelligent device control, personalized service provision, and optimized user experience.

[0075] In an optional embodiment, such as Figure 4 As shown, step 104 specifically includes:

[0076] S402, perform a first data fusion on the user data and the environmental data to obtain a first data fusion result;

[0077] S404, perform a second data fusion on the user data and the physical device data to obtain a second data fusion result;

[0078] S406, perform a third data fusion between the physical device data and the environmental data to obtain a second data fusion result.

[0079] In some embodiments, the fusion of user data and physical device data involves the device model adjusting its operating status according to user needs. The fusion of user data and environmental data involves the user model interacting with the environment model through virtual simulation; for example, simulating a user entering a room, automatically adjusting the lights and raising the temperature. The fusion of device data and environmental data involves the environment model providing the external conditions for device operation, simulating the performance and operating status of the physical device in that environment.

[0080] In this embodiment, data fusion is performed between various data points, and the overall system is made intelligent and dynamically optimized through multimodal data integration and model parameter interaction.

[0081] In an optional embodiment, step S108 specifically includes:

[0082] The first data fusion result is input into the user model and the environment model for training, so as to enable the user model to interact with the environment model through virtual simulation;

[0083] The second data fusion result is input into the user model and the physical model for training, so as to enable the user model to interact with the physical model through virtual simulation;

[0084] The third data fusion result is input into the physical model and the environment model for training, so as to enable the physical model to interact with the environment model through virtual simulation;

[0085] The trained user model, physical model, and environment model are integrated to obtain the digital twin model.

[0086] In this embodiment, interactive training between the user model and the environment model is achieved based on the first data fusion result. This allows the user model to accurately simulate or predict user behavior in a given environment based on the environmental information provided by the environment model; simultaneously, it enables the environment model to respond and adjust reasonably to environmental changes based on user behavior feedback from the user model.

[0087] Based on the results of the second data fusion, the user model and the physical model are trained interactively, enabling the user model to reasonably predict the user's next operation behavior based on the state of the physical devices in the physical model; at the same time, the physical model responds to changes in the operating state that are consistent with the actual situation based on the user behavior fed back by the user model.

[0088] Based on the results of third-party data fusion, the physical model and the environmental model can be trained interactively. The physical model can accurately simulate the operation of physical equipment in the environment based on the environmental conditions provided by the environmental model. At the same time, the environmental model can make reasonable feedback on the environmental impact based on the changes in the equipment's operating status fed back by the physical model.

[0089] In this embodiment, after each model (user model, physical model, and environment model) completes pairwise interactive training, they need to be integrated to construct an organically unified whole, namely, a digital twin model. This integration process must ensure smooth interfaces between the models and accurate, real-time data exchange and transmission, enabling the entire digital twin model to fully simulate the complex relationships and dynamic changes between users, physical devices, and the environment in the real world. This, in turn, improves the adaptability of smart home control methods.

[0090] In an optional embodiment, after step S108, the following is specifically included:

[0091] Speech recognition is performed through the user interaction layer, and scene settings are dynamically adjusted in conjunction with the digital twin model.

[0092] Model self-optimization is achieved through visual feedback from digital twin models.

[0093] In some embodiments, at the user interaction layer, a device capable of acquiring voice signals, such as a microphone array, is first required. A microphone array can collect sound from different directions, improving the quality of voice acquisition and enhancing its resistance to environmental noise. For example, in a smart home environment, a microphone array can be installed in the center of the living room ceiling or on devices such as smart speakers to capture user voice commands from all directions.

[0094] Furthermore, advanced speech recognition technologies, such as deep learning-based speech recognition models, are used to convert preprocessed speech signals into text information. These models are trained on large amounts of speech data to learn the correspondence between different speech features and text, thereby accurately recognizing what the user is saying. Based on the parsed instructions and relevant information in the digital twin model, corresponding scene adjustment operations are triggered.

[0095] Furthermore, create intuitive and easy-to-understand visual interfaces to display relevant information about the digital twin model. These can be presented in two-dimensional or three-dimensional graphical formats. In smart home scenarios, a two-dimensional floor plan can be used to display the room layout and the location and status of various smart devices (such as light on / off status, door and window opening / closing status), allowing users and administrators to intuitively view the overall situation.

[0096] In this embodiment, the interaction between speech recognition and digital twin models, as well as the visualization feedback function of the digital twin model itself, can be fully utilized to improve the intelligence level of the system, user experience, and the performance of the model itself, providing strong support for the digital application of smart homes.

[0097] In an optional embodiment, after step S108 above, the method further includes:

[0098] The digital twin model is deployed on a local server;

[0099] The multimodal data in the digital twin model is encrypted and protected using local edge computing and privacy computing technologies.

[0100] In this embodiment, a local server with appropriate performance is selected based on factors such as the complexity of the digital twin model, the expected amount of data to be processed, concurrent access volume, and confidentiality requirements. It is ensured that the local server has a stable and sufficient internal network connection. A reliable network link must be established with data sources such as sensors and smart devices deployed on physical entities to receive multimodal data in real time (e.g., via wired Ethernet, Wi-Fi, etc.). However, it should be noted that smart home systems can achieve smart home control without cloud support.

[0101] Specifically, appropriate encryption algorithms should be selected based on the characteristics of multimodal data and privacy protection requirements. For example, for sensitive user behavior data (such as user operating habits and preference data in smart home scenarios), symmetric encryption algorithms (such as AES, which has a fast encryption speed and is suitable for encrypting large amounts of data) can be used to ensure that only authorized users or system modules can decrypt and view the data. For data interaction scenarios involving different participants (such as data sharing between different enterprises in the application of digital twin models in supply chain management), asymmetric encryption algorithms (such as RSA, which has better key management and security) can be used to ensure the confidentiality and integrity of data during transmission and storage.

[0102] Furthermore, various mechanisms in privacy-preserving computing technologies are employed to protect multimodal data. For example, homomorphic encryption allows specific computational operations to be performed on encrypted data, such as performing correlation analysis on encrypted device energy consumption data and encrypted ambient temperature data in a digital twin model without first decrypting the data, effectively protecting data privacy. Alternatively, federated learning technology can be used. When a digital twin model needs to integrate information from multiple data sources (such as device data from different users or regions) for model training, each data source can train the data locally, sharing and aggregating only the encrypted information such as the trained model parameters, avoiding the direct transmission of the original data and maximizing the protection of data privacy for all parties.

[0103] In this embodiment, by deploying the digital twin model on a local server and using local edge computing and privacy computing technologies to encrypt and protect the multimodal data, the privacy and security of the data can be fully protected while ensuring the efficient operation of the model, making it more suitable for various scenarios that are sensitive to data and have high security requirements.

[0104] According to another aspect of the embodiments of this application, such as Figure 5 As shown, corresponding to the smart home control method in the above embodiments, this embodiment provides a smart home control device, the device comprising:

[0105] The acquisition module 501 is used to acquire multimodal data of smart homes, including physical device data, environmental data and user data;

[0106] The data fusion module 503 is used to fuse physical device data, environmental data and user data in the multimodal data to obtain data fusion results;

[0107] The construction module 505 is used to construct a virtual model, which includes a physical model, an environment model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environment model, and the user model.

[0108] Training module 507 is used to train the physical model, environment model and user model in the virtual model according to the data fusion result to obtain a digital twin model;

[0109] The control module 509 is used to control the smart home based on the digital twin model using a self-supervised learning algorithm and according to the real-time collected smart home multimodal data.

[0110] It should be noted that in this embodiment, the acquisition module 501 can be used to execute step S102 in this application embodiment, the data fusion module 503 in this embodiment can be used to execute step S104 in this application embodiment, the construction module 505 in this embodiment can be used to execute step S106 in this application embodiment, the training module 407 in this embodiment can be used to execute step S108 in this application embodiment, and the control module 409 in this embodiment can be used to execute step S110 in this application embodiment.

[0111] Optionally, the construction module 405 includes: a first construction unit, used to construct a three-dimensional model of the physical device based on the physical characteristics of the physical device, to obtain a physical model; a second construction unit, used to construct a three-dimensional environmental model based on the environmental characteristics of the physical device, to obtain an environmental model; a third construction unit, used to construct a three-dimensional user model based on user characteristics and user behavior characteristics, to obtain a user model; and an establishment unit, used to establish a dynamic feedback mechanism between the physical model, the environmental model, and the user model, to obtain the virtual model.

[0112] Optionally, the third construction unit includes: a first construction unit for building a user profile model based on user characteristics; a second construction unit for building a user behavior model of the interaction between the physical device and the user based on user behavior characteristics; and a construction subunit for building a three-dimensional user model based on the user profile model and the user behavior model to obtain the user model.

[0113] Optionally, the data fusion module 403 includes: a first data fusion unit, used to perform a first data fusion of the user data and the environmental data to obtain a first data fusion result; a second data fusion unit, used to perform a second data fusion of the user data and the physical device data to obtain a second data fusion result; and a third data fusion unit, used to perform a third data fusion of the physical device data and the environmental data to obtain a second data fusion result.

[0114] Optionally, the training module 407 includes: a first training unit, used to input the first data fusion result into the user model and the environment model for training, so as to enable the user model to interact with the environment model through virtual simulation; a second training unit, used to input the second data fusion result into the user model and the physical model for training, so as to enable the user model to interact with the physical model through virtual simulation; a third training unit, used to input the third data fusion result into the physical model and the environment model for training, so as to enable the physical model to interact with the environment model through virtual simulation; and an integration unit, used to integrate the trained user model, physical model, and environment model to obtain the digital twin model.

[0115] Optionally, the device further includes: an adjustment module for performing speech recognition through a user interaction layer and dynamically adjusting scene settings in conjunction with the digital twin model; and an optimization module for performing model self-optimization through visual feedback from the digital twin model.

[0116] Optionally, the device further includes: a deployment module for deploying the digital twin model on a local server; and an encryption module for encrypting and protecting the multimodal data in the digital twin model using local edge computing technology and privacy computing technology.

[0117] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0118] It should be noted that the suffixes such as module, component, unit, submodule, and subunit used to represent elements in the above-described device are only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, they can be used in combination.

[0119] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 6 As shown, the system includes a memory 601, a processor 603, a communication interface 605, and a communication bus 607. The memory 601 stores a computer program that can run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. When the processor 603 executes the computer program, it implements the steps of the above-mentioned smart home control method.

[0120] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0121] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0122] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0123] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the smart home control method in any of the above embodiments.

[0124] Optionally, in embodiments of this application, a computer-readable medium is configured to store program code for the processor to execute the steps of the smart home control method described in the above embodiments.

[0125] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here. Furthermore, in the specific implementation of this application embodiment, the above embodiments can be consulted, and corresponding technical effects can be achieved.

[0126] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0127] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0133] It should be noted that, in this document, relational terms such as "first," "second," etc., are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprises a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A smart home control method, characterized in that, The method includes: Collect multimodal data of smart homes, including physical device data, environmental data, and user data; The physical device data, environmental data, and user data in the multimodal data are fused to obtain the data fusion result; A virtual model is constructed, which includes a physical model, an environment model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environment model, and the user model. Based on the data fusion results, the physical model, environment model, and user model in the virtual model are trained to obtain a digital twin model; Based on the aforementioned digital twin model, a self-supervised learning algorithm is used to control the smart home based on real-time collected multimodal data of the smart home. The step of training the physical model, environment model, and user model in the virtual model based on the data fusion results to obtain a digital twin model includes: performing bidirectional interactive training on corresponding sub-models based on the pairwise fusion results of multimodal data; and integrating the trained sub-models to obtain the digital twin model. The sub-model includes the physical model, the environment model, and the user model, and the bidirectional interactive training refers to: When the fusion results of user data and environmental data are used to train user models and environmental models, the user model can simulate or predict user behavior in the corresponding environment based on the environmental information provided by the environmental model, and the environmental model can respond to and adjust to environmental changes based on the user behavior fed back by the user model. When the fusion results of user data and physical device data are used to train the user model and the physical model, the user model can predict the user's next operation behavior based on the state of the physical device in the physical model, and the physical model can respond to changes in the operating state that are consistent with the actual situation based on the user behavior fed back by the user model. When the fusion results of physical device data and environmental data are used to train the physical model and the environmental model, the physical model can simulate the operation of the physical device in the environment based on the environmental conditions provided by the environmental model. At the same time, the environmental model can make reasonable responses to environmental impacts based on the changes in the device's operating status fed back by the physical model.

2. The smart home control method according to claim 1, characterized in that, The construction of the virtual model includes: A three-dimensional model of the physical device is constructed based on its physical characteristics to obtain the physical model. A three-dimensional environmental model is constructed based on the environmental characteristics of the physical equipment to obtain the environmental model; A three-dimensional user model is constructed based on user characteristics and user behavior characteristics to obtain the user model; A dynamic feedback mechanism is established between the physical model, the environment model, and the user model to obtain the virtual model.

3. The smart home control method according to claim 2, characterized in that, The process of constructing a three-dimensional user model based on user characteristics and user behavior characteristics to obtain the user model includes: Build user profile models based on user characteristics; Establish a user behavior model of the interaction between the physical device and the user based on user behavior characteristics; The user model is obtained by constructing a three-dimensional user model based on the user profile model and the user behavior model.

4. The smart home control method according to claim 2, characterized in that, The process of fusing physical device data, environmental data, and user data from the multimodal data to obtain the data fusion result includes: The user data and the environmental data are fused together to obtain a first data fusion result. The user data and the physical device data are then fused together to obtain a second data fusion result. The physical device data and the environmental data are fused together to obtain a third data fusion result.

5. The smart home control method according to claim 4, characterized in that, The step of training the physical model, environment model, and user model in the virtual model based on the data fusion results to obtain a digital twin model includes: The first data fusion result is input into the user model and the environment model for training, so as to enable the user model to interact with the environment model through virtual simulation; The second data fusion result is input into the user model and the physical model for training, so as to enable the user model to interact with the physical model through virtual simulation; The third data fusion result is input into the physical model and the environment model for training, so as to enable the physical model to interact with the environment model through virtual simulation; The trained user model, physical model, and environment model are integrated to obtain the digital twin model.

6. The smart home control method according to claim 1, characterized in that, After controlling the smart home based on the real-time collected multimodal data of the smart home using a self-supervised learning algorithm based on the digital twin model, the method further includes: Speech recognition is performed through the user interaction layer, and scene settings are dynamically adjusted in conjunction with the digital twin model. Model self-optimization is achieved through visual feedback from digital twin models.

7. The smart home control method according to any one of claims 1-6, characterized in that, After training the physical model, environment model, and user model in the virtual model based on the data fusion results to obtain a digital twin model, the method further includes: The digital twin model is deployed on a local server; The multimodal data in the digital twin model is encrypted and protected using local edge computing and privacy computing technologies.

8. A smart home control device for implementing the smart home control method as described in any one of claims 1 to 7, characterized in that, The device includes: The data acquisition module is used to collect multimodal data from smart homes, including physical device data, environmental data, and user data. The data fusion module is used to fuse physical device data, environmental data, and user data in the multimodal data to obtain the data fusion result; A construction module is used to build a virtual model, which includes a physical model, an environment model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environment model, and the user model. The training module is used to train the physical model, environment model, and user model in the virtual model based on the data fusion results to obtain a digital twin model. The control module is used to control the smart home based on the digital twin model using a self-supervised learning algorithm and real-time collected smart home multimodal data.

9. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the smart home control method according to any one of claims 1 to 7.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the smart home control method according to any one of claims 1 to 7.

Images