Smart home control method and device, electronic equipment and computer readable medium

Abstract

The invention relates to a smart home control method and device, electronic equipment and a computer readable medium. The method comprises the following steps: collecting multi-modal data of the smart home, wherein the multi-modal data comprises physical equipment data, environment data and user data; performing data fusion on physical equipment data, environment data and user data in the multi-modal data to obtain a data fusion result; a virtual model is constructed, the virtual model comprises a physical model, an environment model and a user model, and a dynamic feedback mechanism is established among the physical model, the environment model and the user model; training a physical model, an environment model and a user model in the virtual model according to the data fusion result to obtain a digital twin model; and controlling the smart home according to smart home multi-modal data collected in real time by adopting a self-supervised learning algorithm based on the digital twin model. According to the invention, the adaptability of the smart home device can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a smart home control method, device, electronic device and computer-readable medium. Background Art

[0002] Smart home technology is widely used in home control, environmental perception and device linkage to achieve remote control and automated management of home devices. In recent years, through the integration of artificial intelligence and the Internet of Things, smart homes have entered the intelligent era.

[0003] However, existing smart home systems mainly rely on simple communications between sensors and devices. Personalized functions are mostly based on pre-set regulations and parameters to predict user behavior and achieve device control and feedback. They lack a deep understanding of user behavior and are unable to dynamically adapt to changes in user needs, resulting in low adaptability of smart home devices. Summary of the invention

[0004] The present application provides a smart home control method, device, electronic device and computer-readable medium to solve the technical problems in the above-mentioned prior art that the prior art lacks a deep understanding of user behavior and cannot dynamically adapt to changes in user needs.

[0005] According to one aspect of an embodiment of the present application, the present application provides a smart home control method, the method comprising: collecting multimodal data of the smart home, the multimodal data comprising physical device data, environmental data and user data; performing data fusion on the physical device data, environmental data and user data in the multimodal data to obtain a data fusion result; constructing a virtual model, the virtual model comprising a physical model, an environmental model and a user model, and establishing a dynamic feedback mechanism between the physical model, the environmental model and the user model; training the physical model, the environmental model and the user model in the virtual model according to the data fusion result to obtain a digital twin model; and using a self-supervised learning algorithm based on the digital twin model to control the smart home according to the real-time collected multimodal data of the smart home.

[0006] Optionally, the constructing of the virtual model includes: constructing a three-dimensional model of the physical device according to the physical characteristics of the physical device to obtain a physical model; constructing a three-dimensional environment model according to the characteristics of the environment in which the physical device is located to obtain an environment model; constructing a three-dimensional user model according to user characteristics and user behavior characteristics to obtain a user model; and establishing a dynamic feedback mechanism between the physical model, the environment model and the user model to obtain the virtual model.

[0007] Optionally, constructing a three-dimensional user model based on user characteristics and user behavior characteristics to obtain the user model includes: establishing a user portrait model based on user characteristics; establishing a user behavior model of the interaction between the physical device and the user based on the user behavior characteristics; and constructing a three-dimensional user model based on the user portrait model and the user behavior model to obtain the user model.

[0008] Optionally, the data fusion of the physical device data, environmental data and user data in the multimodal data to obtain a data fusion result includes: performing a first data fusion on the user data and the environmental data to obtain a first data fusion result; performing a second data fusion on the user data and the physical device data to obtain a second data fusion result; performing a third data fusion on the physical device data and the environmental data to obtain a second data fusion result.

[0009] Optionally, the physical model, environmental model and user model in the virtual model are trained according to the data fusion result to obtain a digital twin model, including: inputting the first data fusion result into the user model and the environmental model for training to complete the interaction between the user model and the environmental model through virtual simulation; inputting the second data fusion result into the user model and the physical model for training to complete the interaction between the user model and the physical model through virtual simulation; inputting the third data fusion result into the physical model and the environmental model for training to complete the interaction between the physical model and the environmental model through virtual simulation; integrating the trained user model, physical model and environmental model to obtain the digital twin model.

[0010] Optionally, after the smart home is controlled according to the real-time collected multimodal data of the smart home using a self-supervised learning algorithm based on the digital twin model, the method further includes: performing voice recognition through a user interaction layer, and dynamically adjusting scene settings in combination with the digital twin model; and performing model self-optimization through visual feedback of the digital twin model.

[0011] Optionally, after training the physical model, environmental model and user model in the virtual model according to the data fusion result to obtain the digital twin model, the method also includes: deploying the digital twin model in a local server; encrypting and protecting the multimodal data in the digital twin model through local edge computing technology and privacy computing technology.

[0012] According to another aspect of an embodiment of the present application, the present application provides a smart home control device, which includes: an acquisition module, used to acquire multimodal data of the smart home, the multimodal data including physical device data, environmental data and user data; a data fusion module, used to perform data fusion on the physical device data, environmental data and user data in the multimodal data to obtain a data fusion result; a construction module, used to construct a virtual model, the virtual model including a physical model, an environmental model and a user model, and a dynamic feedback mechanism is established between the physical model, the environmental model and the user model; a training module, used to train the physical model, the environmental model and the user model in the virtual model according to the data fusion result to obtain a digital twin model; a control module, used to control the smart home according to the real-time collected smart home multimodal data using a self-supervised learning algorithm based on the digital twin model.

[0013] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the processor implements the steps of the smart home control method when executing the computer program.

[0014] According to another aspect of an embodiment of the present application, the present application provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the steps of the smart home control method.

[0015] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0016] This application can be applied to smart home control scenarios. This application collects multimodal data of smart homes, and the multimodal data includes physical device data, environmental data, and user data; performs data fusion on the physical device data, environmental data, and user data in the multimodal data to obtain data fusion results; constructs a virtual model, and the virtual model includes a physical model, an environmental model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environmental model, and the user model; trains the physical model, the environmental model, and the user model in the virtual model according to the data fusion results to obtain a digital twin model; and uses a self-supervised learning algorithm based on the digital twin model to control the smart home according to the real-time collected multimodal data of the smart home. The digital twin model can be fully utilized to build a digital twin model, and the self-supervised learning algorithm can be used to achieve more intelligent, accurate, and adaptive control of the smart home system, so that the smart home can better serve people's lives and create a more comfortable, convenient, and safe home environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of an optional smart home control method flow chart provided according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of another optional smart home control method flow chart provided according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of another optional smart home control method flow chart provided according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of another optional smart home control method flow chart provided according to an embodiment of the present application;

[0023] Figure 5 This is a structural diagram of an optional smart home control device provided according to an embodiment of the present application;

[0024] Figure 6 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a smart home control method is provided.

[0027] like Figure 1 As shown, Figure 1 A flow chart of a smart home control method provided by an embodiment of the present invention. Taking the smart home control method executed by a server as an example, a smart home control method includes the following steps:

[0028] Step S102, collecting multimodal data of the smart home, where the multimodal data includes physical device data, environmental data, and user data.

[0029] In this embodiment, the provided smart home control method is applicable to smart home control scenarios. During the smart home control process, the smart home control system can monitor the physical device data, environmental data and user data involved in all links in real time, including real-time reading, judgment, control and other processes of each data. Among them, the physical devices are various devices controlled in the smart home, such as smart home appliances, smart lighting, smart security and smart environmental control, etc.; among them, smart home appliances include smart TVs, smart refrigerators, smart washing machines, etc.; smart lighting includes smart bulbs, smart light strips and smart light strips, etc.; smart security includes: smart door locks, smart cameras, door and window sensors, etc.; smart environmental control includes: smart air conditioners, smart air purifiers and smart humidifiers / dehumidifiers, etc. It should be noted that in this embodiment, the type and quantity of the physical devices are not limited, and can be determined according to the actual physical devices used.

[0030] The physical equipment data includes the size, shape and material of the physical equipment; the operating status, damage degree and normal maintenance time of the equipment, historical data (past maintenance records, production records), etc. The operating status, damage degree and normal maintenance of the physical equipment are detected by various sensors set in the physical equipment and manually recorded to obtain the physical equipment data in real time.

[0031] The above environmental data include: indoor environmental data and outdoor environmental data, etc.; indoor environmental data includes air quality, light intensity, sound level, temperature, humidity, gas concentration, etc.; outdoor environmental data includes natural environment (weather conditions, air quality, light conditions), geographical ecological factors (geographic location, surrounding vegetation, water bodies) and social and cultural environment (noise level, pollution sources and surrounding activities), etc. Among them, environmental data can be obtained by installing a variety of sensors in the indoor environment where smart home devices are located, such as temperature sensors, humidity sensors, motion sensors, light sensors, locators, cameras and microphones, etc. Of course, various data sensors can also be installed in the outdoor environment. It should be noted that various data of the environment in which the physical device is located can be obtained in real time through various sensors, and then the data from different sensors can be concentrated on a unified data platform for format conversion and standardization for subsequent processing.

[0032] The above user data includes: user characteristics and user behavior characteristics, among which user characteristics include physiological characteristics and psychological characteristics; physiological characteristics can be obtained through sensors and wearable devices, focusing on the health and behavior patterns of users; psychological characteristics are indirectly inferred through voice, expression, behavioral habits, etc., combined with AI algorithms to analyze user emotions, personality and preferences. User behavior characteristics include habitually turning on and off lights, adjusting temperature, operating equipment, etc. User behavior characteristics use equipment, based on the latest time, to ensure the accuracy of user demand conversion recognition.

[0033] In this embodiment, when controlling the smart home, it is necessary to collect relevant data of the corresponding physical devices and user-related data in real time in order to manage and control the smart home. Furthermore, by providing diversified data sources, the smart home control can be analyzed more personalized.

[0034] Step S104, performing data fusion on the physical device data, environment data and user data in the multimodal data to obtain a data fusion result.

[0035] In some embodiments, by extracting physical characteristics of physical device data, such as extracting the size, shape and material of the physical device, including the operating status of the physical device, the degree of damage and the prediction of normal maintenance time; extracting the air quality, light intensity, sound level, temperature, humidity gas concentration of the indoor environment and the natural environment (weather conditions, air quality, light conditions), geographical ecological factors (geographic location, surrounding vegetation, water bodies) and social and cultural environment (noise levels, pollution sources and surrounding activities) of the outdoor environment; extracting user characteristics, including physiological characteristics and psychological characteristics, extracting user behavior characteristics, including the use of the equipment, based on the latest time, to ensure the accuracy of user demand conversion identification.

[0036] In some embodiments, the collected multimodal data, such as physical device data, environmental data, and user data, are cleaned separately to remove erroneous data, duplicate data, and noise. For example, a sensor may generate abnormal data due to environmental interference or failure, which needs to be identified and processed by a data cleaning algorithm. At the same time, standardizing and normalizing the data can improve the data quality of each modality and enable different types of data to be analyzed and modeled at the same scale.

[0037] Specifically, after extracting effective features from physical device data, environmental data, and user data, extract effective features from the original physical device data, environmental data, and user data, and select the most representative and discriminative features for fusion. Common methods include principal component analysis, linear discriminant analysis, and convolutional neural networks in deep learning. This method can reduce the amount of data, highlight key information, and achieve better fusion effects. The fused data is sorted out to have a clear structure and standardized format, and then the corresponding data fusion results are obtained for subsequent analysis and use.

[0038] In this embodiment, each data source has limitations when used alone. Fusion of different modal data can make up for the defects of single modal data and understand the surrounding environment more accurately and completely. Of course, data fusion is used in specific application scenarios of abnormal behavior identification (such as unexpected user behavior, abnormal device operation, etc.); through the dynamic perception model of multimodal data, more accurate prediction of user psychology and behavior trends can be achieved.

[0039] Step S106, constructing a virtual model, wherein the virtual model includes a physical model, an environmental model and a user model, and a dynamic feedback mechanism is established among the physical model, the environmental model and the user model.

[0040] In some embodiments, the virtual model can help designers quickly verify the feasibility of smart home control. The production of smart home physical prototypes is often costly and time-consuming. Virtual models can replace physical prototypes to a certain extent and reduce the number of production times.

[0041] In this embodiment, the physical model is a digital description of a physical entity (such as a device in a smart home), including the physical entity's geometric shape, mechanical properties, electrical properties, etc. For example, in a smart home system, the physical model of a smart refrigerator will include the refrigerator's physical dimensions, internal structure, power and efficiency of the refrigeration system, and other physical properties.

[0042] The above environmental model is used to describe the surrounding environment of the physical entity (physical equipment), which can include indoor environment and outdoor environment. For example, the indoor environment includes environmental factors such as indoor temperature, humidity, light, and mobile entities. The outdoor environment includes information such as the natural environment (weather conditions, air quality, and light conditions), geographical ecological factors (geographic location, surrounding vegetation, and water bodies), and social and cultural environment (noise level, pollution sources, and surrounding activities).

[0043] The above user model is an abstract representation of user behavior, preferences, and intentions. In a smart home, the user model may include the user's preferred range for indoor temperature, favorite light brightness and color mode, and the habit of using home appliances (such as the time of day when the washing machine is used).

[0044] In this embodiment, after data fusion of physical device data, environmental data and user data, a related virtual model can be constructed to simulate various user operations on the device, such as controlling the switching, brightness adjustment and color change of lights through mobile phone applications, and checking whether the linkage functions between devices (such as automatically turning on the living room lights when the door lock is opened) are normal, so as to discover and solve functional problems before the actual deployment of the system.

[0045] Of course, it is also necessary to establish a dynamic feedback mechanism between the various models in the virtual model to adapt and optimize the user experience. For example, according to environmental changes and changes in user needs, the behavior of the physical entity can be adjusted in real time to achieve better performance, higher efficiency and services that better meet user expectations. For example, in a smart home environment, when the ambient temperature rises and the user model shows that the user prefers a lower indoor temperature, the air conditioning equipment in the physical model can automatically adjust the cooling power through the dynamic feedback mechanism.

[0046] Step S108: training the physical model, environment model and user model in the virtual model according to the data fusion result to obtain a digital twin model.

[0047] In this embodiment, after obtaining the fusion data of physical device data, environmental data and user data and constructing a virtual model, the data fusion results are used to train and integrate the sub-models in the virtual model, so that a digital twin model that meets actual needs and has high accuracy and practicality can be constructed, providing strong support for intelligent applications in many fields.

[0048] Step S110, using a self-supervised learning algorithm based on the digital twin model, and controlling the smart home according to the real-time collected smart home multimodal data.

[0049] In some embodiments, the digital twin model uses a self-supervised learning algorithm to continuously collect new data to optimize and update the digital twin model, ensuring that the model can adapt to the changing needs of users and the operating status of the equipment; a real-time feedback mechanism is established so that the virtual model can be dynamically adjusted based on real-time data to reflect the latest user behavior and equipment status. Such a closed-loop process based on the digital twin model and using a self-supervised learning algorithm to process real-time collected smart home multimodal data can achieve more intelligent, accurate and adaptive control of the smart home system, providing users with a more comfortable, convenient and safe home life experience.

[0050] In an embodiment of the present invention, the present application can be applied to a smart home control scenario. The present application collects multimodal data of a smart home, wherein the multimodal data includes physical device data, environmental data, and user data; performs data fusion on the physical device data, environmental data, and user data in the multimodal data to obtain a data fusion result; constructs a virtual model, wherein the virtual model includes a physical model, an environmental model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environmental model, and the user model; trains the physical model, the environmental model, and the user model in the virtual model according to the data fusion result to obtain a digital twin model; and uses a self-supervised learning algorithm based on the digital twin model to control the smart home according to the multimodal data of the smart home collected in real time. The digital twin model can be fully utilized to construct a digital twin model, and the self-supervised learning algorithm can be used to achieve more intelligent, accurate, and adaptive control of the smart home system, so that the smart home can better serve people's lives and create a more comfortable, convenient, and safe home environment.

[0051] In an optional embodiment, if Figure 2 As shown, the above step S102 specifically includes:

[0052] Step S202, constructing a three-dimensional model of the physical device according to the physical characteristics of the physical device to obtain a physical model;

[0053] Step S204, constructing a three-dimensional environment model according to the environment characteristics of the physical device to obtain an environment model;

[0054] Step S206, constructing a user three-dimensional model according to the user characteristics and user behavior characteristics to obtain a user model;

[0055] Step S208, establishing a dynamic feedback mechanism among the physical model, the environment model and the user model to obtain the virtual model.

[0056] In some embodiments, the physical properties of the physical device include material properties (such as density, elastic modulus, thermal conductivity, etc.), mechanical parameters (such as mass, moment of inertia, friction, etc.), electrical parameters (such as resistance, capacitance, inductance, etc.) and other related properties. For example, in the physical model of a smart light bulb, the resistance of the filament is defined to simulate the process of converting its electrical energy into light energy.

[0057] In this embodiment, computer-aided design (3dmax) software is used to construct a three-dimensional model of the physical device according to the physical characteristics of the physical device. For the devices in the smart home, their appearance and internal structure are accurately depicted, such as the shell shape and heat sink layout of the air conditioner, the screen size and body ratio of the TV, etc., to ensure that the spatial form is highly consistent with the actual physical entity, and then the physical model corresponding to the physical device is obtained.

[0058] In some embodiments, key environmental factors related to smart home and application scenarios are identified. In a smart home environment, indoor and outdoor temperature, humidity, light intensity, air quality (such as PM2.5 concentration, CO 2 For some environmental factors that are quantifiable and have relatively stable changing patterns, such as topography, geographic information system (GIS) data and mathematical models can be used to build them. For dynamically changing environmental factors, such as weather and air quality, a real-time data collection and update mechanism can be established, which can be connected with the data source of the meteorological department or environmental monitoring agency to obtain the latest environmental information, and use time series analysis and random process models to simulate its changing trends, and then build the corresponding environmental model.

[0059] In some embodiments, interfaces and mechanisms for how the environmental model affects the physical model can be defined. For example, in a smart home, the ambient temperature and humidity can affect the cooling or heating load of the air conditioner. By setting the heat transfer coefficient and the humidity balance equation, the ambient temperature and humidity data are input into the air conditioner physical model, thereby affecting the operating state and energy consumption of the air conditioner.

[0060] In this embodiment, data analysis and machine learning techniques are used to analyze the collected user characteristics and user behavior characteristics and establish a corresponding user three-dimensional model, thereby obtaining a corresponding user model.

[0061] In some embodiments, an efficient data transmission channel is established to ensure that data can be exchanged between the physical model, the environmental model and the user model in real time or periodically. Feedback trigger conditions are set between the physical model, the environmental model and the user model. For example, in the physical model, thresholds or event trigger conditions are set, such as equipment failure warnings (when the current of the air-conditioning compressor increases abnormally or the motor temperature exceeds the safety threshold), performance indicator changes (such as the energy consumption of the intelligent lighting system exceeds the set range), etc. These situations will trigger feedback requests to the environmental model and the user model; when environmental factors change significantly (such as a sudden drop in temperature, deterioration of air quality, etc.), the environmental model sends adjustment instructions to the physical model (such as requiring the air conditioner to switch to heating mode, the air purifier to increase the purification power), and notifies the user model for user prompts or behavior prediction adjustments; when the user's intention changes or the behavior pattern becomes abnormal, the user model will feedback information to the physical model and the environmental model.

[0062] Furthermore, according to the feedback trigger conditions, feedback control strategies between models are formulated. For the feedback received by the physical model from the environmental model and the user model, corresponding control algorithms are designed to adjust the behavior of the physical entity. For example, when the ambient temperature rises and the user model shows that the user prefers a lower indoor temperature, the air conditioner in the physical model adjusts the cooling power and wind speed according to the feedback information to achieve the user's desired comfortable temperature; after the environmental model receives the operating status feedback of the physical model, such as the energy consumption data of the smart device, the environmental model can analyze the degree of influence of environmental factors on energy consumption and make suggestions for optimizing environmental settings (such as adjusting the opening and closing of curtains to optimize lighting and insulation); the user model updates the user's behavior prediction and preference model based on the feedback from the physical model and the environmental model, and provides personalized information feedback and operation suggestions to the user.

[0063] In this embodiment, a virtual model including a physical model, an environmental model, and a user model and having a dynamic feedback mechanism can be constructed. The model can effectively simulate and optimize the interaction between the physical entity and the environment and the user in different application scenarios, and provide powerful decision support and system optimization tools for smart homes.

[0064] In an optional embodiment, in combination Figure 3 As shown, the above step S206 specifically includes:

[0065] Step 302, establishing a user portrait model according to user characteristics;

[0066] Step 304, establishing a user behavior model of the interaction between the physical device and the user according to the user behavior characteristics;

[0067] Step 306: construct a three-dimensional user model based on the user portrait model and the user behavior model to obtain the user model.

[0068] In some embodiments, user portrait modeling can be performed based on the user's psychological and physiological characteristics, which can provide strong support and decision-making basis for personalized services in smart homes.

[0069] The above-mentioned user behavior characteristics may include behavior frequency characteristics, behavior time characteristics, behavior pattern characteristics, and behavior preference characteristics.

[0070] Specifically, the frequency of users' specific operations on physical devices is calculated. For example, the number of times users use smart washing machines every day, the frequency of adjusting the brightness of smart lights every week, etc., is counted to obtain the corresponding frequency characteristics. These frequency characteristics can reflect the user's dependence on device functions and usage patterns.

[0071] Analyze the time distribution of users' operation of physical devices. This includes the usage time distribution within a day (e.g. users usually use smart TVs from 7 to 10 p.m.), the usage days within a week (e.g. users mainly use smart sweeping robots on weekends), and the time interval characteristics between different operations (e.g. how long does it take for users to adjust the temperature setting after turning on a smart air conditioner). These time characteristics help understand users' daily routines and device usage scenarios, and provide a basis for personalized device control and service recommendations.

[0072] Mining the sequence patterns of users operating physical devices. For example, through data mining algorithms, it is found that when users use smart ovens, they usually set the temperature first, then select the baking mode, and finally set the baking time. Identifying these patterns can help optimize the interactive interface design of the device to make it more in line with the user's operating habits. It can also provide reference for the automatic control and intelligent auxiliary functions of the device, such as automatically recommending the appropriate baking time after the user sets the temperature and baking mode.

[0073] Determine the user's preference characteristics based on their selection of different functional options and parameter settings for physical devices. For example, analyze the user's preference information such as the temperature range, wind speed mode, and whether to enable the energy-saving mode that the user often selects on the smart air conditioner. These preference characteristics can be used for personalized device configuration and operation strategy adjustment, such as automatically adjusting the preset temperature of the air conditioner according to the user's temperature preference, or giving priority to recommending users to use the energy-saving mode when the power supply is tight.

[0074] In this embodiment, the user model obtained by establishing a user portrait model through the above-mentioned user characteristics and a user behavior model established through the above-mentioned user behavior characteristics can effectively capture the rules and characteristics of the user's interaction with the physical device, laying a solid foundation for realizing intelligent device control, personalized service provision and optimizing user experience.

[0075] In an optional embodiment, if Figure 4 As shown, the above step 104 specifically includes:

[0076] S402, performing a first data fusion on the user data and the environment data to obtain a first data fusion result;

[0077] S404, performing a second data fusion on the user data and the physical device data to obtain a second data fusion result;

[0078] S406: Perform a third data fusion on the physical device data and the environment data to obtain a second data fusion result.

[0079] In some embodiments, the fusion of user data and physical device data is that the device model adjusts the working state according to user needs. The fusion of user data and environmental data is that the user model interacts with the environmental model through virtual simulation, for example, simulating a user entering a room, the lights automatically brighten, and the temperature rises. The fusion of device data and environmental data is that the environmental model provides external conditions for device operation, simulating the performance and working state of the physical device in the environment.

[0080] In this embodiment, data fusion is performed between various data, and multimodal data integration and model parameter interaction are performed to achieve the intelligence and dynamic optimization of the overall system.

[0081] In an optional embodiment, the above step S108 specifically includes:

[0082] Inputting the first data fusion result into the user model and the environment model for training, so as to complete the interaction between the user model and the environment model through virtual simulation;

[0083] Inputting the second data fusion result into the user model and the physical model for training, so as to complete the interaction between the user model and the physical model through virtual simulation;

[0084] Inputting the third data fusion result into the physical model and the environmental model for training, so as to complete the interaction between the physical model and the environmental model through virtual simulation;

[0085] The trained user model, physical model and environmental model are integrated to obtain the digital twin model.

[0086] In this embodiment, interactive training of the user model and the environment model is implemented based on the first data fusion result. The user model can accurately simulate or predict the user's behavior in the corresponding environment based on the environmental information provided by the environment model; at the same time, the environment model can make reasonable responses and adjustments to environmental changes based on the user behavior fed back by the user model.

[0087] Based on the second data fusion result, interactive training of the user model and the physical model is realized, so that the user model can reasonably predict the user's next operation behavior according to the state of the physical equipment in the physical model; at the same time, the physical model can make a response to the operation status change in accordance with the actual situation based on the user behavior fed back by the user model.

[0088] The physical model and the environmental model are interactively trained based on the third data fusion result, so that the physical model can accurately simulate the operating performance of the physical equipment in the environment according to the environmental conditions provided by the environmental model; at the same time, the environmental model can make reasonable feedback on the environmental impact based on the changes in the equipment operating status fed back by the physical model.

[0089] In this embodiment, after each model (user model, physical model, and environmental model) has completed the interactive training between each other, they need to be integrated together to build an organic and unified whole, namely, a digital twin model. This integration process must ensure that the interfaces between the models are smooth, and that data can be accurately and real-time interactively transmitted, so that the entire digital twin model can fully simulate the complex relationship and dynamic changes between users, physical devices, and the environment in the real world. This will improve the adaptability of the smart home control method.

[0090] In an optional embodiment, after step S108, the following steps are specifically included:

[0091] Performing speech recognition through the user interaction layer and dynamically adjusting scene settings in combination with the digital twin model;

[0092] Model self-optimization is performed through visual feedback of the digital twin model.

[0093] In some embodiments, at the user interaction layer, a device capable of collecting voice signals is first required, such as a microphone array. The microphone array can collect sounds from different directions, improve the quality of voice collection, and enhance the ability to resist interference from environmental noise. For example, in a smart home environment, the microphone array can be installed in the center of the ceiling of the living room or on devices such as smart speakers to capture the voice commands issued by users in all directions.

[0094] Furthermore, advanced speech recognition technologies, such as deep learning-based speech recognition models, are used to convert preprocessed speech signals into text information. These models are trained with a large amount of speech data to learn the correspondence between different speech features and text, so that they can accurately recognize what the user says. Based on the parsed instructions and relevant information in the digital twin model, the corresponding scene adjustment operation is triggered.

[0095] Furthermore, an intuitive and easy-to-understand visual interface is created to display the relevant information of the digital twin model. It can be presented in a two-dimensional or three-dimensional graphical manner. In the smart home scenario, a two-dimensional floor plan is used to display the room layout and the location and status of each smart device (such as the on and off of lights, the opening and closing of doors and windows, etc.), so that users and managers can intuitively view the overall situation.

[0096] In this embodiment, the interaction between speech recognition and the digital twin model and the visual feedback function of the digital twin model itself can be fully utilized to improve the intelligence level of the system, user experience and the performance of the model itself, providing strong support for the digital application of smart homes.

[0097] In an optional embodiment, after the above step S108, the method further includes:

[0098] Deploy the digital twin model in a local server;

[0099] The multimodal data in the digital twin model is encrypted and protected through local edge computing technology and privacy computing technology.

[0100] In this embodiment, a local server with appropriate performance is selected based on the complexity of the digital twin model and factors such as the expected amount of data to be processed, the amount of concurrent access, and the degree of confidentiality. Ensure that the local server has a stable internal network connection that meets the requirements. Establish a reliable network link with data sources such as sensors and smart devices deployed on physical entities to receive multimodal data in real time (such as through wired Ethernet, Wi-Fi, etc.). However, it should be noted that smart homes can achieve smart home control without cloud support.

[0101] Specifically, according to the characteristics of multimodal data and the need for privacy protection, select the appropriate encryption algorithm. For example, for some sensitive user behavior data (such as user operating habits and preference data in smart home scenarios), symmetric encryption algorithms (such as AES algorithm, which has fast encryption speed and is suitable for encrypting large amounts of data) can be used for encryption to ensure that only authorized users or system modules can decrypt and view; for data interaction scenarios involving different participants (such as the application of digital twin models in supply chain management, data sharing between different enterprises), asymmetric encryption algorithms (such as RSA algorithm, which has better key management and security) can be used to ensure the confidentiality and integrity of data during transmission and storage.

[0102] Furthermore, various mechanisms in privacy computing technology are used to protect multimodal data. For example, homomorphic encryption technology is used to enable specific computing operations to be performed on data in an encrypted state, such as correlation analysis of encrypted device energy consumption data and encrypted ambient temperature data in a digital twin model without first decrypting the data, which effectively protects data privacy; or federated learning technology is used. When a digital twin model needs to integrate information from multiple data sources (such as device data from different users and regions) for model training, each data source can train the data locally and only share and aggregate encrypted information such as trained model parameters, avoiding direct transmission of original data and protecting the data privacy of all parties to the greatest extent.

[0103] In this embodiment, by deploying the digital twin model on a local server and encrypting and protecting the multimodal data with the help of local edge computing technology and privacy computing technology, the privacy and security of the data can be fully protected while ensuring the efficient operation of the model, so that it can be better applied to various scenarios that are sensitive to data and have high security requirements.

[0104] According to another aspect of the embodiment of the present application, Figure 5 As shown, corresponding to the smart home control method in the above embodiment, this embodiment provides a smart home control device, the device comprising:

[0105] The acquisition module 501 is used to acquire multimodal data of the smart home, wherein the multimodal data includes physical device data, environmental data and user data;

[0106] The data fusion module 503 is used to fuse the physical device data, the environment data and the user data in the multimodal data to obtain a data fusion result;

[0107] A construction module 505 is used to construct a virtual model, wherein the virtual model includes a physical model, an environment model and a user model, and a dynamic feedback mechanism is established between the physical model, the environment model and the user model;

[0108] A training module 507 is used to train the physical model, the environment model and the user model in the virtual model according to the data fusion result to obtain a digital twin model;

[0109] The control module 509 is used to control the smart home according to the real-time collected smart home multimodal data by using a self-supervised learning algorithm based on the digital twin model.

[0110] It should be noted that, in this embodiment, the acquisition module 501 can be used to execute step S102 in the embodiment of the present application, the data fusion module 503 in this embodiment can be used to execute step S104 in the embodiment of the present application, the construction module 505 in this embodiment can be used to execute step S106 in the embodiment of the present application, the training module 407 in this embodiment can be used to execute step S108 in the embodiment of the present application, and the control module 409 in this embodiment can be used to execute step S110 in the embodiment of the present application.

[0111] Optionally, the construction module 405 includes: a first construction unit, used to construct a three-dimensional model of a physical device according to the physical characteristics of the physical device to obtain a physical model; a second construction unit, used to construct a three-dimensional environment model according to the environmental characteristics of the physical device to obtain an environment model; a third construction unit, used to construct a three-dimensional user model according to user characteristics and user behavior characteristics to obtain a user model; an establishment unit, used to establish a dynamic feedback mechanism between the physical model, the environmental model and the user model to obtain the virtual model.

[0112] Optionally, the third construction unit includes: a first building unit, used to build a user portrait model based on user characteristics; a second building unit, used to build a user behavior model of the interaction between the physical device and the user based on user behavior characteristics; and a construction sub-unit, used to build a user three-dimensional model based on the user portrait model and the user behavior model to obtain the user model.

[0113] Optionally, the data fusion module 403 includes: a first data fusion unit, used to perform a first data fusion on the user data and the environment data to obtain a first data fusion result; a second data fusion unit, used to perform a second data fusion on the user data and the physical device data to obtain a second data fusion result; and a third data fusion unit, used to perform a third data fusion on the physical device data and the environment data to obtain a second data fusion result.

[0114] Optionally, the training module 407 includes: a first training unit, used to input the first data fusion result into the user model and the environment model for training, so as to complete the interaction between the user model and the environment model through virtual simulation; a second training unit, used to input the second data fusion result into the user model and the physical model for training, so as to complete the interaction between the user model and the physical model through virtual simulation; a third training unit, used to input the third data fusion result into the physical model and the environment model for training, so as to complete the interaction between the physical model and the environment model through virtual simulation; an integration unit, used to integrate the trained user model, physical model and environment model to obtain the digital twin model.

[0115] Optionally, the device also includes: an adjustment module, used to perform voice recognition through a user interaction layer, and dynamically adjust scene settings in combination with the digital twin model; and an optimization module, used to perform model self-optimization through visual feedback of the digital twin model.

[0116] Optionally, the device also includes: a deployment module, used to deploy the digital twin model in a local server; and an encryption module, used to encrypt and protect the multimodal data in the digital twin model through local edge computing technology and privacy computing technology.

[0117] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0118] It should be noted here that the suffixes such as module, component, unit, sub-module, and sub-unit used to represent elements described in the above device are only for the convenience of description of this application and have no specific meaning in themselves. Therefore, they can be used in combination.

[0119] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 6 As shown, it includes a memory 601, a processor 603, a communication interface 605 and a communication bus 607. The memory 601 stores a computer program that can be run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. When the processor 603 executes the computer program, the steps of the above-mentioned smart home control method are implemented.

[0120] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0121] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0122] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0123] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the smart home control method in any of the above embodiments.

[0124] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for the processor to execute the steps of the smart home control method described in the above embodiment.

[0125] Optionally, the specific examples in this embodiment can refer to the examples described in the above embodiments, and this embodiment will not be repeated here. And when the embodiments of this application are specifically implemented, they can refer to the above embodiments and have corresponding technical effects.

[0126] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0127] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0128] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0133] It should be noted that, in this article, relational terms such as first, second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A smart home control method, characterized in that: The method comprises: Collecting multimodal data of the smart home, wherein the multimodal data includes physical device data, environmental data, and user data; Performing data fusion on the physical device data, the environment data and the user data in the multimodal data to obtain a data fusion result; Constructing a virtual model, wherein the virtual model includes a physical model, an environmental model, and a user model, and a dynamic feedback mechanism is established between the physical model, the environmental model, and the user model; Training the physical model, the environment model and the user model in the virtual model according to the data fusion result to obtain a digital twin model; Based on the digital twin model, a self-supervised learning algorithm is adopted to control the smart home according to the multimodal data of the smart home collected in real time.

2. The smart home control method according to claim 1, characterized in that: The construction of the virtual model comprises: Construct a three-dimensional model of the physical device according to the physical characteristics of the physical device to obtain a physical model; Constructing a three-dimensional environment model according to the characteristics of the environment in which the physical device is located to obtain an environment model; Build a three-dimensional user model based on user characteristics and user behavior characteristics to obtain a user model; A dynamic feedback mechanism is established between the physical model, the environment model and the user model to obtain the virtual model.

3. The smart home control method according to claim 2, characterized in that: The user three-dimensional model is constructed according to the user characteristics and the user behavior characteristics to obtain the user model, including: Establish a user portrait model based on user characteristics; Establishing a user behavior model of the interaction between the physical device and the user according to the user behavior characteristics; A user three-dimensional model is constructed according to the user portrait model and the user behavior model to obtain the user model.

4. The smart home control method according to claim 2, characterized in that: The fusing the physical device data, the environment data and the user data in the multimodal data to obtain a data fusion result includes: Performing a first data fusion on the user data and the environment data to obtain a first data fusion result; Performing a second data fusion on the user data and the physical device data to obtain a second data fusion result; The physical device data and the environment data are subjected to a third data fusion to obtain a second data fusion result.

5. The smart home control method according to claim 4, characterized in that: The physical model, the environment model and the user model in the virtual model are trained according to the data fusion result to obtain a digital twin model, including: Inputting the first data fusion result into the user model and the environment model for training, so as to complete the interaction between the user model and the environment model through virtual simulation; Inputting the second data fusion result into the user model and the physical model for training, so as to complete the interaction between the user model and the physical model through virtual simulation; Inputting the third data fusion result into the physical model and the environmental model for training, so as to complete the interaction between the physical model and the environmental model through virtual simulation; The trained user model, physical model and environmental model are integrated to obtain the digital twin model.

6. The smart home control method according to claim 1, characterized in that: After the self-supervised learning algorithm is used based on the digital twin model to control the smart home according to the real-time collected smart home multimodal data, the method further includes: Performing speech recognition through the user interaction layer and dynamically adjusting scene settings in combination with the digital twin model; Model self-optimization is performed through visual feedback of the digital twin model.

7. The smart home control method according to any one of claims 1 to 6, characterized in that: After training the physical model, the environment model, and the user model in the virtual model according to the data fusion result to obtain the digital twin model, the method further includes: Deploy the digital twin model in a local server; The multimodal data in the digital twin model is encrypted and protected through local edge computing technology and privacy computing technology.

8. A smart home control device, characterized in that: The device comprises: A collection module, used to collect multimodal data of the smart home, wherein the multimodal data includes physical device data, environmental data and user data; A data fusion module, used for fusing the physical device data, the environment data and the user data in the multimodal data to obtain a data fusion result; A construction module is used to construct a virtual model, wherein the virtual model includes a physical model, an environmental model and a user model, and a dynamic feedback mechanism is established between the physical model, the environmental model and the user model; A training module, used to train the physical model, the environment model and the user model in the virtual model according to the data fusion result to obtain a digital twin model; A control module is used to control the smart home according to the real-time collected multimodal data of the smart home by using a self-supervised learning algorithm based on the digital twin model.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the smart home control method described in any one of claims 1 to 7 is implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the smart home control method described in any one of claims 1 to 7.

Images