Smart home control method, device, equipment and medium

Abstract

The invention discloses a smart home control method, device and equipment and a medium, which can be applied to the fields of finance and medical treatment. According to the method, a heterogeneous sensor array is applied, multi-modal data are collected from an environment side and a user side, and the diversity of the environment and the actual requirements of users are captured. And based on the joint probability model, performing space-time fusion processing on the acquired multi-modal data to generate a joint state vector of the environment and the user, so that the system can understand the specific scene and situation of the user in real time. In combination with a user scene instruction and a joint state vector, an equipment control instruction set with priority is generated through a reinforcement learning decision engine, so that more intelligent operation logic is realized. According to the method, the use experience of the user in the smart home control process is remarkably improved, so that the smart home system can meet the personalized requirements of the user with higher flexibility and intelligence level, and humanized smart home management is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent device control in medical and financial prediction application scenarios, and particularly relates to a smart home control method and device, equipment and a medium. BACKGROUND

[0002] In the prior art, a smart home and environment adaptive system mainly uses single modal data or simple data fusion technology, and lacks the ability of deep dynamic modeling under the traditional data collection node and preliminary fusion system. This makes the data fusion dimension of the smart home control in the prior art insufficient, the priority allocation mechanism rigid, and the cross-modal collaboration missing, so that the control of the smart home cannot be performed according to the actual needs of the user, resulting in the control actually performed negatively optimizing the user experience, hindering the application of the smart home in more commercially valuable fields such as medical and financial fields. Therefore, a new smart home control method is needed to solve the problem of poor user experience caused by the simple control logic of the smart home control method in the prior art. SUMMARY

[0003] Embodiments of the present application provide a smart home control method, device, equipment and medium, aiming to solve the problem of poor user experience caused by the simple control logic of the smart home control method in the prior art.

[0004] In a first aspect, embodiments of the present application provide a smart home control method applied to a smart home control system, the system including a heterogeneous sensor array and a smart home, the method comprising: collecting multi-modal data from an environment end and a user end through the heterogeneous sensor array; performing spatio-temporal fusion processing on the multi-modal data based on a joint probability model to generate a joint state vector of the environment and the user; generating a device control instruction set with priority through a reinforcement learning decision engine according to a user scene instruction and / or the joint state vector; executing the device control instruction set and collecting feedback data, and dynamically optimizing parameters of the dynamic probability model and the reinforcement learning decision engine.

[0005] In a second aspect, embodiments of the present application further provide a smart home control device for executing the smart home control method as described above.

[0006] In a third aspect, embodiments of the present application further provide a computer device, which includes a memory and a processor connected to the memory; the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to execute the steps of the above-mentioned smart home control method.

[0007] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program includes program instructions, and the program instructions, when executed by a processor, can implement the steps of the smart home control method.

[0008] Compared with the prior art, the present application has the following beneficial effects:

[0009] In the technical scheme of the present application, a heterogeneous sensor array is applied to collect multi-modal data from the environment end and the user end, capturing the diversity of the environment and the actual needs of the user. Based on a joint probability model, the collected multi-modal data is processed for spatio-temporal fusion to generate a joint state vector of the environment and the user, so that the system can understand the specific scene and context of the user in real time. In combination with the user scene instruction and the joint state vector, a device control instruction set with priority is generated through the reinforcement learning decision engine, so that a more intelligent operation logic is realized. The present method significantly improves the user experience in the smart home control process, enables the smart home system to meet the personalized needs of the user at a higher flexibility and intelligent level, and realizes the humanized smart home management. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 Flowchart of the smart home control method provided by the present application;

[0012] Figure 2 First sub-flowchart of the smart home control method provided by the present application;

[0013] Figure 3 Second sub-flowchart of the smart home control method provided by the present application;

[0014] Figure 4 Third sub-flowchart of the smart home control method provided by the present application;

[0015] Figure 5 Fourth sub-flowchart of the smart home control method provided by the present application;

[0016] Figure 6 Fifth sub-flowchart of the smart home control method provided by the present application;

[0017] Figure 7 Sixth sub-flowchart of the smart home control method provided by the present application;

[0018] Figure 8 A schematic block diagram of a unit of the smart home control device provided by the present application is shown in the figure;

[0019] Figure 9 A schematic block diagram of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0021] It should be understood that, when used in the present specification and the appended claims, the terms “comprise” and “include” indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] It should also be understood that the terms used in the present specification of the present application are only for the purpose of describing the medical embodiments and are not intended to limit the present application. As used in the present specification and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms, unless the context clearly indicates otherwise.

[0023] It should be further understood that the term “and / or” used in the present specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0024] The present application aims to solve the problem of poor user experience caused by simple control logic of the smart home control method in the prior art, and provides a smart home control method, device, equipment and medium. Referring to Figures 1 to 7 The smart home control method comprises the following steps:

[0025] S110, collecting multi-modal data from the environment end and the user end through a heterogeneous sensor array;

[0026] S120, performing spatio-temporal fusion processing on the multi-modal data based on a joint probability model to generate a joint state vector of the environment and the user;

[0027] S130, generating a device control instruction set with priority through a reinforcement learning decision engine according to a user scene instruction and / or the joint state vector.

[0028] S140, execute the device control instruction set and collect feedback data, and dynamically optimize the parameters of the dynamic probability model and the reinforcement learning decision engine.

[0029] The smart home control system includes a heterogeneous sensor array and a smart home control terminal. The heterogeneous sensor array is composed of multiple types of sensors, mainly including environmental perception sensors, spatial positioning sensors, and biological state sensors. The sensor array synchronously collects multi-modal data of environmental parameters, user spatial position, and user biological state at different time points. The environmental perception data includes temperature, humidity, and light intensity information, the spatial positioning data provides user spatial coordinate information, and the biological state data includes heart rate and fatigue index. In order to ensure the spatio-temporal synchronization of multi-sensor data collection, a clock synchronization mechanism based on NTP protocol is also used, and the time difference of each sensor is corrected by a timestamp alignment algorithm to ensure the consistency and synchronization of the data in time.

[0030] Based on the joint probability model, the multi-modal data is processed for spatio-temporal fusion to generate a joint state vector of the environment and the user. Specifically, an improved dynamic Bayesian network (DBN) architecture is used to define the state nodes and observation nodes within a time slice, and a time transition probability is used to describe the change of the state. The so-called state node refers to the light intensity, user position, heart rate, and fatigue, and the so-called observation node refers to the original sensor data. Through evidence reasoning algorithm, the optimal estimate of the environment and user state is estimated according to the current observation data using a particle filter or other reasoning method, forming a "joint state vector". This vector not only contains static environmental parameters, but also reflects the dynamically changing user state, providing comprehensive background information for subsequent decision-making.

[0031] After obtaining the joint state vector, the system generates a device control instruction set with priority labels according to user scenario instructions such as "reading mode" or "away mode", and action priority strategy using a reinforcement learning decision engine. The reinforcement learning module considers multiple factors and designs a reward function including comfort, energy saving, and user satisfaction, and adjusts the device priority in real time according to environmental changes and user preferences. For example, when the "away mode" is detected and the temperature rises above the set threshold, the control system will raise the air conditioning adjustment priority to the highest, and according to historical data and model prediction, dynamically adjust the control order of other devices. The device control instruction set is sorted according to the priority level to ensure that the system can respond to the most critical environmental adjustment requirements in various scenarios.

[0032] The control instructions are executed by the smart home control terminal to control lighting, air conditioning, curtains and other devices, and real-time collection of execution feedback data such as device operating status and environmental parameter changes. The feedback information is input into the system again, and the parameters are adjusted and optimized using similar dynamic probability models and reinforcement learning algorithms to form a closed-loop feedback mechanism. The parameters of the Bayesian network and the strategy of the reinforcement learning model are continuously adjusted by optimization methods such as gradient descent to adapt to changes in the environment and the personalized needs of users, ensuring that the autonomous learning and continuous optimization capabilities of the system are continuously improved.

[0033] In the technical solution of the present application, a heterogeneous sensor array is applied to collect multi-modal data from the environment and user end, capturing the diversity of the environment and the actual needs of the user. Based on a joint probability model, the collected multi-modal data is processed for spatio-temporal fusion to generate a joint state vector of the environment and the user, enabling the system to understand the specific scene and context of the user in real time. Combined with the user scene instruction and the joint state vector, a set of device control instructions with priority is generated through a reinforcement learning decision engine, thereby realizing more intelligent operation logic. This method significantly improves the user experience in the process of smart home control, enabling the smart home system to meet the personalized needs of users with higher flexibility and intelligence, and realizing humanized smart home management.

[0034] In an embodiment, the step of S110 includes:

[0035] S111, acquiring environmental state data of the target space in real time through an environmental state perception sensor;

[0036] S112, constructing a positioning grid for the target space through a spatial position positioning sensor and obtaining spatial positioning data by positioning the user in real time;

[0037] S113, monitoring the biological state of the user in real time through a biological state monitoring sensor to obtain biological state parameters.

[0038] To achieve the step of collecting multi-modal data by a heterogeneous sensor array, the system is configured with multiple types of sensors to comprehensively perceive the environment, spatial location, and user biological state. Specifically, the environmental state perception sensors include DHT22 temperature and humidity sensors and BH1750 light sensors, which can collect temperature, humidity, and light intensity data of the target space in real time. These sensors transmit the collected environmental parameter data to the central processing unit or edge computing device through wired or wireless connection. The spatial location positioning sensor uses a Decawave DW1000 UWB module to build a positioning grid of the indoor space and locate the user in real time to obtain spatial positioning data. This module provides high-precision spatial coordinate information through ranging and positioning algorithms, ensuring dynamic updating of the user's position in the space. The biological state monitoring sensor integrates a smart bracelet equipped with a PPG photoelectric sensor to monitor the user's heart rate and other biological parameters and obtain the user's biological state parameters in real time by combining a fatigue algorithm. Specifically, the fatigue algorithm is based on HRV heart rate variability analysis of PPG signals. These multi-modal data are synchronized in time and space through a synchronization mechanism to ensure the consistency of data collected by different sensors in time. Specifically, the synchronization mechanism is based on timestamp alignment of the NTP protocol. In specific operation, the environmental state sensors continuously monitor environmental parameters, the spatial positioning sensor updates the user's location in real time, and the biological state sensor continuously monitors the user's physiological indicators. After preprocessing, all data are used as multi-modal input for subsequent environmental modeling and decision analysis, thereby achieving comprehensive perception of the environment and user state and providing basic data support for intelligent scene generation and control.

[0039] Further, the step of S110 further includes:

[0040] S114, synchronizing and calibrating the environmental state data, spatial positioning data, and biological state parameters in time and space through a timestamp alignment algorithm.

[0041] To ensure that the multi-modal data collected from the environment and the user have a consistent time reference, thereby achieving high-precision time and space fusion, the system uses a timestamp-based alignment algorithm to synchronize and calibrate these data. In specific operation, all sensors are equipped with synchronized timestamp information, and the current system time or synchronized standard timestamp is attached when collecting data. Using the NTP protocol-based time synchronization mechanism, the clocks of each sensor network are highly consistent, and then the sensor data from different sources are time-aligned in the data fusion stage. Specifically, the system matches the environmental state data, spatial positioning data, and biological state parameters according to the timestamp of each data, and uses linear interpolation or interpolation algorithms to fill in missing data at some time points to achieve data continuity and synchronization. The formula for synchronization and calibration is:

[0042] tcalibrated = t sensor + Δt network + Δt processing

[0043] wherein Δt network is the network transmission delay, measured by the two-way timestamp method; Δt processing is the sensor data processing delay, calculated by dynamically monitoring the CPU load of each module to dynamically adjust.

[0044] After the space-time calibration, the multi-modal data collected by different sensors can be analyzed in parallel under a unified time scale, effectively eliminating the influence of time errors, ensuring the accuracy and reliability of subsequent multi-modal data fusion, and providing a solid foundation for environmental modeling and scene dynamic generation.

[0045] In an embodiment, the step of S120 comprises:

[0046] S121, constructing a dynamic Bayesian network comprising state nodes, observation nodes, and transition probabilities;

[0047] S122, generating a joint state vector of the environment and the user based on the multi-modal data through the dynamic Bayesian network.

[0048] To realize the spatio-temporal fusion processing of multi-modal data based on the joint probability model, the system adopts a technical solution of constructing a dynamic Bayesian network (DBN) including state nodes, observation nodes and transition probabilities. The system is set with a model structure with state nodes as the core, which includes environmental state, user spatial position and user biological state. The observation nodes correspond to the original sensor data, such as temperature and humidity readings of temperature and humidity sensors, illumination values of illumination sensors, UWB spatial data and heart rate signals of PPG sensors, etc. The transition probability describes the state transition rule of each state node in the continuous time slice, which is realized by defining the state transition probability matrix of each time slice, and reflects the time sequence correlation of the state. In actual operation, the system inputs the multi-modal observation data at the current time into the dynamic Bayesian network according to the pre-trained and calibrated model parameters. In the network, according to the Bayesian inference principle, the algorithm combines the historical state information and the transition probability to calculate the most likely distribution of the current environment and user state, so as to accurately estimate the future and current joint state. The inference process usually adopts particle filtering or backward smoothing method to ensure the estimation stable and reliable in the noise environment. Finally, through the path inference of the Bayesian network, the estimated value of the state node is fused with the observation data to generate a dynamically changing joint state vector reflecting multi-source information, which is the key basis for subsequent decision and control. This method fully utilizes the expressiveness of the Bayesian network, clearly defines the spatio-temporal relationship of multi-modal data, improves the perception ability of the system to environmental changes and user state, and provides strong support for realizing intelligent and accurate environmental regulation.

[0049] Specifically, the dynamic Bayesian network used in the embodiment is a double-layer network structure including state nodes (S t ), observation nodes (O t ) and transition probabilities (T). The state nodes (S t ) include illumination intensity (L t ), user position (P t ), heart rate (H t ) and fatigue degree (F t ). The sensor raw data (D t ={T t , H t , L t , P' t , H' t}) of the observation nodes (O t ), wherein P' t is the original ranging data of UWB, H ' t is the original PPG signal. The transition probability (T) defines the state transition matrix T (S t |St-1 ). The dynamic Bayesian network model is:

[0050]

[0051] where S 1:t represents the state sequence, describing the complete hidden state evolution of the system in T time slices; O 1:t represents the observation sequence, referring to the original data sequence collected by the sensor in T time slices; S t is the state vector at time t, representing the true environment and user state of the system at time t S t = [L t , P t , H t , F t ]; O t represents the observation vector at time t, the original measurement value of the sensor at time t O t = [T t , I t , D' t , B' t ]; P(S1) represents the initial state probability, which is the prior distribution of the state vector at the start of the system; P(S k |S k-1 ) represents the state transition probability, which defines the law of state evolution over time; P(O k |S k ) represents the observation generation probability, which describes how the state generates the sensor readings. Through the above disclosed joint probability model, a probabilistic space-time engine is constructed, which converts the uncertainty of the physical world into a computable probability framework, providing a mathematical foundation for intelligent decision-making.

[0052] Further, the step of S120 further comprises:

[0053] S123, when the user scene instruction is acquired, using a particle filtering algorithm to perform evidence reasoning on the joint state vector to obtain a state estimation value of the state node.

[0054] In order to perform spatio-temporal fusion processing on multi-modal data based on the joint probability model, the step of generating the joint state vector of the environment and the user, the system uses a particle filtering algorithm to perform evidence reasoning and state estimation on the joint state vector. When the system receives a user scene instruction, such as "reading mode" or "leaving home mode", first, the dynamic Bayesian network constructed before is used to process the current multi-modal observation data. Specifically, the particle filtering algorithm generates a large number of state particles, each particle representing a possible environment and user state sample [L t , P t , H t , F tThen, each particle is weighted according to the conditional probability in the model, and the corresponding posterior probability is calculated. The conditional probability is defined by the transition probability matrix and the observation probability distribution. Through the repeated sampling and resampling process, the particle filter effectively approximates the posterior distribution of the joint state, thereby obtaining the optimal state estimation value:

[0055]

[0056] In a specific implementation, the system will use the latest sensor data and combine the previous state estimation value to gradually converge to the most likely joint state through probability update and sample resampling. In practical applications, for example, when the user's wrist heart rate H t exceeds the threshold or the light intensity L t is below a certain level, the particle filter will preferentially emphasize the impact of these observations and derive whether the current environment is in a specific scenario, such as "fatigue reading" or "high-temperature hot environment", through evidence reasoning. Specifically, if the user's wrist heart rate H and the light intensity L are both above the threshold, a light adjustment instruction L adjust = 500 lux is generated. This particle filter-based evidence reasoning greatly enhances the system's adaptability to dynamic environments and user states, making subsequent environmental regulation and device control decisions more accurate and reliable, and ensuring the intelligence of the entire smart home system and the optimization of user experience.

[0057] In an embodiment, the step of S130 includes:

[0058] S131, based on the user scene instruction and / or the joint state vector, constructing a multi-dimensional state space containing environmental parameters, real-time energy consumption of devices, user preferences, and device states;

[0059] S132, generating a multi-objective reward function by integrating comfort, energy consumption, and user satisfaction;

[0060] S133, based on the multi-dimensional state space and the multi-objective reward function, generating a device control action and a dynamic priority allocation scheme through a deep reinforcement learning model, and generating a device control instruction set according to the device control action and the dynamic priority allocation scheme.

[0061] To realize the joint state vector-based decision-making, a set of prioritized device control instructions are generated by the reinforcement learning decision engine. The system first constructs a multi-dimensional state space containing environmental parameters, device real-time energy consumption, user preferences, and device states using user scenario instructions, joint state vectors generated by multi-modal data, and log information of the smart home. Specifically, environmental parameters include temperature, humidity, light, and other environmental perception data. Device real-time energy consumption is collected by the energy consumption monitoring module in the smart home. User preferences are trained by historical operation behavior and preference vectors. Device states include working mode, fault information, and future remaining life. These information are used as different dimensions of the state space to describe the overall operating state of the system. Specifically, the state space is:

[0062] S = {E env ,E energy ,P user ,D device}

[0063] Where E env is the environmental parameter, including temperature, humidity, light, user location, etc. E energy is the device real-time energy consumption, usually collected by Zigbee to collect power data of each device. P user is the user preference, which is the historical operation record converted into a weighted vector, such as [light preference, temperature preference]. D device is the device state, including working mode, fault state, remaining life, etc.

[0064] Then, a multi-objective reward function is set, taking user comfort, energy saving, and satisfaction as evaluation indicators:

[0065] R = α·C comfort + β·E saving + γ·U satisfaction

[0066] Where α, β, γ are adjustment weight terms, α + β + γ = 1, which are dynamically adjusted according to the actual scene. C comfort is calculated by measuring the degree of deviation of environmental parameters from user preference values, such as the Euclidean distance of temperature deviation from user preference temperature. E saving is the difference between actual energy consumption and predicted energy consumption, reflecting energy saving effect. U satisfaction is the actual feedback and satisfaction of the user to the control result.

[0067] Based on the above multi-dimensional state space and multi-objective reward function, the system adopts a deep reinforcement learning (DRL) model, such as a deep Q network (DQN) or a policy gradient method, to input the current state and output the corresponding device control action and priority allocation scheme. In specific implementation, the system will explore the optimal control policy according to the real-time environmental changes, dynamically adjust the priority of device control, and realize intelligent self-adaptation of the scene. Once the appropriate control policy is found, the control action set will be executed through the system's control module, such as adjusting the light brightness, air conditioner temperature, curtain opening degree, etc., and continuously optimizing the model parameters according to the control effect and feedback to continuously improve the user experience and energy efficiency. This deep reinforcement learning scheme based on multi-objective reward greatly enhances the intelligence level of the smart home system, enabling it to achieve efficient and personalized device control under changing environmental conditions, thereby meeting the diverse needs of different user scenarios.

[0068] In an embodiment, the steps of S140 include:

[0069] S141, executing the device control instruction set, and obtaining user feedback data and device feedback data based on log information in the smart home;

[0070] S142, calculating the deviation of the expected effect of the device control instruction set from the multi-modal data and the device feedback data, and adjusting the dynamic Bayesian network based on the deviation;

[0071] S143, updating the user preference vector through attention mechanism weighting based on user feedback data, and updating the reinforcement learning decision engine through the user preference vector.

[0072] To implement the steps of executing device control instructions and collecting feedback data, and then dynamically optimizing the parameters of the dynamic probability model and the reinforcement learning decision engine, the system first collects log information from the smart home after executing the device control instructions, including device state changes, energy consumption data, user operation behavior, and user active or passive feedback information. These data are used as feedback data. The system analyzes these feedback data to calculate the deviation between the collected multi-modal data and the expected effect. Specifically, by using indicators such as environmental parameter changes before and after control, user satisfaction changes, and energy consumption comparisons, the actual effect of the device control strategy is evaluated, and the deviation value can be expressed as:

[0073] Δ = || actual state - expected state ||

[0074] Wherein, the deviation Δ reflects whether the device control strategy achieves the expected target. Based on such deviation, the system will adjust and optimize the dynamic Bayesian network in real time, update the transition probability and observation probability of the state node, in order to improve the accuracy and robustness of the model, so as to better adapt to environmental changes and user needs. In addition, the system will also update the user preference vector V user According to the actual feedback information of the user, the attention mechanism is used to update the user preference vector V

[0075] Through the embodiments of the present application, an intelligent home control method based on multi-modal data fusion and deep machine learning can be clearly determined, which synchronously collects multi-source information such as environment, spatial position and user biological state through a heterogeneous sensor array, uses time stamp alignment to ensure the space-time synchronization of data, and establishes a multi-level dynamic Bayesian network to model the multi-modal data in real time. The joint estimation of environment and user state is realized by using particle filtering and Bayesian inference, and intelligent decision is made by combining deep reinforcement learning, so as to generate device control instructions with dynamic priority. The system optimizes the model parameters in the actual control process by using real-time feedback, so as to continuously realize the environment adaptive, personalized intelligent scheduling. The present application combines multi-modal perception, dynamic modeling and intelligent decision, which significantly improves the environmental perception ability, decision accuracy and dynamic self-optimization level of the intelligent home system.

[0076] The smart home control method of the present application can be applied to the fields of finance and medicine. In the field of finance, it can be used for smart bank site environment optimization, tracking customer flow through UWB positioning, and monitoring waiting anxiety through PPG bracelet. When the anxiety index is detected to rise, the customer's waiting anxiety can be reduced by lowering the environment light color temperature and playing soothing music. At the same time, based on the prediction of passenger flow, the idle area air conditioner can be dynamically closed, which can also save expenses for service sites. In the field of medicine, it can be used for intelligent monitoring and remote medical management. Through multi-modal sensors, the physiological parameters (heart rate, blood pressure, respiratory rate, blood oxygen concentration, etc.), environmental conditions (such as indoor temperature, air quality), and spatial positioning information of patients are collected, and a joint model of the patient's health status is constructed in real time by using space-time synchronization and Bayesian modeling. Combined with deep reinforcement learning, the drug dosage adjustment and health intervention strategy can be dynamically optimized, and the risk prediction and health management can be automatically performed in different medical scenarios. For example, for patients with chronic diseases, through continuous monitoring of their physiological indicators, living environment and behavior patterns, personalized health intervention and automatic early warning can be realized, and the efficiency of medical care and the quality of life of patients can be improved.

[0077] Figure 8 is a schematic block diagram of a smart home control device 600 provided by an embodiment of the present application. As shown in Figure 8 Corresponding to the above smart home control method, the present application also provides a smart home control device 600. The smart home control device 600 includes units for executing the above smart home control method, and the device can be configured in a desktop computer, a tablet computer, a smart phone, etc. terminal.

[0078] Specifically, please refer to Figure 8 The smart home control device 600 includes:

[0079] The data acquisition unit 610 is configured to acquire multi-modal data from the environment end and the user end through a heterogeneous sensor array;

[0080] The data fusion unit 620 is configured to perform space-time fusion processing on the multi-modal data based on a joint probability model to generate a joint state vector of the environment and the user;

[0081] The control instruction generation unit 630 is configured to generate a set of device control instructions with priorities through a reinforcement learning decision engine according to the user scene instruction and / or the joint state vector;

[0082] The self-optimization unit 640 is configured to execute the set of device control instructions and acquire feedback data, and dynamically optimize the parameters of the dynamic probability model and the reinforcement learning decision engine.

[0083] In an embodiment, the data acquisition unit 610 includes:

[0084] An environment state perception unit is configured to acquire environment state data of a target space in real time through an environment state perception sensor.

[0085] A spatial position positioning unit is configured to construct a positioning grid for the target space through a spatial position positioning sensor and to acquire spatial positioning data by positioning the user in real time.

[0086] A biological state monitoring unit is configured to acquire biological state parameters by monitoring the biological state of the user in real time through a biological state monitoring sensor.

[0087] Further, the step of S110 further includes:

[0088] A space-time synchronization calibration unit is configured to calibrate the environment state data, the spatial positioning data, and the biological state parameters through a time stamp alignment algorithm.

[0089] In an embodiment, the data fusion unit 620 includes:

[0090] A joint probability model construction unit is configured to construct a dynamic Bayesian network including state nodes, observation nodes, and transition probabilities.

[0091] A joint state vector generation unit is configured to generate a joint state vector of the environment and the user based on the multi-modal data through the dynamic Bayesian network.

[0092] Further, the data fusion unit 620 further includes:

[0093] A state estimation value generation unit is configured to generate state estimation values of the state nodes by performing evidence inference on the joint state vector through a particle filter algorithm when a user scenario instruction is acquired.

[0094] In an embodiment, the control instruction generation unit 630 includes:

[0095] A multi-dimensional state space construction unit is configured to construct a multi-dimensional state space including environment parameters, real-time energy consumption of devices, user preferences, and device states based on the user scenario instruction and / or the joint state vector.

[0096] A reward function generation unit is configured to generate a multi-objective reward function by comprehensively considering comfort, energy consumption, and user satisfaction.

[0097] A unit is configured to generate a device control action and a dynamic priority allocation scheme through a deep reinforcement learning model based on the multi-dimensional state space and the multi-objective reward function, and to generate a device control instruction set according to the device control action and the dynamic priority allocation scheme.

[0098] In an embodiment, the self-optimization unit 640 includes:

[0099] a control instruction execution unit configured to execute the device control instruction set and obtain user feedback data and device feedback data based on log information in the smart home;

[0100] a joint probability model fine-tuning unit configured to calculate a deviation from an expected effect of the device control instruction set according to the multi-modal data and the device feedback data, and adjust the dynamic Bayesian network based on the deviation;

[0101] a reinforcement learning decision engine updating unit configured to update a user preference vector by attention mechanism weighting based on the user feedback data, and update the reinforcement learning decision engine by the user preference vector.

[0102] The smart home control apparatus 600 can be implemented in the form of a computer program that can run on a computer device as shown in Figure 9 .

[0103] Please refer to Figure 9 , Figure 9 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a terminal or a server, wherein the terminal can be a desktop computer, a tablet computer, a smart phone or the like electronic device having a communication function. The server can be a stand-alone server or a server cluster composed of multiple servers.

[0104] Refer to Figure 9 , the computer device 500 includes a processor 502, a memory and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0105] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions which, when executed, can cause the processor 502 to perform a smart home control method.

[0106] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0107] The internal memory 504 provides an environment for the running of the computer program 5032 in the non-volatile storage medium 503, which, when executed by the processor 502, can cause the processor 502 to perform a smart home control method.

[0108] The network interface 505 is configured to perform network communication with other devices. Those skilled in the art can understand that Figure 9The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0110] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0111] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0112] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the above method.

[0113] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0114] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0115] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed.

[0116] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the apparatus embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0117] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a terminal or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0118] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A smart home control method, characterized in that: Applied to a smart home control system, the system includes a heterogeneous sensor array and a smart home, and the method includes: Collect multimodal data from the environment and user end through heterogeneous sensor arrays; Performing spatiotemporal fusion processing on the multimodal data based on a joint probability model to generate a joint state vector of the environment and the user; Generate a prioritized device control instruction set through a reinforcement learning decision engine based on the user scenario instructions and / or the joint state vector; Execute the device control instruction set and collect feedback data, and dynamically optimize the parameters of the dynamic probability model and the reinforcement learning decision engine.

2. The smart home control method according to claim 1, characterized in that: The step of performing spatiotemporal fusion processing on the multimodal data based on the joint probability model to generate a joint state vector of the environment and the user includes: Construct a dynamic Bayesian network including state nodes, observation nodes, and transition probabilities; A joint state vector of the environment and the user is generated through the dynamic Bayesian network based on the multimodal data.

3. The smart home control method according to claim 2, characterized in that: The step of performing spatiotemporal fusion processing on the multimodal data based on the joint probability model to generate a joint state vector of the environment and the user further includes: When a user scenario instruction is obtained, a particle filter algorithm is used to perform evidential reasoning on the joint state vector to obtain a state estimation value of the state node.

4. The smart home control method according to claim 2, characterized in that: The step of generating a prioritized device control instruction set by a reinforcement learning decision engine according to the user scenario instruction and / or the joint state vector includes: Based on the user scenario instructions and / or the joint state vector, construct a multidimensional state space including environmental parameters, real-time energy consumption of the device, user preferences, and device status; Generate a multi-objective reward function by integrating comfort, energy consumption and user satisfaction; Based on the multi-dimensional state space and multi-objective reward function, a deep reinforcement learning model is used to generate device control actions and a dynamic priority allocation scheme, and a device control instruction set is generated according to the device control actions and the dynamic priority allocation scheme.

5. The smart home control method according to claim 2, characterized in that: The steps of executing the device control instruction set, collecting feedback data, and dynamically optimizing the parameters of the dynamic probability model and the reinforcement learning decision engine include: Executing the device control instruction set, and obtaining user feedback data and device feedback data based on log information in the smart home; calculating a deviation from an expected effect of the device control instruction set based on the multimodal data and the device feedback data, and adjusting the dynamic Bayesian network based on the deviation; The user preference vector is updated based on the user feedback data through the attention mechanism, and the reinforcement learning decision engine is updated through the user preference vector.

6. The smart home control method according to claim 1, characterized in that: The heterogeneous sensor array includes an environmental state perception sensor, a spatial position positioning sensor, and a biological state monitoring sensor. The step of collecting multimodal data from the environment end and the user end through the heterogeneous sensor array includes: Acquire the environmental status data of the target space in real time through the environmental status perception sensor; The spatial positioning sensor is used to construct a positioning grid for the target space and locate the user in real time to obtain spatial positioning data; The biological state monitoring sensor is used to monitor the user's biological state in real time to obtain biological state parameters.

7. The smart home control method according to claim 6, characterized in that: The step of collecting multimodal data from the environment end and the user end through the heterogeneous sensor array further includes: The environmental state data, spatial positioning data, and biological state parameters are calibrated in time and space synchronization through a timestamp alignment algorithm.

8. A smart home control device, characterized in that: Used to execute the smart home control method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images