Control strategy determination method and device, storage medium and electronic device

Abstract

The invention discloses a control strategy determination method and device, a storage medium and an electronic device, and relates to the technical field of smart home. Comprising the following steps: constructing state spaces of a plurality of household appliances in a target area according to state data of each household appliance in the target area, operation data of a target object on each household appliance and environment data of the target area; constructing action spaces corresponding to the plurality of household appliances; determining a first action value of each action in a first state execution action space of each household appliance in the state space according to a target algorithm, and updating model parameters of the strategy model according to a plurality of first action values; and determining a preliminary control strategy corresponding to each household appliance according to the strategy model after the model parameters are updated, and sending each preliminary control strategy to each household appliance, so that each household appliance determines a target control strategy corresponding to each household appliance according to the preliminary control strategy and the real-time state data of each household appliance.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to a method and device for determining a control strategy, a storage medium, and an electronic device. Background Art

[0002] With the rapid development of smart home technology, people are increasingly demanding intelligent control and collaborative operation of home appliances. However, current smart home systems have many problems, such as the lack of effective coordination mechanisms between appliances. Most appliances operate independently, making it difficult to intelligently adjust to the overall environment and user needs.

[0003] The home appliance control method in the related art does not consider the collaborative relationship and resource allocation between multiple home appliances, resulting in operation conflicts and uneven resource allocation among multiple home appliances. No effective solution has been proposed yet. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for determining a control strategy, a storage medium, and an electronic device to at least solve the problem that the home appliance control method in the related art does not take into account the collaborative relationship and resource allocation between multiple home appliances, resulting in operation conflicts and uneven resource allocation among multiple home appliances.

[0005] According to one embodiment of the embodiments of the present application, a method for determining a control strategy is provided, comprising: constructing a state space of multiple household appliances in the target area based on status data of each household appliance in the target area, operation data of the target object on each household appliance, and environmental data of the target area, wherein the state space includes: a set of multiple operating states of each household appliance; constructing an action space corresponding to the multiple household appliances, wherein the action space includes: a set of multiple actions allowed to be performed by each household appliance; determining a first action value for each household appliance to perform each action in the action space in a first state in the state space according to a target algorithm, and updating model parameters of a strategy model according to the multiple first action values, wherein the first state is any state in the state space; determining a preliminary control strategy corresponding to each household appliance according to the strategy model after updating the model parameters, and sending each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance according to the preliminary control strategy and the real-time status data of each household appliance.

[0006] In an exemplary embodiment, updating the model parameters of the strategy model according to multiple first action values ​​includes: determining a second action value among the multiple first action values, and determining the first action corresponding to the second action value, wherein the second action value is the maximum value among the multiple first action values; calculating the immediate reward value of each household appliance performing the first action in the first state; determining a third action value of each household appliance performing each action in the next state after the first state according to the target algorithm, and determining a fourth action value among multiple third action values, wherein the next state is other states in the state space except the first state; updating the model parameters according to the immediate reward value and the fourth action value.

[0007] In an exemplary embodiment, calculating the instantaneous reward value of each household appliance performing the first action in the first state includes: obtaining the instantaneous energy consumption of each household appliance performing the first action in the first state, and determining the energy saving degree of each household appliance performing the first action in the first state based on the instantaneous energy consumption; determining the feedback data of the target object on the performance of the first action by each household appliance in the first state, and determining the user satisfaction of each household appliance performing the first action in the first state based on the feedback data; determining the stable operation time of the device when each household appliance performs the first action in the first state and the environmental comfort of the target area in a target time period, wherein the target time period is the time period when each household appliance performs the first action in the first state; determining the instantaneous reward value based on the energy saving degree, the user satisfaction, the stable operation time of the device, and the environmental comfort.

[0008] In an exemplary embodiment, determining the immediate reward value based on the energy saving degree, the user satisfaction, the stable operation time of the device, and the environmental comfort level includes: determining the immediate reward value R(s,a) based on a first formula, wherein the first formula is: R(s,a)=E(s,a)×w1+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4, s is the first state, a is the first action, E(s,a) is the energy saving degree of each household appliance performing the first action in the first state, w1 is a first weight corresponding to the energy saving degree, S(s,a) is the user satisfaction of each household appliance performing the first action in the first state, w2 is a second weight corresponding to the user satisfaction, D(s,a) is the stable operation time of each household appliance in the first state, w3 is a third weight corresponding to the stable operation time of the appliance, C(s,a) is the environmental comfort level of the target area in the target time period, and w4 is a fourth weight corresponding to the environmental comfort level.

[0009] In an exemplary embodiment, updating the model parameters according to the immediate reward value and the fourth action value includes: determining a characteristic function corresponding to each household appliance performing the first action in the first state; and updating the second action value according to a second formula, wherein the second formula is:

[0010] Q(s1,a1) is the updated second action value, Q(s,a) is the second action value, α is the learning rate, R(s,a) is the immediate reward value, γ is the discount factor, max a′ Q(s′,a′) is the fourth action value, β is the regularization coefficient, is the characteristic function, N is the number of the characteristic functions, θ i is the model parameter; and the model parameter is updated according to the updated second action value.

[0011] In an exemplary embodiment, after determining the control strategy corresponding to each household appliance according to the strategy model after updating the model parameters, the method further includes: when it is determined that the first energy consumption data corresponding to the current operating mode of the first household appliance in the target area is greater than the preset energy consumption data, and the second energy consumption data corresponding to the current operating mode of the second household appliance in the target area is greater than the preset energy consumption data, predicting the first power demand of the first household appliance and the second power demand of the second household appliance, and determining the current power supply value of the target area; determining the priority of the first household appliance and the second household appliance, and determining the first weight of the first household appliance and the second weight corresponding to the second household appliance according to the priority; and modifying the first operating parameter corresponding to the current operating model of the first household appliance and the second operating parameter corresponding to the current operating mode of the second household appliance according to a third formula to obtain the third operating parameter of the modified first operating mode of the first household appliance and the fourth operating parameter corresponding to the modified second operating mode of the second household appliance, wherein the third formula is: mode a The third operating parameter corresponding to the first household appliance, mode b is the fourth operating parameter corresponding to the second household appliance, w a is the first weight, w b is the second weight, T in is the current temperature of the target area, T desired is the target object's expected temperature for the target area, t b-finish is the estimated completion time of the second household appliance performing the action corresponding to the current operation mode of the second household appliance, t desired-finish is the expected completion time for the target object to perform the action corresponding to the current operation mode of the second home appliance on the second home appliance, P a (t) is the first power requirement, P b (t) is the second power requirement, P total-limit is the current power supply value; adjusting the control strategy corresponding to the first household appliance according to the third operating parameter, and adjusting the control strategy corresponding to the second household appliance according to the fourth operating parameter.

[0012] In an exemplary embodiment, predicting the first power requirement of the first household appliance and the second power requirement of the second household appliance includes: predicting the first power requirement according to a fourth formula, wherein the fourth formula is: P a (t) = f a (T in ,T out ,modea1 ,t),T out is the ambient temperature, mode a1 is the first operating parameter, t is the target time, the target time is the time between the current time and the estimated completion time of the first household appliance performing the action corresponding to the current operating mode of the first household appliance, and the target time is the time between the current time and the estimated completion time of the second household appliance performing the action corresponding to the current operating mode of the second household appliance; the second power demand is predicted according to the fifth formula, wherein the fifth formula is: P b (t) = f b (load b ,mode b1 ,t),load b is the load of the second household appliance, mode b1 is the second operating parameter.

[0013] According to another embodiment of the embodiments of the present application, a control strategy determination device is further provided, including: a first construction module, configured to construct a state space of multiple household appliances in a target area based on status data of each household appliance in a target area, operation data of a target object on each household appliance, and environmental data of the target area, wherein the state space includes: a set of multiple operating states of each household appliance; a second construction module, configured to construct an action space corresponding to the multiple household appliances, wherein the action space includes: a set of multiple actions allowed to be performed by each household appliance; a first determination module, configured to determine, according to a target algorithm, a first action value of each household appliance performing each action in the action space in a first state in the state space, and to update model parameters of a strategy model according to the multiple first action values, wherein the first state is any state in the state space; and a second determination module, configured to determine a preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and send each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned method for determining the control strategy when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the control strategy determination method through the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to perform the method.

[0017] In an embodiment of the present application, a state space of the target area including a set of multiple operating states of each household appliance is constructed based on the status data of each household appliance in the target area, the operation data of the target object on each household appliance, and the environmental data of the target area; an action space including a set of multiple actions allowed to be performed by each household appliance is constructed; the first action value of each action in the action space executed by each household appliance in any state in the state space is determined based on a deep learning algorithm, and the model parameters of the strategy model are updated based on the multiple first action values; the preliminary control strategy corresponding to each household appliance is determined based on the strategy model after the model parameters are updated, and each preliminary control strategy is sent to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance. That is to say, the state of the embodiment of the present application includes the operating state of multiple home appliances, and the action space includes actions that are allowed to be executed by multiple home appliances. Then, by determining the first action value of each action in the action space executed by each home appliance in any state in the state space to update the model parameters of the strategy model, the preliminary control strategy determined according to the strategy model can take into account the collaborative relationship and resource allocation problems between multiple home appliances, and then send the preliminary control strategy to each home appliance, so that each home appliance can further determine the target control strategy of each home appliance according to the preliminary control strategy and its actual needs, so that the target control strategy finally corresponding to each home appliance takes into account the collaborative relationship of multiple home appliances and meets the flexibility and adaptability of control strategy formulation. Through the embodiment of the present application, it is possible to solve the problem that the home appliance control method in the related art does not take into account the collaborative relationship and resource allocation between multiple home appliances, resulting in operation conflicts and uneven resource allocation among multiple home appliances. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a hardware environment diagram of a method for determining a control strategy according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for determining a control strategy according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the interaction process between the central control screen and multiple smart home appliances according to an optional embodiment of the present application;

[0023] Figure 4 This is a structural block diagram of a device for determining a control strategy according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to one aspect of the embodiment of the present application, a method for determining a control strategy is provided. The method for determining a control strategy is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, Figure 1 This is a hardware environment diagram of a method for determining a control strategy according to an embodiment of the present application. In this embodiment, the method for determining a control strategy can be applied to Figure 1 In the hardware environment shown in FIG. 1 , a central control device 102 and a server 104 are formed. Figure 1 As shown, the server 104 is connected to the central control device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data calculation services for the server 104.

[0027] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, and a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: WIFI (Wireless Fidelity) and Bluetooth. The central control device 102 may be, but is not limited to, a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing machine, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, smart curtains, smart audio and video, a smart socket, a smart speaker, a smart fresh air device, smart kitchen and bathroom equipment, smart bathroom equipment, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0028] In this embodiment, a method for determining a control strategy is provided. Figure 2 : is a flow chart of a method for determining a control strategy according to an embodiment of the present application, the flow includes the following steps:

[0029] Step S202: constructing a state space of multiple home appliances in the target area based on the state data of each home appliance in the target area, the operation data of the target object on each home appliance, and the environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance;

[0030] The status data of each household appliance may include, but is not limited to, the current operating mode of each appliance (e.g., cooling / heating mode for an air conditioner, washing / spinning mode for a washing machine), operating parameters (e.g., temperature setting, wind speed level, washing time), internal sensor data (e.g., temperature inside a refrigerator, load weight for a washing machine, indoor / outdoor temperature difference for an air conditioner), and energy consumption data of the appliance. It may also include microscopic operating status data of key components within each appliance, such as instantaneous power fluctuations of a refrigerator compressor, fine-tuning changes in air conditioner fan speed, and instantaneous torque values ​​of a washing machine motor.

[0031] The target subject's operational data on each home appliance refers to the target subject's interaction data with each home appliance. This data may include the target subject's (e.g., the user of each home appliance) control operations on the home appliance via a central control screen, mobile phone, remote control, or voice commands, such as adjusting the temperature, selecting a wash mode, and setting a timer. This operational data not only reflects the target subject's current needs and preferences but may also include long-term trends in the target subject's user behavior patterns, such as differences in device usage habits on weekdays and weekends, and the impact of seasonal changes on device operating patterns.

[0032] The environmental data of the above-mentioned target area includes but is not limited to: indoor and outdoor temperature, humidity, light intensity, air quality, noise level, and dynamic information of family members (for example: whether they are at home, the location and activity status of family members), etc.

[0033] Step S204: constructing an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of a plurality of actions that each home appliance is allowed to perform;

[0034] Among them, the actions allowed for each household appliance to perform may include: for air conditioners, this may include adjusting temperature settings, changing wind speeds, turning on or off air purification functions, etc.; for refrigerators, this may include switching cooling modes, adjusting freezer temperature, starting defrost programs, etc.; and the actions of washing machines may involve selecting washing programs, setting washing time and water temperature, starting dehydration programs, etc.

[0035] Step S206, determining, according to a target algorithm, a first action value for each home appliance in a first state in the state space to perform each action in the action space, and updating model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space;

[0036] Among them, the above-mentioned target algorithm can be: deep learning algorithm.

[0037] Step S208, determining a preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and sending each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0038] Through the above steps, a state space of the target area including a set of multiple operating states of each household appliance is constructed according to the status data of each household appliance in the target area, the operation data of each household appliance on the target object and the environmental data of the target area; an action space including a set of multiple actions allowed to be performed by each household appliance is constructed; the first action value of each action in the action space executed by each household appliance in any state in the state space is determined according to the deep learning algorithm, and the model parameters of the strategy model are updated according to the multiple first action values; the preliminary control strategy corresponding to each household appliance is determined according to the strategy model after the model parameters are updated, and each preliminary control strategy is sent to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance according to the preliminary control strategy and the real-time status data of each household appliance. That is to say, the state of the embodiment of the present application includes the operating state of multiple home appliances, and the action space includes actions that are allowed to be executed by multiple home appliances. Then, by determining the first action value of each action in the action space executed by each home appliance in any state in the state space to update the model parameters of the strategy model, the preliminary control strategy determined according to the strategy model can take into account the collaborative relationship and resource allocation problems between multiple home appliances, and then send the preliminary control strategy to each home appliance, so that each home appliance can further determine the target control strategy of each home appliance according to the preliminary control strategy and its actual needs, so that the target control strategy finally corresponding to each home appliance takes into account the collaborative relationship of multiple home appliances and meets the flexibility and adaptability of control strategy formulation. Through the embodiment of the present application, it is possible to solve the problem that the home appliance control method in the related art does not take into account the collaborative relationship and resource allocation between multiple home appliances, resulting in operation conflicts and uneven resource allocation among multiple home appliances.

[0039] Optionally, the above-mentioned step S206 of updating the model parameters of the strategy model according to multiple first action values ​​includes: determining a second action value among the multiple first action values, and determining the first action corresponding to the second action value, wherein the second action value is the maximum value among the multiple first action values; calculating the immediate reward value of each household appliance performing the first action in the first state; determining a third action value of each household appliance performing each action in the next state after the first state according to the target algorithm, and determining a fourth action value among multiple third action values, wherein the next state is other states in the state space except the first state; updating the model parameters according to the immediate reward value and the fourth action value.

[0040] It is understandable that the model parameters of the strategy model can be updated according to multiple first action values. Specifically, the second action value and its corresponding first action are determined: in the Q-learning algorithm (Q-value learning algorithm), the intelligent agent (for example, the central control screen, i.e., the executive subject of this application) evaluates the expected future cumulative reward of performing each action (a) in the current state (s) through the Q-value function (Q value is the first action value). The intelligent agent finds the maximum value from the first action values, i.e., the second action value, and determines the first action corresponding to the second action value.

[0041] Calculate the immediate reward value: After the agent performs the first action, it can receive an immediate reward value (r). The immediate reward value can reflect the immediate effect of action a in the current state s, such as reduced energy consumption, improved environment, or increased user satisfaction.

[0042] Determine the third and fourth action values: The third action value is the Q-value of each action performed by the agent in the next state (s'), as predicted by the deep learning algorithm. The fourth action value is the maximum value found among multiple third action values. The fourth action value represents the expected future cumulative reward for the best action the agent can take in the next state (s').

[0043] Then, the model parameters can be updated according to the immediate reward value and the fourth action value.

[0044] Among them, calculating the instant reward value of each household appliance performing the first action in the first state includes: obtaining the instantaneous energy consumption of each household appliance performing the first action in the first state, and determining the energy saving degree of each household appliance performing the first action in the first state based on the instantaneous energy consumption; determining the feedback data of the target object on the performance of the first action by each household appliance in the first state, and determining the user satisfaction of each household appliance performing the first action in the first state based on the feedback data; determining the stable operation time of the device when each household appliance performs the first action in the first state and the environmental comfort of the target area in the target time period, wherein the target time period is the time period when each household appliance performs the first action in the first state; determining the instant reward value based on the energy saving degree, the user satisfaction, the stable operation time of the device, and the environmental comfort.

[0045] It is understandable that when determining the immediate reward value based on the deep learning algorithm, it is necessary to determine the energy saving level, user satisfaction, stable operation time of the equipment, environmental comfort, and other parameters that affect the immediate reward value, specifically:

[0046] 1) Determining the degree of energy saving: The instantaneous energy consumption data for each home appliance performing the first action a in the first state s is obtained. This data may include information such as the device's power consumption and energy efficiency during operation. The intelligent agent (e.g., the central control panel) evaluates the energy-saving effect of the action based on the instantaneous energy consumption data to determine the degree of energy saving.

[0047] 2) Determining User Satisfaction: The agent also needs to collect feedback data from the target object regarding each home appliance's first action a in the first state s. This feedback data may include user evaluations of the appliance's performance through applications, remote controls, or voice commands, such as whether the indoor temperature is comfortable or the washing results are satisfactory. Based on this feedback data, the agent uses sentiment analysis and semantic understanding techniques to assess user satisfaction with the appliance's control strategy, thereby determining user satisfaction.

[0048] 3) Determine the stable operation time of the equipment: Determine the time that the home appliance can operate stably after performing an action.

[0049] 4) Determination of environmental comfort: The intelligent agent evaluates the environmental comfort of the target area during the target time period (i.e., the time period when the device performs the first action), including temperature, humidity, lighting, air quality and other factors.

[0050] The instant reward value is determined according to the energy saving degree, the user satisfaction, the stable operation time of the device, and the environmental comfort level, including: determining the instant reward value R(s,a) according to a first formula, wherein the first formula is: R(s,a)=E(s,a)×w1+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4, s is the first state, a is the first action, E(s,a) is the energy saving degree of each household appliance performing the first action in the first state, w1 is a first weight corresponding to the energy saving degree, S(s,a) is the user satisfaction of each household appliance performing the first action in the first state, w2 is a second weight corresponding to the user satisfaction, D(s,a) is the stable operation time of each household appliance in the first state, w3 is a third weight corresponding to the stable operation time of the device, C(s,a) is the environmental comfort level of the target area in the target time period, and w4 is a fourth weight corresponding to the environmental comfort level.

[0051] It is understood that after determining the energy saving level, user satisfaction, stable operation time of the equipment, and environmental comfort, the instant reward value can be calculated according to the first formula, where the first formula can be R(s,a)=E(s,a)×w+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4. Other factors that affect the instant reward value can also be included, such as equipment response speed, safety, convenience, etc. When other factors are included, the first formula can be:

[0052] R(s,a)=E(s,a)×w1+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4+X(s,a)×w5, where X(s,a) represents other factors and w5 represents the weights of other factors.

[0053] Wherein, updating the model parameters according to the instant reward value and the fourth action value includes: determining a characteristic function corresponding to each household appliance performing the first action in the first state; and updating the second action value according to a second formula, wherein the second formula is:

[0054]

[0055] Q(s1,a1) is the updated second action value, Q(s,a) is the second action value, α is the learning rate, R(s,a) is the immediate reward value, γ is the discount factor, max a′ Q(s′,a′) is the fourth action value, β is the regularization coefficient, is the characteristic function, N is the number of the characteristic functions, θ iis the model parameter; and the model parameter is updated according to the updated second action value.

[0056] It is understandable that updating the model parameters according to the fourth action value and the immediate reward value can be achieved by the second formula above, specifically:

[0057] Determine a characteristic function corresponding to each household appliance performing the first action in the first state: identify the characteristic function of performing the first action in the first state.

[0058] According to the formula:

[0059]

[0060] Update the second action value and update the model parameters according to the updated second action value.

[0061] Optionally, after determining the control strategy corresponding to each household appliance according to the strategy model after updating the model parameters in the above step S208, the method further includes: when it is determined that the first energy consumption data corresponding to the current operating mode of the first household appliance in the target area is greater than the preset energy consumption data, and the second energy consumption data corresponding to the current operating mode of the second household appliance in the target area is greater than the preset energy consumption data, predicting the first power demand of the first household appliance and the second power demand of the second household appliance, and determining the current power supply value of the target area; determining the priority of the first household appliance and the second household appliance, and determining the first weight of the first household appliance and the second weight corresponding to the second household appliance according to the priority; modifying the first operating parameter corresponding to the current operating model of the first household appliance and the second operating parameter corresponding to the current operating mode of the second household appliance according to a third formula to obtain the third operating parameter of the modified first operating mode of the first household appliance and the fourth operating parameter corresponding to the modified second operating mode of the second household appliance, wherein the third formula is:

[0062]

[0063] mode a The third operating parameter corresponding to the first household appliance, mode b is the fourth operating parameter corresponding to the second household appliance, w a is the first weight, w b is the second weight, T in is the current temperature of the target area, T desired is the target object's expected temperature for the target area, t b-finishis the estimated completion time of the second household appliance performing the action corresponding to the current operation mode of the second household appliance, t desired-finish is the expected completion time for the target object to perform the action corresponding to the current operation mode of the second home appliance on the second home appliance, P a (t) is the first power requirement, P b (t) is the second power requirement, Pt otal-limit is the current power supply value; adjusting the control strategy corresponding to the first household appliance according to the third operating parameter, and adjusting the control strategy corresponding to the second household appliance according to the fourth operating parameter.

[0064] The first power requirement of the first household appliance and the second power requirement of the second household appliance are predicted, including: predicting the first power requirement according to a fourth formula, wherein the fourth formula is: P a (t) = f a (T in ,T out ,mode a1 ,t),T out is the ambient temperature, mode a1 is the first operating parameter, t is the target time, the target time is the time between the current time and the estimated completion time of the first household appliance performing the action corresponding to the current operating mode of the first household appliance, and the target time is the time between the current time and the estimated completion time of the second household appliance performing the action corresponding to the current operating mode of the second household appliance; the second power demand is predicted according to the fifth formula, wherein the fifth formula is: P b (t) = f b (load b ,mode b1 ,t),load b is the load of the second household appliance, mode b1 is the second operating parameter.

[0065] It is understandable that the first household appliance may be an air conditioner, and the second household appliance may be a washing machine.

[0066] The energy consumption and operating modes of home appliances can be dynamically adjusted, specifically to optimize power distribution in high-energy consumption scenarios while taking into account user expectations for temperature comfort and appliance task completion time.

[0067] Energy consumption determination and power demand forecasting: Energy consumption determination: Detects whether the first and second energy consumption data corresponding to the current operating modes of the first and second appliances (such as an air conditioner) in the target area exceed a preset energy consumption threshold. The preset energy consumption data can be an upper limit determined based on historical data, power supply capacity, and household usage habits.

[0068] Power demand prediction: When it is detected that the energy consumption of both the first and second appliances exceeds a preset value, the power demand of the two appliances is further predicted. The power demand prediction can be based on the current power supply value and the expected future power demand to ensure the appropriate distribution of power.

[0069] Priority determination and weight allocation: Priority determination: Determine the priority of the first home appliance and the second home appliance based on the user's expectations of temperature comfort, home appliance task completion time, and the current status of the device and resource allocation strategy.

[0070] Weight Assignment: Assign weights to the primary and secondary devices based on their priorities. A higher weight means a higher weight is assigned to the device in the control strategy, prioritizing its operation during power shortages.

[0071] Furthermore, the power requirements of the first household appliance and the second household appliance are determined according to the fourth formula and the fifth formula, and the operating parameters of the first household appliance and the second household appliance are modified according to the first power requirement and the second power requirement.

[0072] Through the above embodiments, in high energy consumption scenarios, the operating mode and parameters of home appliances can be dynamically adjusted according to power supply, device priority, and user expectations for temperature and task completion time, thereby achieving efficient, energy-saving and personalized operation of the smart home system.

[0073] In order to better understand the process of the method for determining the above-mentioned control strategy, the implementation method flow of determining the above-mentioned control strategy is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of this application.

[0074] Some existing smart home solutions employ a simple centralized control model, where a central control device sends unified commands to appliances. However, this approach lacks flexibility and adaptability. While some solutions attempt to incorporate machine learning algorithms for optimization, these approaches are often limited to optimizing individual appliances and fail to fully consider the synergy and resource allocation between multiple appliances. Other solutions also lack efficiency in data processing and model training, making them unable to respond to rapid changes in the home environment in real time, resulting in inaccurate and in-time control strategies.

[0075] Existing smart home solutions rely on a simple centralized control model, lacking flexibility and adaptability. This makes them unable to accurately respond to complex and changing home environments and user needs. Because each appliance operates independently or only optimizes a single appliance, without considering synergy and resource allocation, conflicts and resource waste are common. Furthermore, inefficient data processing and model training prevent real-time response to environmental changes, resulting in inaccurate and timely control strategies that fail to fully improve energy efficiency and user experience.

[0076] In response to the above problems, the optional embodiment of the present application proposes a smart home collaborative control method based on multi-agent self-learning. Specifically: The technical problem to be solved by the optional embodiment of the present application is to achieve efficient collaborative self-learning between multiple smart home appliances and the central control screen in the smart home, solve the shortcomings of the existing smart home system in terms of device collaboration and self-learning capabilities, avoid conflicts in the operation of home appliances and waste of resources, optimize control strategies and operation effects, improve energy utilization efficiency, enhance user experience, and provide users with a smarter, energy-saving and comfortable home environment. The purpose of the optional embodiment of the present application is to overcome the shortcomings of the existing technology in terms of lack of flexibility, adaptability, collaboration and efficiency by constructing a smart home collaborative control system based on multi-agent self-learning. Achieve efficient collaborative self-learning between multiple smart home appliances and the central control screen, accurately adapt to the dynamic changes of the home environment and user needs, avoid conflicts in the operation of home appliances and waste of resources, optimize energy utilization efficiency, and provide a smarter, energy-saving and comfortable home experience.

[0077] Figure 3 The central control screen and multiple smart home appliances (i.e. Figure 3 Schematic diagram of the interaction process between smart home appliance 1 and smart home appliance 2 (in actual application, multiple smart home appliances may also be included, such as smart home appliance 3, smart home appliance 4, etc.), as shown in FIG. Figure 3 As shown:

[0078] Build a deep learning-based agent model for each smart appliance (e.g., refrigerator, air conditioner, washing machine, etc.) and central control screen:

[0079] 1) Input layer:

[0080] Smart home appliance 1 and home appliance 2 (appliances 1 and 2 can be refrigerators, air conditioners, washing machines, etc.): Refrigerator input data includes the current refrigerator and freezer compartment temperatures, set temperature, door opening frequency, cooling mode (such as quick freeze, energy saving, etc.), and energy consumption data. Air conditioner input includes indoor and outdoor temperatures, set temperature, fan speed mode, operating mode (cooling, heating, dehumidification, etc.), and energy consumption information. Washing machine input includes wash mode (standard wash, quick wash, strong wash, etc.), clothing weight, water level selection, operation phase (wash, rinse, spin), and energy consumption.

[0081] Central control screen: Receives the above data from all home appliances, and obtains operation instructions input by users through mobile phone applications, remote controls or voice, such as setting the working time of specific home appliances, preferred energy-saving mode, desired indoor environment parameters, etc.

[0082] 2) Hidden layer:

[0083] Both use Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) units, and their number is dynamically determined based on data complexity and computing resources.

[0084] For refrigerators with complex temperature control and freshness preservation requirements, more LSTM or GRU units may be needed to process time series data to accurately predict temperature changes and energy consumption optimization strategies.

[0085] For air conditioning, considering the rapid changes in indoor and outdoor temperatures and users' different requirements for comfort, the data complexity is relatively high, and it may be necessary to appropriately increase the number of hidden units to capture these dynamic features.

[0086] For washing machines, their operating modes are relatively fixed, but they still need to be optimized based on variables such as clothing type and weight. The amount of data is moderate, and the number of hidden units can be reasonably configured based on actual computing resources.

[0087] As the coordination and control center, the central control screen needs to process a large amount of diverse data from different home appliances. Therefore, more hidden units are usually required to integrate and analyze this information to generate accurate and effective control strategies.

[0088] 3) Output layer:

[0089] Home appliances:

[0090] The output of a refrigerator might be instructions to adjust cooling power, change freshness mode, or optimize the defrost cycle.

[0091] The output of an air conditioner might be a parameter that adjusts the temperature setting, changes the fan speed, or switches operating modes.

[0092] The output of a washing machine might be instructions to adjust the wash time, change the water level, or optimize the spin speed.

[0093] Central control screen: Generates specific control strategies for each home appliance, such as controlling the refrigerator to enter energy-saving mode during specific time periods, the air conditioner to dynamically adjust the temperature and wind speed based on the number of people in the room, and the washing machine to select the appropriate washing program based on the material of the clothes, etc., to achieve coordinated optimization and efficient operation of the entire smart home system.

[0094] In addition, the control method for household appliances in the optional embodiments of the present application may also include: a rule-based centralized control system: controlling the operation of household appliances by pre-setting a series of detailed rules. For example, the operation mode and parameters of household appliances are determined according to time, environmental conditions and user-preset priorities. Distributed intelligent control: each household appliance has a certain intelligent decision-making ability, and is autonomously controlled through local sensors and simple algorithms, and simple coordination is carried out between household appliances through local communication. Control scheme based on the Internet of Things cloud platform: uploading household appliance data to the cloud platform, analyzing and making decisions in the cloud, and then sending control instructions to the household appliances. Control scheme based on fuzzy logic: using fuzzy logic to process uncertain and imprecise information to control household appliances. Single intelligent agent control based on reinforcement learning: relying only on the central control screen as a single intelligent agent to learn and make decisions, control all household appliances, etc. The above control schemes for household appliances are all within the protection scope of this application.

[0095] Furthermore, each agent has a self-learning mechanism, specifically:

[0096] (1) Data collection and preprocessing: Each home appliance and central control panel collects data at an extremely fine time granularity (e.g., every second, every millisecond). The collected data includes not only common operating parameters but also the microscopic operating status data of key components within the device. For example, the instantaneous power fluctuation of the refrigerator compressor, the fine-tuning change of the air conditioner fan speed, and the instantaneous torque value of the washing machine motor.

[0097] Data standardization: Utilize sophisticated statistical methods to convert heterogeneous data into a standardized form with uniform dimensions and distribution characteristics. For power data, a nonlinear normalization function based on device characteristics is used to more accurately reflect its practical significance.

[0098] Data normalization: Use advanced normalization techniques, such as kernel density estimation-based normalization methods, to ensure that the distribution of data in the [0, 1] interval is more consistent with the actual situation while retaining the complex characteristics of the original data.

[0099] Missing value processing: For missing data, complex interpolation methods based on time series forecasting models (such as ARIMA and LSTM) are used to fill in the missing data. At the same time, the accuracy and reliability of the filled values ​​are evaluated through multiple validation and cross-validation.

[0100] (2) Central control screen learning:

[0101] Reinforcement learning algorithms, such as deep Q-learning combined with multi-layer neural networks, are applied to approximate the Q-value function. The state space is constructed as an ultra-high-dimensional feature vector, encompassing not only the current and historical operating parameters of the appliance, but also the time series patterns of user behavior and the complex dynamics of environmental variables (such as the impact of seasonality, weather, and holidays on appliance usage). The action space is meticulously divided into highly specific control command combinations for each appliance, including multi-level parameter adjustments and complex mode switching options.

[0102] Design the reward function (i.e. the first formula):

[0103] R(s,a) = E(s,a) × w1 + S(s,a) × w2 + D(s,a) × w3 + C(s,a) × w4, where: Energy saving degree E is calculated by real-time monitoring and integration of the device's instantaneous energy consumption, and compared with the optimal energy consumption curve predicted based on a complex physical model to accurately quantify the energy saving effect. User satisfaction S is evaluated in depth using sentiment analysis and semantic understanding technologies based on detailed user feedback (such as specific evaluations of different temperatures, humidity levels, and washing effects). Device stable operating time D comprehensively measures the device's stable operating capability by combining real-time fault monitoring data, a key component life prediction model based on reliability theory, and stability assessment indicators under complex operating conditions. New factors, such as environmental comfort C, are added. This factor uses indoor environmental data (including air quality and noise levels) collected by multiple sensors and calculates its impact on the user experience based on a human comfort model. Other factors include device response speed, safety, convenience, and other factors (X). Each weight w1-w5 is dynamically optimized and adjusted through machine learning algorithms based on large-scale data (such as random forests and gradient boosting trees) to adapt to the complex needs of different users and environments.

[0104] After determining the reward function, the Q value can be updated. Specifically, the Q value update formula (i.e. the second formula) is used:

[0105]

[0106] Among them, the new policy network parameters θ and feature functions are added The regularization term is used to prevent overfitting and unstable updates of the Q value. The learning rate α and the regularization coefficient β are dynamically optimized through a gradient-based adaptive adjustment algorithm to ensure efficient and stable convergence of the learning process.

[0107] (3) Local learning of home appliances:

[0108] In the local learning of each home appliance, the policy gradient algorithm is used for optimization based on the control strategy and feedback information received from the central control screen, combined with the high-precision data collected in real time.

[0109] Strategy function:

[0110] Among them, π(a|s): policy function, which represents the probability of taking action a in a given state s. In the above formula, πθ(a t |s t ) indicates that the agent (e.g., home appliance or central control screen) is in state s t Next, select action a according to the current parameter θ t probability.

[0111] θ: Parameters of the policy function, which need to be optimized through the learning process.

[0112] J(θ): The expected cumulative reward of the policy, which is a function of the policy parameters θ. J(θ) is maximized by optimizing θ.

[0113] The gradient of the policy function J(θ) with respect to the parameters θ indicates how changes in the parameters θ affect the expected cumulative reward.

[0114] E τ～pθ(τ) : Expectation operator, τ represents a complete trajectory, which includes a series of states, actions and rewards; pθ(τ) is the probability distribution of generating trajectory τ according to the policy parameters θ.

[0115] R(τ): The reward based on the trajectory τ, which can be a single-step reward or a cumulative reward, is used to evaluate the quality of the strategy.

[0116] a t and s t : are the action and state at time t in the trajectory τ respectively.

[0117] In order to calculate the gradient more efficiently, we can use the automatic differentiation technology to accurately solve At the same time, a variant algorithm based on stochastic gradient descent is introduced to adaptively adjust the learning step size according to the historical information of the gradient to speed up the convergence and avoid falling into the local optimum.

[0118] In addition, model pruning and quantization techniques can be used to compress the size of the policy model and reduce computational complexity without significantly losing performance, making it more suitable for the limited local computing resources of home appliances.

[0119] Through the above series of strategy optimization technologies, home appliances can achieve efficient and accurate learning locally, continuously improving their operating efficiency and responsiveness to user needs.

[0120] In addition, the models corresponding to home appliances and central control screens can also be automatically updated. Specifically:

[0121] Dynamic update triggering: The model update process for central control screens and home appliances relies not only on fixed time intervals or data change thresholds, but is based on a complex event-driven mechanism. For example, when a significant change in user behavior patterns (such as returning home after a long absence, changes in family composition, etc.), a dramatic fluctuation in environmental conditions (such as extreme weather conditions lasting for several days), or an early warning signal of potential equipment failure is detected, model retraining and updates are immediately triggered.

[0122] Parameter Update Strategy: When updating model parameters, we use a hyperparameter adjustment method based on Bayesian optimization, combined with techniques based on model compression and knowledge distillation, to compress large pre-trained models into lightweight models suitable for local computing resources in home appliances while retaining key learning capabilities. Furthermore, we introduce a generative adversarial network to simulate the distribution of new data to assist in model updates and improve generalization capabilities.

[0123] Collaborative Synchronization Mechanism: When the central control panel synchronizes updated control strategies with home appliances, it utilizes encrypted communication protocols and blockchain technology to ensure data security, integrity, and immutability. Upon receiving the new strategy, the appliances perform pre-execution and effectiveness evaluation using a reinforcement learning-based verification and simulation mechanism, and provide encrypted feedback to the central control panel. Based on this feedback, the central control panel further optimizes and adjusts the control strategy using a distributed consensus algorithm, creating a highly secure, reliable, and efficient closed-loop collaborative optimization system.

[0124] Through the above extremely complex and sophisticated self-learning mechanism, ultra-precise control and optimization of the smart home system can be achieved, which can meet user needs to the greatest extent, improve energy efficiency, enhance device stability and improve the overall user experience.

[0125] Furthermore, in a smart home system, collaboration between multiple appliances is crucial. This collaboration is achieved through coordination and information sharing on the central control screen.

[0126] When the central control panel generates control strategies, it applies complex calculation and analysis methods, taking into account multiple factors, including the real-time operating status of each appliance, expected demand, and resource constraints. It not only considers the appliance's current operating mode and energy consumption level, but also predicts how its status will change over time.

[0127] For example, when the air conditioner requires high power and the washing machine is also in high-energy mode, the central control screen will first obtain the current power supply status, including total power supply, real-time power fluctuations, and peak power limits. It will also analyze the user's set priorities, such as whether the user prefers to prioritize maintaining a comfortable indoor temperature or completing the laundry task as quickly as possible.

[0128] The central control screen then performs precise calculations based on this information. By building an energy consumption model, it predicts the energy consumption of the air conditioner and washing machine under different operating parameter combinations. Simultaneously, it uses optimization algorithms, such as linear programming or genetic algorithms, to search for the optimal operating parameter adjustment plan, minimizing overall energy consumption and rationally allocating resources while still meeting user needs.

[0129] Specific calculations may include:

[0130] Assume that the power demand function of the air conditioner (i.e. the first household appliance) is P a (t) = f a (T in ,T out ,mode a1 ,t)(i.e. the fourth formula); the power demand function of the washing machine (i.e. the second household appliance) is P b (t) = f b (load b ,mode b1 ,t)(i.e. the fifth formula).

[0131] The total power supply is limited to Pt otal-limit , the user's demand weight for air conditioning comfort is w a , the demand weight for laundry completion time is w b .

[0132] Based on the above power demand function and the above parameters, the optimization objective (i.e., the third formula) can be determined as:

[0133]

[0134] By solving the above optimization problems, the central control screen can reasonably adjust the operating parameters of the air conditioner and washing machine, such as the temperature setting and wind speed of the air conditioner, and the washing time and speed of the washing machine, so as to achieve a balance between overall energy consumption optimization and user experience.

[0135] Through the above-mentioned sophisticated collaborative calculation and strategy adjustment, efficient collaborative self-learning is achieved between the central control screen and smart home appliances, continuously optimizing the control strategy and operation effect of the smart home, and providing users with a more intelligent, energy-saving and comfortable home experience.

[0136] In summary, an optional embodiment of the present application constructs a deep learning-based agent model for each smart home appliance and central control screen, using LSTM or GRU units, and determining the number of hidden layer units based on data complexity and computing resources. Furthermore, an optional embodiment of the present application designs a sophisticated self-learning mechanism, including data collection and preprocessing, learning algorithms for the central control screen and home appliances, and model update strategies. Furthermore, an optional embodiment of the present application enables efficient collaboration between multiple home appliances through the central control screen, rationally allocating resources and avoiding conflicts through complex computation and optimization algorithms. In other words, an optional embodiment of the present application can accurately capture and process each appliance's unique data characteristics and needs, thereby providing more personalized and precise control strategies. Furthermore, because of the sophisticated self-learning mechanism, including efficient data collection and preprocessing, advanced learning algorithms, and a reasonable model update strategy, it can quickly adapt to changes in the home environment and user needs, continuously optimize the control strategy, and enhance the system's intelligence and adaptability. Furthermore, by enabling efficient collaboration between multiple home appliances through the central control screen and employing complex computation and optimization algorithms to rationally allocate resources and avoid conflicts, it can minimize appliance operation conflicts and resource waste, improve energy efficiency, and reduce operating costs. It also provides users with a more stable, comfortable and convenient smart home experience, meeting users' needs for an intelligent, energy-saving and comfortable home environment. In summary, the optional embodiments of this application have the advantages of personalized and precise control, intelligent adaptation to changes, high efficiency and energy saving, and high-quality user experience.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0138] Figure 4 is a structural block diagram of a device for determining a control strategy according to an embodiment of the present application; Figure 4 As shown, including:

[0139] A first constructing module 42 is configured to construct a state space of multiple home appliances in the target area based on the state data of each home appliance in the target area, the operation data of the target object on each home appliance, and the environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance;

[0140] A second constructing module 44 is configured to construct an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform;

[0141] a first determining module 46 configured to determine, according to a target algorithm, a first action value for each household appliance in a first state in the state space to perform each action in the action space, and to update model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space;

[0142] The second determination module 48 is used to determine the preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and send each preliminary control strategy to each household appliance so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0143] Through the above-mentioned device, a state space of the target area including a set of multiple operating states of each household appliance is constructed based on the status data of each household appliance in the target area, the operation data of each household appliance on the target object and the environmental data of the target area; an action space including a set of multiple actions allowed to be performed by each household appliance is constructed; the first action value of each action in the action space executed by each household appliance in any state in the state space is determined according to the deep learning algorithm, and the model parameters of the strategy model are updated according to the multiple first action values; the preliminary control strategy corresponding to each household appliance is determined according to the strategy model after the model parameters are updated, and each preliminary control strategy is sent to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance according to the preliminary control strategy and the real-time status data of each household appliance. That is to say, the state of the embodiment of the present application includes the operating state of multiple home appliances, and the action space includes actions that are allowed to be executed by multiple home appliances. Then, by determining the first action value of each action in the action space executed by each home appliance in any state in the state space to update the model parameters of the strategy model, the preliminary control strategy determined according to the strategy model can take into account the collaborative relationship and resource allocation problems between multiple home appliances, and then send the preliminary control strategy to each home appliance, so that each home appliance can further determine the target control strategy of each home appliance according to the preliminary control strategy and its actual needs, so that the target control strategy finally corresponding to each home appliance takes into account the collaborative relationship of multiple home appliances and meets the flexibility and adaptability of control strategy formulation. Through the embodiment of the present application, it is possible to solve the problem that the home appliance control method in the related art does not take into account the collaborative relationship and resource allocation between multiple home appliances, resulting in operation conflicts and uneven resource allocation among multiple home appliances.

[0144] In an exemplary embodiment, the first determination module 46 is further used to determine a second action value among the multiple first action values, and determine the first action corresponding to the second action value, wherein the second action value is the maximum value among the multiple first action values; calculate the immediate reward value of each household appliance performing the first action in the first state; determine the third action value of each household appliance performing each action in the next state after the first state according to the target algorithm, and determine a fourth action value among the multiple third action values, wherein the next state is other states in the state space except the first state; and update the model parameters according to the immediate reward value and the fourth action value.

[0145] In an exemplary embodiment, the first determination module 46 is further used to obtain the instantaneous energy consumption of each household appliance when performing the first action in the first state, and determine the energy saving degree of each household appliance when performing the first action in the first state based on the instantaneous energy consumption; determine the feedback data of the target object on the performance of the first action by each household appliance in the first state, and determine the user satisfaction of each household appliance when performing the first action in the first state based on the feedback data; determine the stable operation time of the device when each household appliance performs the first action in the first state and the environmental comfort of the target area in the target time period, wherein the target time period is the time period when each household appliance performs the first action in the first state; determine the instant reward value based on the energy saving degree, the user satisfaction, the stable operation time of the device, and the environmental comfort.

[0146] In an exemplary embodiment, the first determination module 46 is further used to determine the immediate reward value R(s,a) according to a first formula, wherein the first formula is: R(s,a)=E(s,a)×w1+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4, s is the first state, a is the first action, E(s,a) is the energy saving degree of each household appliance performing the first action in the first state, w1 is the first weight corresponding to the energy saving degree, S(s,a) is the user satisfaction of each household appliance performing the first action in the first state, w2 is the second weight corresponding to the user satisfaction, D(s,a) is the stable operation time of each household appliance in the first state, w3 is the third weight corresponding to the stable operation time, C(s,a) is the environmental comfort of the target area in the target time period, and w4 is the fourth weight corresponding to the environmental comfort.

[0147] In an exemplary embodiment, the first determining module 46 is further configured to determine a characteristic function corresponding to each household appliance performing the first action in the first state; and update the second action value according to a second formula, wherein the second formula is:

[0148] is the updated second action value, Q(s,a) is the second action value, α is the learning rate, R(s,a) is the immediate reward value, γ is the discount factor, max a′ Q(s′,a′) is the fourth action value, β is the regularization coefficient, is the characteristic function, N is the number of the characteristic functions, θ i is the model parameter; and the model parameter is updated according to the updated second action value.

[0149] In an exemplary embodiment, the second determination module 48 is further configured to, when it is determined that the first energy consumption data corresponding to the current operating mode of the first household appliance in the target area is greater than the preset energy consumption data, and the second energy consumption data corresponding to the current operating mode of the second household appliance in the target area is greater than the preset energy consumption data, predict the first power demand of the first household appliance and the second power demand of the second household appliance, and determine the current power supply value of the target area; determine the priorities of the first household appliance and the second household appliance, and determine the first weight of the first household appliance and the second weight corresponding to the second household appliance based on the priorities; and modify the first operating parameter corresponding to the current operating mode of the first household appliance and the second operating parameter corresponding to the current operating mode of the second household appliance according to a third formula to obtain the third operating parameter of the modified first operating mode of the first household appliance and the fourth operating parameter corresponding to the modified second operating mode of the second household appliance, wherein the third formula is:

[0150] mode a The third operating parameter corresponding to the first household appliance, mode b is the fourth operating parameter corresponding to the second household appliance, w a is the first weight, w b is the second weight, T in is the current temperature of the target area, T desired is the target object's expected temperature for the target area, t b-finish is the estimated completion time of the second household appliance performing the action corresponding to the current operation mode of the second household appliance, t desired-finishis the expected completion time of the target object performing the action corresponding to the current operation mode of the second home appliance on the second home appliance, P a (t) is the first power requirement, P b (t) is the second power requirement, P total-limit is the current power supply value; adjusting the control strategy corresponding to the first household appliance according to the third operating parameter, and adjusting the control strategy corresponding to the second household appliance according to the fourth operating parameter.

[0151] In an exemplary embodiment, the second determining module 48 is further configured to predict the first power requirement according to a fourth formula, wherein the fourth formula is: P a (t) = f a (T in ,T out ,mode a1 ,t),T out is the ambient temperature, mode a1 is the first operating parameter, t is the target time, the target time is the time between the current time and the estimated completion time of the first household appliance performing the action corresponding to the current operating mode of the first household appliance, and the target time is the time between the current time and the estimated completion time of the second household appliance performing the action corresponding to the current operating mode of the second household appliance; the second power demand is predicted according to the fifth formula, wherein the fifth formula is: P b (t) = f b (load b ,mode b1 ,t),load b is the load of the second household appliance, mode b1 is the second operating parameter.

[0152] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0153] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0154] S1, constructing a state space of multiple home appliances in a target area based on state data of each home appliance in a target area, operation data of each home appliance on which a target object operates, and environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance;

[0155] S2, constructing an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform;

[0156] S3, determining, according to a target algorithm, a first action value for each action in the action space executed by each household appliance in a first state in the state space, and updating model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space;

[0157] S4, determining a preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and sending each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0158] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0159] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0160] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0161] S1, constructing a state space of multiple home appliances in a target area based on state data of each home appliance in a target area, operation data of each home appliance on which a target object operates, and environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance;

[0162] S2, constructing an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform;

[0163] S3, determining, according to a target algorithm, a first action value for each action in the action space executed by each household appliance in a first state in the state space, and updating model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space;

[0164] S4, determining a preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and sending each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0165] An embodiment of the present application further provides a computer program product, including a computer program, which is used by a processor to execute the steps in any of the above method embodiments.

[0166] Optionally, in this embodiment, the computer program product may be processed by a processor to execute the following steps:

[0167] S1, constructing a state space of multiple home appliances in a target area based on state data of each home appliance in a target area, operation data of each home appliance on which a target object operates, and environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance;

[0168] S2, constructing an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform;

[0169] S3, determining, according to a target algorithm, a first action value for each action in the action space executed by each household appliance in a first state in the state space, and updating model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space;

[0170] S4, determining a preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and sending each preliminary control strategy to each household appliance, so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

[0171] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0172] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0173] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into separate integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0174] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a control strategy, characterized in that: include: Constructing a state space of multiple home appliances in the target area based on the state data of each home appliance in the target area, the operation data of the target object on each home appliance, and the environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance; Constructing an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform; Determine, according to a target algorithm, a first action value for each action in the action space to be executed by each household appliance in a first state in the state space, and update model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space; Determine the preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and send each preliminary control strategy to each household appliance so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

2. The method for determining a control strategy according to claim 1, wherein: The model parameters of the policy model are updated according to the multiple first action values, including: Determining a second action value from the plurality of first action values, and determining a first action corresponding to the second action value, wherein the second action value is a maximum value among the plurality of first action values; Calculating an immediate reward value for each household appliance performing the first action in the first state; determining, according to the target algorithm, a third action value for each household appliance performing each action in a state next to the first state, and determining a fourth action value from the plurality of third action values, wherein the next state is a state other than the first state in the state space; The model parameters are updated according to the immediate reward value and the fourth action value.

3. The method for determining a control strategy according to claim 2, wherein: Calculating an instant reward value for each household appliance performing the first action in the first state includes: Obtaining instantaneous energy consumption of each household appliance when performing the first action in the first state, and determining an energy saving degree of each household appliance when performing the first action in the first state according to the instantaneous energy consumption; Determining feedback data from the target object regarding each home appliance performing the first action in the first state, and determining user satisfaction with each home appliance performing the first action in the first state based on the feedback data; Determining a stable operation time of each household appliance performing the first action in the first state and an environmental comfort level of the target area in a target time period, wherein the target time period is a time period during which each household appliance performs the first action in the first state; The instant reward value is determined according to the energy saving degree, the user satisfaction, the stable operation time of the equipment and the environmental comfort.

4. The method for determining a control strategy according to claim 3, wherein: Determining the instant reward value according to the energy saving degree, the user satisfaction, the stable operation time of the equipment, and the environmental comfort level includes: The instant reward value R(s,a) is determined according to a first formula, wherein the first formula is: R(s,a)=E(s,a)×w1+S(s,a)×w2+D(s,a)×w3+C(s,a)×w4, s is the first state, a is the first action, E(s,a) is the energy saving degree of each household appliance performing the first action in the first state, w1 is the first weight corresponding to the energy saving degree, S(s,a) is the user satisfaction of each household appliance performing the first action in the first state, w2 is the second weight corresponding to the user satisfaction, D(s,a) is the stable operation time of each household appliance in the first state, w3 is the third weight corresponding to the stable operation time of the appliance, C(s,a) is the environmental comfort of the target area in the target time period, and w4 is the fourth weight corresponding to the environmental comfort.

5. The method for determining a control strategy according to claim 2, wherein: Updating the model parameters according to the immediate reward value and the fourth action value includes: Determine a characteristic function corresponding to each household appliance performing the first action in the first state; The second action value is updated according to a second formula, wherein the second formula is: Q(s1,a1) is the updated second action value, Q(s,a) is the second action value, α is the learning rate, R(s,a) is the immediate reward value, γ is the discount factor, max a′ Q(s′,a′) is the fourth action value, β is the regularization coefficient, is the characteristic function, N is the number of the characteristic functions, θ i are the model parameters; The model parameters are updated according to the updated second action value.

6. The method for determining a control strategy according to claim 1, wherein: After determining the control strategy corresponding to each household appliance according to the strategy model after updating the model parameters, the method further includes: When it is determined that first energy consumption data corresponding to the current operating mode of a first household appliance in the target area is greater than a preset energy consumption data, and second energy consumption data corresponding to the current operating mode of a second household appliance in the target area is greater than the preset energy consumption data, predicting a first power demand of the first household appliance and a second power demand of the second household appliance, and determining a current power supply value of the target area; Determining priorities of the first home appliance and the second home appliance, and determining a first weight of the first home appliance and a second weight corresponding to the second home appliance according to the priorities; The first operating parameter corresponding to the current operating mode of the first household appliance and the second operating parameter corresponding to the current operating mode of the second household appliance are modified according to a third formula to obtain a third operating parameter of the modified first operating mode of the first household appliance and a fourth operating parameter corresponding to the modified second operating mode of the second household appliance, wherein the third formula is: mode a The third operating parameter corresponding to the first household appliance, mode b is the fourth operating parameter corresponding to the second household appliance, w a is the first weight, w b is the second weight, T in is the current temperature of the target area, T desired is the target object's expected temperature for the target area, t b-finish is the estimated completion time of the second household appliance performing the action corresponding to the current operation mode of the second household appliance, t desired-finish is the expected completion time of the target object performing the action corresponding to the current operation mode of the second home appliance on the second home appliance, P a (t) is the first power requirement, P b (t) is the second power requirement, Pt otal-limit is the current power supply value; The control strategy corresponding to the first household appliance is adjusted according to the third operating parameter, and the control strategy corresponding to the second household appliance is adjusted according to the fourth operating parameter.

7. The method for determining a control strategy according to claim 6, wherein: Predicting a first power requirement of the first household appliance and a second power requirement of the second household appliance includes: The first power requirement is predicted according to a fourth formula, wherein the fourth formula is: a (t) = f a (T in ,T out ,mode a1 ,t),T out is the ambient temperature, mode a1 is the first operating parameter, t is the target time, the target time is the time between the current time and the estimated completion time for the first household appliance to perform the action corresponding to the current operating mode of the first household appliance, and the target time is the time between the current time and the estimated completion time for the second household appliance to perform the action corresponding to the current operating mode of the second household appliance; The second power requirement is predicted according to the fifth formula, wherein the fifth formula is: b (t) = f b (load b ,mode b1 ,t),load b is the load of the second household appliance, mode e b1 is the second operating parameter.

8. A device for determining a control strategy, characterized in that: include: A first constructing module is configured to construct a state space of multiple home appliances in the target area based on the state data of each home appliance in the target area, the operation data of the target object on each home appliance, and the environmental data of the target area, wherein the state space includes: a set of multiple operating states of each home appliance; A second building module is configured to build an action space corresponding to the plurality of home appliances, wherein the action space includes: a set of multiple actions that each home appliance is allowed to perform; a first determining module, configured to determine, according to a target algorithm, a first action value of each action in the action space performed by each household appliance in a first state in the state space, and update model parameters of a strategy model according to the plurality of first action values, wherein the first state is any state in the state space; The second determination module is used to determine the preliminary control strategy corresponding to each household appliance based on the strategy model after updating the model parameters, and send each preliminary control strategy to each household appliance so that each household appliance determines the target control strategy corresponding to each household appliance based on the preliminary control strategy and the real-time status data of each household appliance.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the method according to any one of claims 1 to 7 is executed when the program is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images