Air conditioning control strategy generation method and system

Abstract

The present application relates to the technical field of building energy, and in particular to a method and system for generating an air conditioning control strategy, which includes: obtaining current environmental parameters and time information; drawing a thermal comfort curve based on time information and environmental parameters; obtaining a reward range including a maximum temperature value and a minimum temperature value; initializing a deep neural network, performing preliminary training, judging whether the new state after the action is within the reward range to generate a corresponding first reward or penalty, and if there is a penalty, updating the parameters of the deep neural network based on the penalty; introducing user feedback information, and performing advanced training, obtaining user feedback information based on the action with the largest first reward to enter a new state to generate a corresponding second reward; after completion, inputting the current temperature as the state, and adjusting the air conditioner according to the action corresponding to the highest final reward. The present application has the effect of improving the accuracy of air conditioning reinforcement learning when training data is lacking.

Description

Technical Field

[0001] The present application relates to the technical field of building energy, and in particular to a method and system for generating an air conditioning control strategy. Background Art

[0002] In building energy systems, heating, ventilation and air-conditioning systems are the main source of energy consumption in buildings. Optimizing these systems is not an easy task. It requires comprehensive consideration of the randomness of environmental conditions, dynamically changing thermal comfort needs, and diverse building envelope characteristics. The intelligent air-conditioning control system based on reinforcement learning can intelligently generate air-conditioning instructions while satisfying user comfort by establishing an environmental perception model and a user preference model, thereby reducing energy consumption and achieving the goal of quality-energy balance.

[0003] There are methods in the related art to generate air conditioning control strategies through reinforcement learning, but usually reinforcement learning requires a large amount of data to ensure the learning effect and the generalization ability of the model. However, in the real life construction field, it is not possible to collect a large amount of data to train reinforcement learning. At the same time, the amount of reinforcement learning data when the intelligent air conditioning control system is just running cannot support the training of the intelligent model, which will cause the model to output wrong strategies. At the same time, the reinforcement learning in the related art can only train and learn the same building model, and the same model cannot be applied to different buildings. In addition, because different seasons have different strategies for air conditioning energy consumption and comfort, it cannot be applied to different seasons. Summary of the invention

[0004] In order to improve the accuracy of air conditioning reinforcement learning when training data is lacking, the present application provides an air conditioning control strategy generation method and system.

[0005] In a first aspect, the present application provides a method for generating an air conditioning control strategy, which adopts the following technical solution:

[0006] A method for generating an air conditioning control strategy comprises the following steps:

[0007] Get current environmental parameters and time information;

[0008] Selecting a corresponding thermal comfort standard in a preset season template based on the time information, analyzing the environmental parameters according to the thermal comfort standard to draw a corresponding thermal comfort curve, wherein the thermal comfort curve includes a time corresponding to an x-axis and an air-conditioning temperature corresponding to a y-axis;

[0009] Acquire thermal comfort experience results and energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and add the reward range to the thermal comfort curve;

[0010] Initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate a corresponding first reward or penalty, and if there is a penalty, update the parameters of the deep neural network based on the penalty until a preset termination condition is reached to end the preliminary training of the deep reinforcement learning model, wherein the agent is an air conditioner, the action is to adjust the temperature of the air conditioner, and the input state is set to be the temperature of the air conditioner;

[0011] After the initial training is completed, user feedback information is introduced, and the deep reinforcement learning model corresponding to the agent is advanced trained through reinforcement learning, and the user feedback information based on the action with the largest first reward entering a new state is obtained to generate a corresponding second reward, until a preset termination condition is reached to end the advanced training of the deep reinforcement learning model;

[0012] After the deep reinforcement learning model training is completed, the current temperature is input as the state, and the air conditioner is adjusted according to the action corresponding to the highest final reward.

[0013] In some embodiments, selecting a corresponding thermal comfort standard in a preset season template based on the time information, analyzing the environmental parameters by the thermal comfort standard to draw a corresponding thermal comfort curve includes the following steps:

[0014] Determine the current season according to the time information to select a summer template, a winter template or a transitional season template, and obtain the corresponding thermal comfort standard, which is characterized by the 24-hour air-conditioning comfort temperature corresponding to different temperature ranges in the season;

[0015] The temperature information in the environmental parameters is obtained, and the air-conditioning comfort temperature of each time period is selected according to the temperature range matched by the temperature information and a corresponding thermal comfort curve is drawn, wherein the thermal comfort curve can be manually modified.

[0016] In some embodiments, obtaining the thermal comfort experience result and the energy consumption experience result to generate a reward range including a maximum temperature value and a minimum temperature value includes the following steps:

[0017] According to the air conditioner usage survey data combined with the empirical algorithm, a first range including the lowest ideal body temperature and the highest ideal body temperature under different ambient temperatures is obtained;

[0018] Acquire a second range including the lowest ideal energy consumption temperature and the highest ideal energy consumption temperature at different ambient temperatures according to the policy data combined with an empirical algorithm;

[0019] If the selected season template is a summer template, the lower value of the highest ideal body temperature and the highest ideal energy consumption temperature is used as the maximum temperature value, and the lowest ideal energy consumption temperature is used as the minimum temperature value;

[0020] If the selected season template is a winter template, the higher value of the lowest ideal body temperature and the lowest ideal energy consumption temperature is used as the minimum temperature value, and the highest ideal energy consumption temperature is used as the maximum temperature value;

[0021] If the selected season template is a transition season template, the union of the first range and the second range is used as the reward range.

[0022] In some embodiments, generating a random action based on a target state, and determining whether a new state after the action is within the reward range to generate a corresponding first reward or penalty includes the following steps:

[0023] generating the target state based on the air conditioning temperature on the thermal comfort curve corresponding to the current time;

[0024] Input the current air-conditioning temperature as the current state, and predict the Q values ​​corresponding to all possible actions through a deep neural network based on the current state and the target state;

[0025] An action corresponding to the highest Q value is executed to enter a new state;

[0026] If the new state is within the reward range, a first reward is obtained based on the convergence between the new state and the target state, wherein if convergence occurs, the first reward is a positive number, if discrete, the first reward is a negative number, and if discreteness remains unchanged, the first reward is 0;

[0027] If the new state is outside the reward range, a penalty is generated.

[0028] In some embodiments, if the new state is within the reward range, obtaining a corresponding first reward based on the convergence between the new state after the random action and the target state further includes the following steps:

[0029] Acquire the user's usage habit information of the air conditioner, and generate a number of sweet spots based on the usage habit information;

[0030] A corresponding point position is found in the thermal comfort curve according to the sweet spot, and the first reward of the target state corresponding to the point position is increased.

[0031] In some embodiments, obtaining the user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward includes the following steps:

[0032] Waiting for the user feedback information to exist in the new state, where the user feedback information is characterized by the user manually adjusting the air-conditioning temperature upward or downward;

[0033] If it does not exist, the generated second reward is a positive number;

[0034] If it exists, the absolute value of the difference between the air-conditioning temperature corresponding to the new state and the air-conditioning temperature corresponding to the user feedback information is calculated and defined as the feedback evaluation result, and the second reward of the corresponding negative number is generated according to the feedback evaluation result. If the value of the feedback evaluation result is larger, the value of the second reward is smaller.

[0035] In some embodiments, obtaining the user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward further includes the following steps:

[0036] Determine the air conditioning temperature value distribution corresponding to the current state, the new state and the feedback evaluation result;

[0037] Generate a same-direction action reward and a reverse action reward according to the numerical distribution of the air-conditioning temperature, wherein in the same temperature adjustment unit, the same-direction action reward is greater than the reverse action reward;

[0038] The same-direction action reward is characterized by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on the same side of the value of the new state, and the reverse action reward is characterized by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on both sides of the value of the new state.

[0039] In some of the embodiments, after generating the corresponding second reward, the following steps are further included:

[0040] Determining whether the generation time of the user feedback information corresponds to the sweet spot;

[0041] If so, the second reward for the air-conditioning temperature corresponding to the user feedback information is reduced.

[0042] In some embodiments, the final reward is the sum of the first reward and the second reward corresponding to the action, wherein the absolute value of the second reward corresponding to the same action is greater than the absolute value of the first reward.

[0043] In a second aspect, the present application provides an air conditioning control strategy generation system, which adopts the following technical solution:

[0044] An air conditioning control strategy generation system, comprising:

[0045] Internet of Things module, used to obtain current environmental parameters and time information;

[0046] a thermal comfort curve drawing module, configured to select a corresponding thermal comfort standard in a preset season template based on the time information, analyze the environmental parameters according to the thermal comfort standard to draw a corresponding thermal comfort curve, wherein the thermal comfort curve includes a time corresponding to an x-axis and an air-conditioning temperature corresponding to a y-axis;

[0047] A reward range provision module, used for acquiring thermal comfort experience results and energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and adding the reward range to the thermal comfort curve;

[0048] An initial reinforcement learning module, used to initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate a corresponding first reward or penalty. If there is a penalty, the parameters of the deep neural network are updated based on the penalty until a preset termination condition is reached to end the preliminary training of the deep reinforcement learning model, wherein the agent is an air conditioner, the action is to adjust the temperature, and the input state is set to be the air conditioner temperature;

[0049] An advanced reinforcement learning module, used to introduce user feedback information after the initial training is completed, and perform advanced training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, and generate a corresponding second reward based on the user feedback information obtained when the action with the largest first reward enters a new state, until a preset termination condition is reached to terminate the advanced training of the deep reinforcement learning model;

[0050] The strategy instruction generation module is used to input the current temperature as the state after the deep reinforcement learning model training is completed, and adjust the air conditioner according to the action corresponding to the highest final reward.

[0051] The technical solution provided by the embodiments of the present application has the following technical effects:

[0052] The corresponding thermal comfort curve is drawn in the template of the corresponding season by obtaining the current time and environmental data. When the building cannot obtain real-time air-conditioning control data, the thermal comfort curve can be used to provide a large amount of reliable training data for the training of the reinforcement learning model. Rewards or penalties are given directly according to the results of the thermal comfort curve. Through the reward range, the results of deep reinforcement learning can be converged within the effective range when there is a lack of actual usage data, which solves the problem of insufficient reinforcement learning training data. When it is put into use later, the intelligent effect of deep reinforcement learning can be evaluated in real time based on user feedback information. Based on the user's thermal comfort habits, the appropriate temperature control in each area and at each time point can be slowly learned, so as to achieve energy conservation and emission reduction on the basis of thermal comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the steps of a method for generating an air conditioning control strategy provided in an embodiment of the present application.

[0054] Figure 2 Schematic diagram of thermal comfort curve in the embodiment of the present application.

[0055] Figure 3 It is a schematic diagram of the reinforcement learning process in the embodiment of the present application.

[0056] Figure 4 This is a schematic diagram of module connections of an air conditioning control strategy generation system in an embodiment of the present application. DETAILED DESCRIPTION

[0057] To more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. However, it should be understood by those of ordinary skill in the art that the present application can be implemented without these details. In some cases, in order to avoid unnecessary descriptions that make various aspects of the present application obscure, well-known methods, processes, systems, components and / or circuits that have been described at a higher level will not be described in detail. For those of ordinary skill in the art, it is obvious that various changes can be made to the embodiments disclosed in the present application, and without departing from the principles and scope of the present application, the general principles defined in the present application can be applied to other embodiments and application scenarios. Therefore, the present application is not limited to the embodiments shown, but conforms to the broadest scope consistent with the scope claimed for protection of the present application.

[0058] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as there is no conflict between them.

[0059] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used to distinguish the technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0060] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0061] The embodiment of the present application discloses a method for generating an air conditioning adjustment strategy.

[0062] like Figure 1 As shown, a method for generating an air conditioning control strategy includes the following steps:

[0063] S100, obtaining current environmental parameters and time information.

[0064] The current environmental data and time data are obtained through the Internet of Things module. The environmental parameters include the current outdoor temperature, humidity, wind speed and other information, while the time information is the current month and date.

[0065] S200, selecting a corresponding thermal comfort standard in a preset season template based on time information, and analyzing environmental parameters according to the thermal comfort standard to draw a corresponding thermal comfort curve.

[0066] like Figure 2 As shown, the preset season template stores thermal comfort standards corresponding to several different seasons. The thermal comfort standards can be imported from corresponding research reports, literature, standards, etc. Because users have different requirements for heating or cooling of air conditioners in different seasons, the way to save energy is to prevent over-cooling or over-heating, so the models for different seasons are also different.

[0067] After setting up templates and standards for different seasons, first determine the current season based on the current time information and select the appropriate season template. Draw the corresponding thermal comfort curve based on the thermal comfort standard of the season template combined with the current environmental parameters. For example, when the outdoor temperature is 35 degrees in summer, draw the comfortable temperature for each time period of the day based on the thermal comfort standard. Different seasons, different time points, and different outdoor temperatures correspond to different thermal comfort curves.

[0068] The thermal comfort curve includes time corresponding to the x-axis and air-conditioning temperature corresponding to the y-axis.

[0069] S300: Acquire thermal comfort experience results and energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and add the reward range to a thermal comfort curve.

[0070] The thermal comfort experience result is characterized by the empirical value of the outdoor temperature-indoor air-conditioning temperature ratio obtained through historical data on human body perception, which means the air-conditioning temperature corresponding to the ambient temperature and time when the human body feels thermally comfortable calculated based on historical experience.

[0071] The energy consumption experience results are characterized by the corresponding policy information of regions, parks, etc. where different buildings are located, combined with the empirical algorithm to obtain the allowed or recommended air-conditioning temperatures in different seasons and under different ambient temperatures under the policy-specified energy consumption.

[0072] The maximum and minimum temperature values ​​for the thermal comfort curve are generated by combining the subjective experience data of human body sensation and the objective energy consumption data of policy energy consumption requirements. The range between the maximum and minimum temperature values ​​is defined as the reward range. The reward range is used for subsequent deep reinforcement learning, which is a constraint condition for the actions of the intelligent agent in deep reinforcement learning.

[0073] S400, initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the intelligent agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate a corresponding first reward or penalty. If there is a penalty, update the parameters of the deep neural network based on the penalty until the preset termination condition is reached to end the preliminary training of the deep reinforcement learning model.

[0074] The intelligent agent is initially trained through reinforcement learning. In the case that there is no air-conditioning control data in the new building and it is impossible to obtain it, a certain number of training parameters can be quickly collected according to the constructed thermal comfort curve to quickly train the reinforcement learning model, and the model can be directly rewarded or punished according to the results of the thermal comfort curve. If the new state is within the upper and lower limits of thermal comfort, a reward is given; if the new state is outside the upper and lower limits of thermal comfort, a penalty is given.

[0075] like Figure 3 As shown, reinforcement learning is an important branch of machine learning, which mainly focuses on how agents can take strategies in the environment to maximize rewards and benefits. Reinforcement learning learns which action to choose in a specific environment to get the greatest reward in the process of continuous attempts. Reinforcement learning is mainly composed of agents, environments, states, actions, rewards, and penalties. The agent is an air conditioner, and the action is the adjustment of the air conditioner temperature. In this application, the actions include increasing the temperature by 1 degree, decreasing the temperature by 1 degree, and the air conditioner temperature in the state of constant temperature input. The rewards are energy saving and comfort, and the penalties are non-energy saving and discomfort. When the action of the agent corresponds to a reward, it means that the agent is encouraged to perform the action at the current time point, and if there is a penalty, it means that the agent is punished so that the agent will not perform the action at the current time point in the future.

[0076] S500, after the initial training is completed, user feedback information is introduced, and the deep reinforcement learning model corresponding to the intelligent agent is advanced trained through reinforcement learning, and user feedback information based on the action with the largest first reward entering a new state is obtained to generate a corresponding second reward, until the preset termination condition is reached to end the advanced training of the deep reinforcement learning model.

[0077] After the initial training is completed, the deep reinforcement learning model will obtain a range of standardized behaviors. At this time, the user feedback information will be input and the specific temperature control process of the intelligent agent will be reinforced through the user feedback information.

[0078] User feedback information is represented by the user's active adjustment of the air-conditioning temperature. When the intelligent agent performs intelligent temperature control based on the ambient temperature and thermal comfort curve, it will detect the instructions actively issued by the user. The process of issuing instructions is also a kind of action, and evaluate the current temperature, which is the second reward.

[0079] If the user raises or lowers the current air-conditioning temperature after the intelligent agent controls the air-conditioning temperature, it means that the user is not satisfied with the current temperature. If the user does not actively adjust the air-conditioning temperature after the intelligent agent controls the air-conditioning temperature, it means that the user is satisfied with the current temperature.

[0080] Satisfied or unsatisfied evaluations correspond to different second rewards, so that the initially trained model can be further trained through user feedback to improve the accuracy of subsequent intelligent energy-saving control of the air conditioner.

[0081] S600, after the deep reinforcement learning model training is completed, the current temperature is input as the state, and the air conditioner is adjusted according to the action corresponding to the highest final reward.

[0082] When the deep reinforcement model training is completed, the current temperature at a certain point in time can be sent to the intelligent agent as a state. The intelligent agent will calculate the action with the largest reward to control whether the air-conditioning temperature in the current environment needs to be increased, unchanged or decreased as the corresponding strategy, thereby realizing intelligent energy-saving temperature control.

[0083] Through the above steps, the corresponding thermal comfort curve is drawn in the template of the corresponding season by obtaining the current time and environmental data, and when the building cannot obtain real-time air-conditioning control data, the thermal comfort curve is used to provide a large amount of reliable training data for the training of the reinforcement learning model. Rewards or penalties are given directly according to the results of the thermal comfort curve. Through the reward range, the results of deep reinforcement learning can be converged within the effective range when there is a lack of actual usage data, which solves the problem of insufficient reinforcement learning training data. When it is put into use later, the intelligent effect of deep reinforcement learning can be evaluated in real time according to user feedback information, and the appropriate temperature control in each area and at each time point can be slowly learned based on the user's thermal comfort habits, so as to achieve energy conservation and emission reduction on the basis of thermal comfort.

[0084] In some other embodiments, a corresponding thermal comfort standard is selected in a preset season template based on time information, and environmental parameters are analyzed by the thermal comfort standard to draw a corresponding thermal comfort curve, including the following steps:

[0085] S210, judging the current season according to the time information to select a summer template, a winter template or a transitional season template, and obtaining a corresponding thermal comfort standard.

[0086] Buildings can be divided into summer, winter and transition seasons based on energy consumption. The transition seasons are spring and autumn. Since there is no extreme cold or heat in spring and autumn, the temperature adjustment range of air conditioners is generally relatively stable. The energy consumption control methods in summer and winter are completely different. In summer, cooling is used, and the energy-saving method is to prevent over-cooling. In winter, heating is used, so the energy-saving method is to avoid over-heating. Therefore, the models corresponding to different seasons are also different.

[0087] The thermal comfort standard is characterized by the 24-hour air-conditioning comfort temperature corresponding to different temperature ranges in the season. For example, the comfort temperature at various times of the day when the ambient temperature in summer is 35 degrees.

[0088] S220, obtaining temperature information in the environmental parameters, and selecting the air-conditioning comfort temperature for each time period according to the temperature range matched by the temperature information and drawing a corresponding thermal comfort curve, wherein the thermal comfort curve can be manually modified.

[0089] The temperature information in the environmental parameters, that is, the outdoor temperature, is obtained in real time, and the air-conditioning comfort temperature corresponding to the outdoor temperature is determined according to the temperature range of the outdoor temperature. For example, from 8 am to 4 pm, the outdoor temperature is between 30 and 35 degrees, and this period corresponds to one air-conditioning temperature. From 4 pm to 8 pm, the outdoor temperature is between 25 and 30 degrees, and this period corresponds to another air-conditioning temperature. Finally, the corresponding thermal comfort curve is drawn according to the air-conditioning comfort temperature corresponding to different time points.

[0090] At the same time, the thermal comfort curve can be modified manually. For example, different building environments correspond to different uses. For example, commercial buildings have a large flow of people. In order to improve the thermal comfort during high flow of people, the air-conditioning temperature at each time point in the thermal comfort curve can be set lower. If the building environment corresponds to a small-scale office area, because the number of people is relatively small and they are in a static state when working, the air-conditioning temperature at each time point in the thermal comfort curve can be set higher.

[0091] At the same time, it is also possible to draw multiple independent and customized thermal comfort curves in one building according to the different zoning uses in the overall building.

[0092] In some other embodiments, obtaining the thermal comfort experience result and the energy consumption experience result to generate a reward range including a maximum temperature value and a minimum temperature value includes the following steps:

[0093] S310, obtaining a first range including a minimum ideal body temperature and a maximum ideal body temperature under different ambient temperatures according to the air conditioner use survey data combined with an empirical algorithm.

[0094] Through historical data on air conditioning usage surveys, such as the ideal temperature habits preferred by users in various buildings in the past, a minimum and maximum temperature based on the user's subjective speculation is generated. The temperatures generated here are all ideal perceived temperatures.

[0095] S320, obtaining a second range including a minimum ideal energy consumption temperature and a maximum ideal energy consumption temperature under different ambient temperatures according to the policy data combined with an empirical algorithm.

[0096] According to the policy information corresponding to the building and the empirical algorithm, the objectively estimated minimum and maximum temperatures are obtained. The temperatures generated here are all ideal energy consumption required temperatures.

[0097] S330: If the selected season template is a summer template, the lower value of the highest ideal body temperature and the highest ideal energy consumption temperature is used as the maximum temperature value, and the lowest ideal energy consumption temperature is used as the minimum temperature value.

[0098] If the selected season template is the summer template, it means that cooling is required. In the cooling scenario, the lower the cooling temperature, the greater the corresponding energy consumption. Therefore, when selecting the reward range in the first range and the second range, it is necessary to focus on the selection of the minimum temperature value. Therefore, the selected minimum temperature value defaults to the lowest ideal energy consumption temperature. Regardless of whether the lowest ideal perceived temperature is lower than the lowest ideal energy consumption temperature, the lowest ideal energy consumption temperature is used as the lowest critical value. The selection of the maximum temperature value is based on the lower value of the highest ideal perceived temperature and the highest ideal energy consumption temperature. This ensures that when the minimum temperature value is fixed, the size of the reward range is minimized, because a larger reward range will lead to unstable training or difficult convergence.

[0099] S340: If the selected season template is a winter template, a higher value between the lowest ideal body temperature and the lowest ideal energy consumption temperature is used as the minimum temperature value, and the highest ideal energy consumption temperature is used as the maximum temperature value.

[0100] If the selected season template is a winter template, the principle is opposite to that of the summer template. It is necessary to focus on the selection of the maximum temperature value. Therefore, the selected maximum temperature value defaults to the highest ideal energy consumption temperature, and the minimum temperature value still selects the temperature value that minimizes the reward range.

[0101] S350: If the selected season template is a transition season template, the union of the first range and the second range is used as the reward range.

[0102] If the transition season template is selected, it means that the current requirements for subjective physical comfort and objective energy saving are not very high. At this time, when selecting the reward range, you can directly select the union of the first range and the second range to obtain the smallest reward range.

[0103] In another rare case, if there is no intersection between the first range and the second range, the reward range is the smallest range that includes the first range and the second range.

[0104] In some other embodiments, generating a random action based on a target state and determining whether a new state after the action is within a reward range to generate a corresponding first reward or penalty includes the following steps:

[0105] S410, generating a target state based on the air-conditioning temperature on the thermal comfort curve corresponding to the current time.

[0106] S420, input the current air-conditioning temperature as the current state, and predict the Q values ​​corresponding to all possible actions through a deep neural network based on the current state and the target state.

[0107] S430, executing an action corresponding to the highest Q value to enter a new state.

[0108] The target is the air conditioning temperature corresponding to the thermal comfort curve at the current time. The agent enters different states through random actions to finally reach the target state. Each action corresponds to a reward. A corresponding Q value is obtained through different rewards. The Q value is calculated by the neural network algorithm. The agent will select an action with the largest Q value to execute among multiple random actions. The Q-value function is a function that maps states and actions to Q values, which represents the expected return obtained by executing the action in a specific state. In reinforcement learning, the goal is to find the optimal strategy so that the optimal action can be taken in any state to obtain the maximum expected return. The Q-value function provides a way to calculate the quality of a strategy because the Q-value of the optimal strategy is the highest among all strategies.

[0109] S440, if the new state is within the reward range, a first reward is obtained based on the convergence between the new state and the target state. If converged, the first reward is a positive number, if discrete, the first reward is a negative number, and if the discrete type remains unchanged, the first reward is 0.

[0110] If the new state after the action is within the reward range, a reward is given so that the agent has the motivation to act within the specified range. At the same time, the size of the reward depends on the convergence between the new state and the target state. If the new state is close to the target state, it is represented as convergence, and its first reward is a positive number. If it is far away from the target state, it is represented as discrete, and its first reward is a negative number. If the new state is neither close to nor far from the target state, its first reward is 0. The larger the reward, the greater the agent's vision for taking the action.

[0111] The size of the first reward can be set according to different reward strategies in different situations, for example:

[0112] A +1 reward is set for each action in the convergence direction between the new state and the target state, and a -1 reward is set for each action in the discrete direction between the new state and the target state. Since the reward in the convergence direction is always greater than the reward in the discrete direction, the agent is encouraged to act in the convergence direction with a greater reward.

[0113] In the convergence direction between the new state and the target state, a positive reward in the order of an arithmetic progression is set. The closer to the target state, the greater the reward. In the discrete direction between the new state and the target state, a negative reward in the order of an arithmetic progression is set. The farther away from the target state, the smaller the reward. In this case, the agent may behave greedily and choose to repeat the action back and forth in order to get more rewards. At this time, you can choose to increase the value of Ԑ to give the agent a chance to update the state value in the convergence direction to encourage the agent to move in the convergence direction. Ԑ is a probability. The agent will choose a random behavior with probability Ԑ and choose the best behavior currently known with probability 1-Ԑ, allowing the agent to use known information in most cases while retaining a certain opportunity for exploration.

[0114] S450, if the new state is outside the reward range, a penalty is generated.

[0115] If the new state is outside the reward range, it means that the new state exceeds the maximum temperature value or the minimum temperature value. This is a behavior that needs to be suppressed, so a corresponding penalty will be generated to suppress the recurrence of this action by the agent.

[0116] It should be noted that there is a difference between punishment and negative rewards. Although negative rewards themselves are also represented as punishment by deducting points, negative rewards are essentially a kind of incentive in the overall dimension. They are used to motivate the agent to act towards a state with a higher reward value, but they will not inhibit the agent from acting towards a state with a negative reward. The essence of punishment is to reduce the probability of an action by giving disgust or reducing pleasurable stimulation.

[0117] In this way, the first reward and punishment can make the agent keep its actions within the set encouragement range during the initial training, avoiding overcooling, overheating or high energy consumption.

[0118] In some other embodiments, if the new state is within the reward range, obtaining a corresponding first reward based on the convergence between the new state after the random action and the target state further includes the following steps:

[0119] S460, obtaining the user's usage habit information of the air conditioner, and generating a plurality of sweet spots based on the usage habit information.

[0120] User habit information can be obtained through historical data or corresponding Internet data combined with empirical algorithms, which refers to the predicted air conditioning usage habits of a wide range of users.

[0121] For example, in summer, 8-9 am is a high-probability working time. When users arrive at the company during working hours, the temperature is generally hot. At this time, users are accustomed to turning on the air conditioner and lowering the temperature. For example, 11-12 noon is the peak meal time. At this time, users are also in a hot state when they come back after eating outside. After 4 pm, the outdoor temperature drops, and users are accustomed to increasing the air conditioner temperature.

[0122] An empirical algorithm based on user habits is used to obtain the time point at which key behaviors are likely to occur, and this time point is considered as the sweet spot.

[0123] S470, finding a corresponding point position in the thermal comfort curve according to the sweet spot, and increasing a first reward of the target state corresponding to the point position.

[0124] At the time corresponding to the sweet spot, the first reward for taking actions to reach the target state will increase, so that at this point in time, the agent will be more encouraged to quickly adjust to a more reasonable temperature.

[0125] In some other embodiments, obtaining user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward includes the following steps:

[0126] S510, waiting for whether there is user feedback information in the new state.

[0127] The user feedback information is characterized by the user manually adjusting the air-conditioning temperature upward or downward.

[0128] Advanced training is a real-time training process based on user behavior when the smart air conditioning system is put into operation. It uses the user's manual control of the current air conditioning temperature as feedback to evaluate whether the air conditioning automatic temperature adjustment behavior is reasonable and satisfactory to the user. The action here is represented by the user manually adjusting the air conditioning temperature up by 1 degree, down by 1 degree, or unchanged.

[0129] S520: If it does not exist, the generated second reward is a positive number.

[0130] S530, if it exists, then calculate the absolute value of the difference between the air-conditioning temperature corresponding to the new state and the air-conditioning temperature corresponding to the user feedback information and define it as the feedback evaluation result, and generate a corresponding negative second reward according to the feedback evaluation result. If the value of the feedback evaluation result is larger, the value of the second reward is smaller.

[0131] In advanced training, there are only positive and negative rewards, but no 0. If the second reward is a positive number, it means that after the air conditioner is intelligently adjusted, the user did not manually adjust the air conditioner temperature. If the user did not manually adjust it, it means that the user is comfortable and satisfied with the current temperature adjusted by the air conditioner. If the second reward is a negative number, it means that after the air conditioner is intelligently adjusted, the user manually adjusted the air conditioner temperature. This means that the user is uncomfortable and dissatisfied with the temperature adjusted by the air conditioner.

[0132] The user's manual behavior is used to provide feedback on whether the temperature of the air conditioner's intelligent temperature control is "comfortable". Combined with the rewards for different actions given by reinforcement learning, the intelligent agent can continuously learn from people's comfort feedback, achieve better intelligent temperature control, and better improve user satisfaction.

[0133] At the same time, when the user is dissatisfied, the size of the second reward varies according to the degree of dissatisfaction of the user. It is necessary to calculate the temperature difference between the new state of the air conditioner after the action based on the largest first reward and the temperature corresponding to the user's feedback result. When the difference is large, it means that the user has manually adjusted the temperature more, and it is considered that the user is more dissatisfied with the current air conditioner temperature. When the difference is small, it means that the user has manually adjusted the temperature less, and it is considered that the user is less dissatisfied with the current air conditioner temperature.

[0134] And because the second reward is a negative number, the smaller the first reward, the smaller the benefit of the agent moving to that state. Conversely, the larger the first reward, the greater the benefit of the agent moving to that state. Therefore, when the feedback evaluation result is larger, the smaller the first reward.

[0135] For example, when the air conditioner adjusts the air conditioner temperature from 24 degrees to 26 degrees based on the maximum first reward, if the user adjusts from 26 degrees to 28 degrees, the feedback evaluation result is 2, and the second reward is -4. If the user adjusts from 26 degrees to 30 degrees, the feedback evaluation result is 4, and the second reward is -8. The smaller the second reward, the more it encourages the agent to move in the direction of a larger second reward, and the closer it is to the temperature corresponding to the user's feedback information, the larger its second reward, so that the agent learns the user's air conditioner usage habits.

[0136] In some other embodiments, obtaining user feedback information based on the action with the largest first reward and entering a new state to generate a second reward further includes the following steps:

[0137] S540, determining the air conditioning temperature value distribution corresponding to the current state, the new state and the feedback evaluation result.

[0138] In order to improve the accuracy of reinforcement learning, the size of the second reward can be further set according to the numerical distribution of air-conditioning temperatures corresponding to the current state, the new state, and the feedback evaluation results.

[0139] S550, generating a same-direction action reward and a reverse action reward according to the numerical distribution of the air-conditioning temperature. In the same temperature adjustment unit, the same-direction action reward is greater than the reverse action reward.

[0140] Specifically, the same-direction action reward is represented by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on the same side of the value of the new state, and the reverse action reward is represented by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on both sides of the value of the new state.

[0141] An example of a same-direction action reward is: the current state is 26 degrees, and the new state after the air conditioner adjusts the temperature is 28 degrees. The feedback evaluation result indicates that the user adjusts the temperature from 28 degrees to 30 degrees. At this time, the current state to the new state is a heating process, and the user's feedback evaluation result is also a heating process, that is, the temperature adjustment logic of the intelligent agent is the same as the temperature adjustment logic of the user. At this time, it means that the temperature adjustment action of the intelligent agent conforms to the correct temperature adjustment logic, but the temperature adjustment size does not meet the user's requirements.

[0142] An example of a reverse action reward is: the current state is 26 degrees, and the new state after the air conditioner adjusts the temperature is 28 degrees. The feedback evaluation result indicates that the user readjusts 28 degrees to 26 degrees. At this time, the current state to the new state is a heating process, while the user's feedback evaluation result is a cooling process. That is, the temperature adjustment logic of the intelligent agent is opposite to the temperature adjustment logic of the user. This means that the temperature adjustment action of the intelligent agent does not conform to the correct temperature adjustment logic.

[0143] In the above two examples, although the user feedback results are that the user adjusted the temperature by two degrees, the logic of the air conditioner's intelligent temperature adjustment is completely opposite to the temperature adjustment logic that the user wants. Therefore, for the same action of adjusting the temperature by two degrees, the size of the second reward corresponding to different temperature adjustment logic directions is also different. When the second reward for the same direction action is greater than the second reward for the reverse action, the intelligent agent can strengthen the learning of the user's temperature adjustment habit logic when adjusting the temperature, reducing the situation where the user wants to cool down but the air conditioner turns up the temperature or the user wants to heat up but the air conditioner turns down the temperature.

[0144] In some other embodiments, after the corresponding second reward is generated, the following steps are further included:

[0145] S560: Determine whether the generation time of the user feedback information corresponds to the sweet spot.

[0146] S561: If yes, reduce the second reward of the air-conditioning temperature corresponding to the user feedback information.

[0147] If the generation time of the user feedback information corresponds to the sweet spot mentioned above, it is considered that the user's high-incidence time of temperature adjustment action coincides with the high-probability time of user habits predicted by the system, and the user's feedback behavior serves as a proof of the user habit prediction. At this time, it is confirmed that the user has a demand for a certain value of air-conditioning temperature at the time corresponding to the sweet spot, and the size of the second reward can be reduced.

[0148] The smaller the second reward value is, the smaller the positive reward for the agent is. The more the agent is encouraged to act in the direction of a larger positive reward, and the smaller the probability of the agent taking an incorrect action at the sweet spot is. This enables the agent to learn the user's common behaviors and common air-conditioning temperatures at the user's habitual time points.

[0149] In other embodiments, the final reward is the sum of the first reward and the second reward corresponding to the action, wherein the absolute value of the second reward corresponding to the same action is greater than the absolute value of the first reward.

[0150] The final reward is obtained by the sum of the first reward and the second reward. When the agent performs intelligent temperature control after the subsequent reinforcement learning is completed, it will select the action with the largest final reward to enter a new state after inputting the current state, so as to determine whether the air conditioning in the current environment needs to be increased, unchanged or decreased.

[0151] At the same time, in some cases, the same action may correspond to different second rewards and first rewards. For example, in the thermal comfort curve, when the air-conditioning temperature is adjusted from 26 degrees to 25 degrees at noon, it corresponds to the largest first reward, such as +1, and the second reward corresponds to -2, which means that this action is incorrect for the user's habits. When the agent adjusts the temperature to 25, the user feels cold and needs to adjust the temperature back to 26 degrees. Therefore, in order to make the agent learn during reinforcement learning that controlling the air-conditioning temperature from 26 degrees to 25 degrees at this point in time is a negative penalty greater than a positive penalty, the agent is encouraged to act towards a temperature that the user feels comfortable with.

[0152] That is to say, in the process of deep reinforcement learning, when there is a conflict between the first reward with the thermal comfort curve as the learning benchmark and the second reward with the user feedback information as the learning benchmark for the same action, the consideration priority value of the second reward is greater than the consideration priority value of the first reward.

[0153] like Figure 4 As shown, the present application also discloses an air conditioning control strategy generation system, including:

[0154] The Internet of Things module is used to obtain the current environmental parameters and time information.

[0155] The IoT module is used to retrieve data and control the air conditioner. Users connect to the HVAC through the IoT protocol, and the connection protocol includes but is not limited to Bacnet, Opc, Onvif, Modbus, 485, 232, 422 and other protocols. The IoT module obtains the control instructions of the user on the air conditioner panel in a certain area through the IoT protocol, that is, the user feedback information. The IoT module can not only obtain the current temperature, operating status, ambient temperature, current time, etc. of the air conditioner, but also record the user's behavior of controlling the air conditioner.

[0156] The thermal comfort curve drawing module is used to select the corresponding thermal comfort standard in the preset season template based on time information, analyze the environmental parameters according to the thermal comfort standard to draw the corresponding thermal comfort curve. The thermal comfort curve includes the time corresponding to the x-axis and the air-conditioning temperature corresponding to the y-axis.

[0157] The reward range specification module is used for obtaining the thermal comfort experience results and the energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and adding the reward range to the thermal comfort curve.

[0158] The initial reinforcement learning module is used to initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the intelligent agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate the corresponding first reward or penalty. If there is a penalty, the parameters of the deep neural network are updated based on the penalty until the preset termination condition is reached to end the preliminary training of the deep reinforcement learning model, wherein the intelligent agent is an air conditioner, the action is to adjust the temperature, and the input state is set to the air conditioner temperature.

[0159] The advanced reinforcement learning module is used to introduce user feedback information at the end of preliminary training, and perform advanced training on the deep reinforcement learning model corresponding to the intelligent agent through reinforcement learning. The corresponding second reward is generated based on the user feedback information obtained when entering a new state based on the action with the largest first reward, until the preset termination condition is reached to end the advanced training of the deep reinforcement learning model.

[0160] The policy instruction generation module is used to input the current temperature as the state after the deep reinforcement learning model training is completed, and adjust the air conditioner according to the action corresponding to the highest final reward.

[0161] The implementation principle is:

[0162] The corresponding thermal comfort curve is drawn in the template of the corresponding season by obtaining the current time and environmental data. When the building cannot obtain real-time air-conditioning control data, the thermal comfort curve can be used to provide a large amount of reliable training data for the training of the reinforcement learning model. Rewards or penalties are given directly according to the results of the thermal comfort curve. Through the reward range, the results of deep reinforcement learning can be converged within the effective range when there is a lack of actual usage data, which solves the problem of insufficient reinforcement learning training data. When it is put into use later, the intelligent effect of deep reinforcement learning can be evaluated in real time based on user feedback information. Based on the user's thermal comfort habits, the appropriate temperature control in each area and at each time point can be slowly learned, so as to achieve energy conservation and emission reduction on the basis of thermal comfort.

[0163] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise clearly stated in this document, the execution of these steps is not strictly limited in order and can be performed in other orders.

[0164] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating an air conditioning control strategy, characterized in that: The following steps are involved: Get current environmental parameters and time information; Selecting a corresponding thermal comfort standard in a preset season template based on the time information, analyzing the environmental parameters according to the thermal comfort standard to draw a corresponding thermal comfort curve, wherein the thermal comfort curve includes a time corresponding to an x-axis and an air-conditioning temperature corresponding to a y-axis; Acquire thermal comfort experience results and energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and add the reward range to the thermal comfort curve; Initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate a corresponding first reward or penalty. Specifically, generating the target state based on the air conditioning temperature on the thermal comfort curve corresponding to the current time; Input the current air-conditioning temperature as the current state, and predict the Q values ​​corresponding to all possible actions through a deep neural network based on the current state and the target state; An action corresponding to the highest Q value is executed to enter a new state; If the new state is within the reward range, a first reward is obtained based on the convergence between the new state and the target state, wherein if convergence occurs, the first reward is a positive number, if discrete, the first reward is a negative number, and if discreteness remains unchanged, the first reward is 0; If the new state is outside the reward range, a penalty is generated; If the penalty exists, the parameters of the deep neural network are updated based on the penalty until a preset termination condition is reached to end the preliminary training of the deep reinforcement learning model, wherein the agent is an air conditioner, the action is adjusting the temperature of the air conditioner, and the state of the set input is the temperature of the air conditioner; After the initial training is completed, user feedback information is introduced, and the deep reinforcement learning model corresponding to the agent is advanced trained through reinforcement learning, and the user feedback information based on the action with the largest first reward entering a new state is obtained to generate a corresponding second reward, until a preset termination condition is reached to end the advanced training of the deep reinforcement learning model; Wherein, obtaining the user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward includes the following steps: Waiting for the user feedback information to exist in the new state, where the user feedback information is characterized by the user manually adjusting the air-conditioning temperature upward or downward; If it does not exist, the generated second reward is a positive number; If it exists, then the absolute value of the difference between the air-conditioning temperature corresponding to the new state and the air-conditioning temperature corresponding to the user feedback information is calculated and defined as the feedback evaluation result, and the second reward of a corresponding negative number is generated according to the feedback evaluation result. If the value of the feedback evaluation result is larger, the value of the second reward is smaller; After the deep reinforcement learning model training is completed, the current temperature is input as the state, and the air conditioner is adjusted according to the action corresponding to the highest final reward, wherein the final reward is the sum of the first reward and the second reward corresponding to the action.

2. The air conditioning control strategy generation method according to claim 1, characterized in that: Selecting a corresponding thermal comfort standard in a preset season template based on the time information, analyzing the environmental parameters by the thermal comfort standard to draw a corresponding thermal comfort curve, including the following steps: Determine the current season according to the time information to select a summer template, a winter template or a transitional season template, and obtain the corresponding thermal comfort standard, which is characterized by the 24-hour air-conditioning comfort temperature corresponding to different temperature ranges in the season; The temperature information in the environmental parameters is obtained, and the air-conditioning comfort temperature of each time period is selected according to the temperature range matched by the temperature information and a corresponding thermal comfort curve is drawn, wherein the thermal comfort curve can be manually modified.

3. The air conditioning control strategy generation method according to claim 2, characterized in that: Obtaining the thermal comfort experience results and energy consumption experience results to generate a reward range that includes the maximum temperature value and the minimum temperature value includes the following steps: According to the air conditioner usage survey data combined with the empirical algorithm, a first range including the lowest ideal body temperature and the highest ideal body temperature under different ambient temperatures is obtained; Acquire a second range including the lowest ideal energy consumption temperature and the highest ideal energy consumption temperature at different ambient temperatures according to the policy data combined with an empirical algorithm; If the selected season template is a summer template, the lower value of the highest ideal body temperature and the highest ideal energy consumption temperature is used as the maximum temperature value, and the lowest ideal energy consumption temperature is used as the minimum temperature value; If the selected season template is a winter template, the higher value of the lowest ideal body temperature and the lowest ideal energy consumption temperature is used as the minimum temperature value, and the highest ideal energy consumption temperature is used as the maximum temperature value; If the selected season template is a transition season template, the union of the first range and the second range is used as the reward range.

4. The air conditioning control strategy generation method according to claim 1, characterized in that: If the new state is within the reward range, obtaining a corresponding first reward based on the convergence between the new state after the random action and the target state, further comprising the following steps: Acquire the user's usage habit information of the air conditioner, and generate a number of sweet spots based on the usage habit information; A corresponding point position is found in the thermal comfort curve according to the sweet spot, and the first reward of the target state corresponding to the point position is increased.

5. The air conditioning control strategy generation method according to claim 4, characterized in that: Acquiring the user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward, further comprising the following steps: Determine the air conditioning temperature value distribution corresponding to the current state, the new state and the feedback evaluation result; Generate a same-direction action reward and a reverse action reward according to the numerical distribution of the air-conditioning temperature, wherein in the same temperature adjustment unit, the same-direction action reward is greater than the reverse action reward; The same-direction action reward is characterized by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on the same side of the value of the new state, and the reverse action reward is characterized by the second reward corresponding to when the values ​​corresponding to the current state and the feedback evaluation result are on both sides of the value of the new state.

6. The air conditioning control strategy generation method according to claim 5, characterized in that: After the corresponding second reward is generated, the following steps are also included: Determining whether the generation time of the user feedback information corresponds to the sweet spot; If so, the second reward for the air-conditioning temperature corresponding to the user feedback information is reduced.

7. The air conditioning control strategy generation method according to claim 1, characterized in that: The absolute value of the second reward corresponding to the same action is greater than the absolute value of the first reward.

8. An air conditioning control strategy generation system, characterized in that: include: Internet of Things module, used to obtain current environmental parameters and time information; a thermal comfort curve drawing module, configured to select a corresponding thermal comfort standard in a preset season template based on the time information, analyze the environmental parameters according to the thermal comfort standard to draw a corresponding thermal comfort curve, wherein the thermal comfort curve includes a time corresponding to an x-axis and an air-conditioning temperature corresponding to a y-axis; A reward range provision module, used for acquiring thermal comfort experience results and energy consumption experience results to generate a reward range including a maximum temperature value and a minimum temperature value, and adding the reward range to the thermal comfort curve; The initial reinforcement learning module is used to initialize the deep neural network, perform preliminary training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, generate random actions based on the target state, and determine whether the new state after the action is within the reward range to generate the corresponding first reward or penalty. Specifically, generating the target state based on the air conditioning temperature on the thermal comfort curve corresponding to the current time; Input the current air-conditioning temperature as the current state, and predict the Q values ​​corresponding to all possible actions through a deep neural network based on the current state and the target state; An action corresponding to the highest Q value is executed to enter a new state; If the new state is within the reward range, a first reward is obtained based on the convergence between the new state and the target state, wherein if convergence occurs, the first reward is a positive number, if discrete, the first reward is a negative number, and if discreteness remains unchanged, the first reward is 0; If the new state is outside the reward range, a penalty is generated; If the penalty exists, the parameters of the deep neural network are updated based on the penalty until a preset termination condition is reached to end the preliminary training of the deep reinforcement learning model, wherein the agent is an air conditioner, the action is to adjust the temperature, and the state of the set input is the air conditioner temperature; An advanced reinforcement learning module, used to introduce user feedback information after the initial training is completed, and perform advanced training on the deep reinforcement learning model corresponding to the agent through reinforcement learning, and generate a corresponding second reward based on the user feedback information obtained when the action with the largest first reward enters a new state, until a preset termination condition is reached to terminate the advanced training of the deep reinforcement learning model; Wherein, obtaining the user feedback information based on the action with the largest first reward entering a new state to generate a corresponding second reward includes the following steps: Waiting for the user feedback information to exist in the new state, where the user feedback information is characterized by the user manually adjusting the air-conditioning temperature upward or downward; If it does not exist, the generated second reward is a positive number; If it exists, then the absolute value of the difference between the air-conditioning temperature corresponding to the new state and the air-conditioning temperature corresponding to the user feedback information is calculated and defined as the feedback evaluation result, and the second reward of a corresponding negative number is generated according to the feedback evaluation result. If the value of the feedback evaluation result is larger, the value of the second reward is smaller; A strategy instruction generation module is used to input the current temperature as a state after the deep reinforcement learning model training is completed, and adjust the air conditioner according to the action corresponding to the highest final reward, wherein the final reward is the sum of the first reward and the second reward corresponding to the action.

Images