Air conditioner control method and device based on reinforcement learning, medium and program product

Abstract

The invention relates to the field of air conditioner control, in particular to an air conditioner control method based on reinforcement learning, an air conditioner control device capable of achieving the method, a computer readable storage medium and a computer program product. The method comprises the following steps: constructing a state vector reflecting a current situation based on user preference information, vehicle internal and external environment information, vehicle state information and a user driving path; analyzing the state vector by utilizing a reinforcement learning model, and generating offset information for adjusting the air conditioner control instruction; the current air conditioner control instruction is corrected through the offset information; and the adjustment behavior of the user on the air conditioner setting serves as a reward signal to be fed back to the reinforcement learning model, so that model parameters of the reinforcement learning model and the generated offset information are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of air conditioner control, and more particularly to an air conditioner control method based on reinforcement learning, an air conditioner control device capable of implementing the method, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Vehicle air conditioning system is an important part of improving the comfort of driving and riding. The traditional vehicle air conditioning system mainly adopts manual control or automatic control based on fixed rules. Manual control requires the driver to constantly adjust the air conditioning settings according to his own feelings, which is tedious and distracting to driving, and has safety hazards. Although the automatic control based on fixed rules can automatically adjust the temperature in the car, it cannot fully consider the individual needs of users and changes in external environment, and it is difficult to balance comfort and energy saving. For example, different users have different preferences for parameters such as temperature and air volume; under different weather conditions and driving conditions, the user's demand for air conditioning will also change.

[0003] In order to improve the comfort and energy saving of the vehicle air conditioning system, some schemes attempt to predict the temperature control according to historical information. However, such schemes still give the predicted absolute value of the air conditioning temperature, while the vehicle air conditioning system is actually running in a non-steady state. Even under the same environment and the same user group, the historical use habits cannot ensure that the predicted air conditioning temperature can meet the comfort requirements of the passengers, and it will still be affected by many factors. Therefore, how to make accurate temperature adjustment on this basis to meet the individual comfort needs is a problem to be solved in the prior art.

[0004] It is to be noted that the information disclosed in the above background section is only for strengthening the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] In order to solve or at least alleviate one or more of the above problems, an air conditioner control method based on reinforcement learning, an air conditioner control device capable of implementing the method, a computer readable storage medium and a computer program product are provided, which utilize reinforcement learning to realize adaptive optimization of air conditioner control strategy, can accurately meet the individual needs of users and adapt to dynamic environment, thereby improving comfort and user satisfaction.

[0006] According to a first aspect of the present application, a reinforcement learning-based air conditioner control method is provided, which comprises the following steps: constructing a state vector reflecting a current situation based on user preference information, vehicle internal and external environment information, vehicle state information, and a user driving path; analyzing the state vector using a reinforcement learning model and generating bias information for adjusting air conditioner control instructions; modifying the current air conditioner control instructions using the bias information; and feeding back the user's adjustment behavior of the air conditioner settings to the reinforcement learning model as a reward signal to optimize the model parameters of the reinforcement learning model and the generated bias information.

[0007] Alternatively or additionally to the above solutions, the air conditioner control method according to an embodiment of the present application further comprises: in response to vehicle starting, obtaining vehicle model parameters, air conditioner control parameters, and the user preference information from the cloud; and collecting the vehicle internal and external environment information, the vehicle state information, and the user driving path in real time.

[0008] Alternatively or additionally to the above solutions, in the air conditioner control method according to an embodiment of the present application, modifying the current air conditioner control instructions using the bias information comprises: generating an initial air conditioner control instruction based on the vehicle model parameters and the air conditioner control parameters; and applying the bias information to the initial air conditioner control instruction to generate a modified air conditioner control instruction.

[0009] Alternatively or additionally to the above solutions, in the air conditioner control method according to an embodiment of the present application, the user preference information comprises user historical temperature settings, user historical air volume settings, user air conditioner usage frequency, and user preference for air conditioner modes.

[0010] Alternatively or additionally to the above solutions, in the air conditioner control method according to an embodiment of the present application, analyzing the state vector using a reinforcement learning model and generating bias information for adjusting air conditioner control instructions comprises: inputting the state vector into the reinforcement learning model; the reinforcement learning model determining an adjustment action for the air conditioner control instruction according to a learned strategy based on the current state vector; and converting the determined adjustment action into corresponding bias information.

[0011] Alternatively or additionally to the above solutions, in the air conditioner control method according to an embodiment of the present application, the bias information comprises a bias item and a bias value, wherein the bias item comprises one or more of the following: temperature adjustment direction, air volume adjustment direction, mode selection, and air supply area, and the bias value comprises one or more of the following: temperature adjustment amplitude, air volume adjustment level, mode selection probability, and area air supply proportion.

[0012] As an alternative or supplement to the above scheme, in the air conditioner control method according to an embodiment of the present application, the adjustment behavior of the user to the air conditioner setting is fed back to the reinforcement learning model as a reward signal to optimize the model parameters and the generated bias information of the reinforcement learning model, including: monitoring in real time whether the correction triggers the adjustment behavior of the user to the air conditioner setting; if the correction does not trigger the adjustment behavior, a positive reward is given; or if the correction triggers the adjustment behavior, a negative reward is given.

[0013] As an alternative or supplement to the above scheme, in the air conditioner control method according to an embodiment of the present application, the adjustment behavior of the user to the air conditioner setting is fed back to the reinforcement learning model as a reward signal to optimize the model parameters and the generated bias information of the reinforcement learning model, including: monitoring in real time the adjustment amplitude of the user to the air conditioner setting, wherein the greater the adjustment amplitude, the greater the absolute value of the reward signal.

[0014] According to a second aspect of the present application, there is provided an air conditioner control device, comprising: a memory; a processor; and a computer program stored on the memory and executable on the processor, wherein execution of the computer program causes any one of the methods according to the first aspect of the present application to be performed.

[0015] According to a third aspect of the present application, there is provided a computer readable storage medium, comprising instructions which, when executed, perform any one of the methods according to the first aspect of the present application.

[0016] According to a fourth aspect of the present application, there is provided a computer program product, comprising a computer program which, when executed by a processor, implements any one of the methods according to the first aspect of the present application.

[0017] The air conditioner control scheme according to one or more embodiments of the present application can more accurately capture the user's individualized needs and dynamic environmental changes, and realize adaptive optimization of the air conditioner control strategy, while improving comfort and reducing energy consumption, thereby overcoming the problem that existing schemes are difficult to balance individualization, environmental adaptability and adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and / or other aspects and advantages of the present application will become more apparent and more readily appreciated by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals refer to like elements throughout the drawings. In the drawings:

[0019] Figure 1 a schematic flowchart of an air conditioning control method 10 according to one or more embodiments of the present application;

[0020] Figure 2 a schematic flowchart of an air conditioning control method 20 according to one or more embodiments of the present application; and

[0021] Figure 3 a schematic block diagram of an air conditioning control apparatus 30 according to one or more embodiments of the present application. DETAILED DESCRIPTION

[0022] The following detailed description is merely exemplary in nature and is not intended to limit the disclosed technology or the application and uses of the disclosed technology. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background or the following detailed description.

[0023] In the following detailed description of embodiments, numerous specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it will be apparent to one of ordinary skill in the art having the benefit of the present disclosure that the disclosed technology can be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0024] The terms such as "comprise" and "include" mean that the technology of the present application does not exclude the presence of other elements not directly or explicitly mentioned in the specification, in addition to the elements and steps directly and explicitly mentioned in the specification. The terms such as "first" and "second" do not mean the order of the elements in time, space, size, etc., but are merely used to distinguish the elements. The technology of the present application is typically used for electric vehicles, which include, but are not limited to, battery electric vehicles (BEVs), hybrid electric vehicles (HEVs), fuel cell electric vehicles (FCEVs), etc.

[0025] Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings.

[0026] Reference will now be made to the drawings, wherein: Figure 1 a schematic flowchart of an air conditioning control method 10 according to one or more embodiments of the present application.

[0027] As Figure 1 shown in step 110, a state vector reflecting the current situation is constructed based on user preference information, vehicle interior and exterior environment information, vehicle state information, and a user driving route.

[0028] Step 110 is the basis for building a reinforcement learning-based air conditioning control scheme, which aims to create a state vector reflecting the current situation to provide a basis for subsequent intelligent decision-making. The so-called "state vector reflecting the current situation" refers to a highly generalized, multi-dimensional data set that can represent the current user's air conditioning environment containing personalized preferences. This vector considers multiple factors that affect user comfort, striving to comprehensively and accurately describe the user's personalized needs and environmental conditions at the current time. First, user preference information, which can include the user's historical habits during the use of the air conditioner, including but not limited to: user historical temperature settings, user historical air volume settings, user air conditioner usage frequency, user preferences for air conditioner modes, etc., reflecting the user's personalized needs. Second, vehicle internal and external environmental information, which can include vehicle interior temperature, humidity, air quality, and external weather conditions, light intensity, etc., all of which directly affect the user's demand for the air conditioning system. In addition, vehicle state information, such as vehicle speed, engine load, window state, etc., will indirectly affect air conditioning demand, for example, the temperature inside the vehicle may rise when driving at high speed. Finally, the user's driving path, such as the driving path, congestion, estimated arrival time, etc., can help the system predict future environmental changes in order to adjust the air conditioning settings in advance, for example, reducing air conditioning power when entering a congested section to save energy.

[0029] The construction of the state vector is the process of quantifying and integrating the above multiple dimensions of information, converting these information into numerical forms that reinforcement learning models can understand and process. Data processing techniques such as normalization, standardization, feature selection, etc. can be used for preprocessing to improve the training effect of the model.

[0030] In step 120, the reinforcement learning model is used to analyze the state vector and generate bias information for adjusting the air conditioning control instructions.

[0031] The reinforcement learning model, as a key component of step 120, functions to receive the state vector constructed in step 110 and, based on this vector, output bias information for adjusting the air conditioner control instruction through the learned strategy. Reinforcement learning is a machine learning method that learns and optimizes its own strategy through the interaction of an agent with the environment to obtain the maximum cumulative reward. In this scheme, the reinforcement learning model plays the role of the agent, and the vehicle air conditioning system and its environment constitute the environment of reinforcement learning. The agent observes the current situation (for example, the state vector), selects an action (for example, generates bias information), and obtains feedback from the environment (for example, the user's adjustment behavior), thereby continuously learning and optimizing its own strategy. The strategy here comes from both pre-training and real-time adjustment and optimization. The model is trained through a large amount of historical data to preliminarily master the basic rules of air conditioner control and user preferences, and the model continuously adjusts its parameters according to the feedback (for example, manual adjustment of air conditioner settings) in the actual use process of the user to adapt to the individual needs of the user and changes in the environment.

[0032] Specifically, step 120 can be further refined into the following sub-steps: first, input the state vector constructed in step 110 into the reinforcement learning model as the basis for the model to make decisions; second, the reinforcement learning model determines the adjustment action for the air conditioner control instruction based on the current state vector and according to the strategy it has learned, where the adjustment action refers to the adjustment scheme for the air conditioner control instruction, such as increasing or decreasing the temperature, increasing or decreasing the air volume, changing the air supply mode, etc.; finally, convert the determined adjustment action into corresponding bias information to specifically adjust the air conditioner control instruction. Exemplarily, the bias information can include two parts: bias item and bias value. The bias item is used to indicate the direction of adjustment, such as temperature adjustment direction (increase or decrease), air volume adjustment direction (increase or decrease), mode selection (cooling, heating, etc.), air supply area (head, foot, etc.). The bias value is used to indicate the amplitude of adjustment, such as temperature adjustment amplitude (in degrees Celsius or Fahrenheit), air volume adjustment level (increase or decrease several levels), mode selection probability (probability of selecting a certain mode), regional air supply proportion (proportion of air supply to different areas), etc. By converting the adjustment action into specific bias item and bias value, the air conditioner control instruction can be adjusted more finely, thereby better meeting the individual needs of the user.

[0033] In step 130, the current air conditioner control instruction is corrected using the bias information.

[0034] Step 130 aims to apply the bias information generated in step 120 to the current air conditioning control instruction, thereby generating the final air conditioning control instruction that can meet the user's individual needs. The "current air conditioning control instruction" refers to the original instruction that has not been modified by the bias information. It can be generated based on vehicle sensor data (such as an in-vehicle temperature sensor) and preset rules, or it can be manually set by the user. The bias information is an adjustment to these original instructions, which can fine-tune the air conditioning control instruction according to the user's preferences, environmental changes, and vehicle status.

[0035] Optionally, step 130 can be further divided into two sub-steps. First, based on vehicle model parameters and air conditioning control parameters, an initial air conditioning control instruction is generated. Vehicle model parameters include the thermodynamic characteristics of the vehicle, such as body structure, thermal insulation performance, etc., which affect the change of the in-vehicle temperature. Air conditioning control parameters include the refrigeration capacity, heating capacity, air volume, etc. of the air conditioning system, which determine the adjustment capability of the air conditioning system. Based on these parameters, a preliminary air conditioning control instruction that has not been individually adjusted (e.g., including target temperature, air volume, air supply mode, etc.) can be generated as the basis for subsequent modification. Second, the bias information generated in step 120 is applied to the initial air conditioning control instruction to generate a modified air conditioning control instruction. The bias information (e.g., including a bias term (indicating the adjustment direction) and a bias value (indicating the adjustment magnitude)) can fine-tune the initial air conditioning control instruction according to the user's preferences, environmental changes, and vehicle status. For example, if the target temperature of the initial air conditioning control instruction is 24 degrees Celsius, and the bias information indicates that it needs to be reduced by 2 degrees Celsius, the modified target temperature is 22 degrees Celsius. For example, for users who prefer a cooler environment, a negative temperature bias can be generated to lower the in-vehicle temperature. For users who feel dry when driving at high speed, a higher humidity bias can be generated to increase the in-vehicle humidity. In addition, the bias information can also be dynamically adjusted according to environmental changes, such as generating a larger negative temperature bias when the sun is strong to offset the warming effect of sunlight.

[0036] In step 140, the user's adjustment behavior on the air conditioning settings is fed back to the reinforcement learning model as a reward signal to optimize the model parameters of the reinforcement learning model and the generated bias information.

[0037] Step 140 is a key step to achieve continuous optimization of the model and improve the control effect. The core idea of this step is that the user's behavior is the ultimate standard for measuring the control effect of the model, and the user's satisfaction directly reflects the quality of the model. In order to achieve this goal, it is necessary to monitor in real time whether the modification of the air conditioning control instruction in step 130 triggers the user's adjustment behavior on the air conditioning settings.

[0038] Specifically, if the revised air conditioning control instruction can meet the user's demand, the user does not make any manual adjustment, which indicates that the control strategy of the model is effective, and a positive reward can be given to the reinforcement learning model at this time. On the contrary, if the user is not satisfied with the revised air conditioning control instruction and makes manual adjustment (for example, adjusts the temperature, air volume, etc.), which indicates that the control strategy of the model is deviated, a negative reward (i.e., punishment) can be given to the reinforcement learning model at this time.

[0039] In addition, in order to more accurately evaluate the control effect of the model, the adjustment amplitude of the user to the air conditioning setting can also be monitored in real time. The greater the adjustment amplitude, the greater the deviation between the model and the user's demand, and the absolute value of the reward signal should also be greater at this time. For example, if the user only adjusts the temperature slightly, a smaller punishment is given to the model; and if the user adjusts the temperature greatly, a larger punishment is given to the model.

[0040] By feeding back the adjustment behavior of the user to the air conditioning setting to the reinforcement learning model as a reward signal, the continuous learning and optimization of the model can be realized. The reinforcement learning model will adjust its model parameters and optimize the generation strategy of the bias information according to the received reward signal, so as to gradually master the individualized preferences of the user and generate air conditioning control instructions that are more in line with the user's demand, and finally realize individualized and intelligent air conditioning control.

[0041] Reference will be made below in continuation Figure 2 , Figure 2 to a schematic flowchart of an air conditioning control method 20 according to one or more embodiments of the present application.

[0042] In step 201, in response to the start of the vehicle, the vehicle model parameters, the air conditioning control parameters and the user preference information are obtained from the cloud. Exemplarily, when the vehicle starts, the system will actively connect to the cloud server. The vehicle model parameters describe the thermodynamic characteristics of the vehicle, such as the vehicle body structure, the heat insulation performance, the window area, etc., which are crucial for accurately predicting the temperature variation trend in the vehicle, and they will affect the efficiency of air conditioning refrigeration or heating and ultimately affect the user's comfort. Since different models of vehicles have different thermodynamic characteristics, it is necessary to obtain corresponding parameters for different vehicle models. The air conditioning control parameters describe the performance and characteristics of the air conditioning system, such as the refrigeration capacity, the heating capacity, the air volume adjustment range, etc., which determine the adjustment capability that the air conditioning system can provide, and they will affect the execution effect of the air conditioning control instruction. Different air conditioning systems may have different control parameters, so it is necessary to obtain corresponding parameters for different air conditioning systems.

[0043] In step 202, the vehicle internal and external environment information, vehicle state information and user driving path are collected in real time. The user driving path can be obtained through the navigation system of the vehicle or through the mobile phone application connected to the user. By analyzing the driving path, the system can predict the area to be entered (such as a congested section, a tunnel, etc.), and adjust the air conditioning setting in advance to improve the user's comfort and energy saving.

[0044] In step 203, a state vector reflecting the current situation is constructed based on the user preference information, vehicle internal and external environment information, vehicle state information and user driving path.

[0045] In step 204, the state vector is analyzed by using a reinforcement learning model, and bias information for adjusting the air conditioning control instruction is generated.

[0046] In step 205, the bias information is used to correct the current air conditioning control instruction.

[0047] In step 206, the user's adjustment behavior of the air conditioning setting is fed back to the reinforcement learning model as a reward signal to optimize the model parameters of the reinforcement learning model and the generated bias information.

[0048] Figure 3 A schematic block diagram of the air conditioning control device 30 according to one or more embodiments of the present application. The air conditioning control device 30 comprises a memory 310, a processor 320, and a computer program 330 stored on the memory 310 and executable on the processor 320, the execution of the computer program 330 causing the above-mentioned air conditioning control method 10 or 20 to be performed. Exemplarily, the air conditioning control device 30 can be part of an electronic control unit (ECU) of a car infotainment system, or a smart cockpit domain controller. Exemplarily, the air conditioning control device 30 can also be a control unit in other terminals (such as a smart phone or other edge devices with limited computing power) capable of communicating with the vehicle.

[0049] In addition, as described above, the present application can also be implemented as a computer readable storage medium, in which a program for causing a computer to execute the method described in any of the above embodiments is stored. Here, as a computer readable storage medium, various types of computer readable storage media such as disk (e.g., magnetic disk, optical disk, etc.), card (e.g., memory card, optical card, etc.), semiconductor memory (e.g., ROM, non-volatile memory, etc.), tape (e.g., magnetic tape, cassette tape, etc.) and the like can be used.

[0050] The present application can also be implemented as a computer program product comprising a computer program which, when executed by a processor, implements the program of the method described in any of the above embodiments.

[0051] The relevant user personal information that may be involved in the embodiments of the present application is strictly in accordance with the requirements of laws and regulations, and is processed based on the principles of legality, legitimacy and necessity, and the reasonable purpose of business scenarios, and the personal information that is actively provided by the user in the process of using the product / service or generated due to the use of the product / service, and authorized by the user. The user personal information processed by the present application will be different due to the specific product / service scenario, and the specific scenario of the user using the product / service shall prevail, which may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information with the highest diligence and obligation. The present application attaches great importance to the security of user personal information, and has taken security protection measures in accordance with industry standards, which are reasonable and feasible to protect the user's information, prevent unauthorized access, public disclosure, use, modification, damage or loss of personal information.

[0052] In applicable cases, various embodiments provided by the present application can be implemented using hardware, software, or a combination of hardware and software. Moreover, in applicable cases, various hardware components and / or software components set forth herein can be combined into a composite component including software, hardware and / or both without departing from the scope of the present application. In applicable cases, various hardware components and / or software components set forth herein can be divided into sub-components including software, hardware or both without departing from the scope of the present application. In addition, in applicable cases, it is contemplated that software components can be implemented as hardware components, and vice versa.

[0053] Software according to the present application, such as program code and / or data, can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more general purpose or special purpose computers and / or computer systems, networked and / or otherwise. In applicable cases, the order of the steps described herein can be changed, combined into composite steps and / or split into sub-steps to provide features described herein.

[0054] The embodiments and examples presented herein are provided to best illustrate the embodiments of the present application and its particular application. As such, the above description and examples should not be construed to limit the various aspects of the present application or to otherwise serve as a restriction on the general scope thereof.

Claims

1. A method for controlling an air conditioner based on reinforcement learning, the method comprising: The method comprises the following steps: Construct a state vector reflecting the current situation based on user preference information, vehicle internal and external environment information, vehicle status information, and user driving path; Analyzing the state vector using a reinforcement learning model and generating bias information for adjusting air conditioning control instructions; Using the bias information to modify the current air conditioning control instruction; and The user's adjustment behavior of the air conditioning setting is fed back to the reinforcement learning model as a reward signal to optimize the model parameters of the reinforcement learning model and the generated bias information.

2. The air conditioner control method according to claim 1, characterized by, The method further comprises: In response to vehicle startup, obtaining vehicle model parameters, air conditioning control parameters, and the user preference information from the cloud; and Collect the vehicle's internal and external environment information, the vehicle status information, and the user's driving path in real time.

3. The air conditioner control method according to claim 2, characterized by, Using the bias information to modify the current air conditioning control instruction includes: generating an initial air conditioning control instruction based on the vehicle model parameter and the air conditioning control parameter; The bias information is applied to the initial air-conditioning control instruction to generate a revised air-conditioning control instruction.

4. The air conditioner control method according to claim 1, characterized by, The user preference information includes the user's historical temperature settings, the user's historical air volume settings, the user's air conditioning usage frequency, and the user's preference for air conditioning modes.

5. The air conditioner control method according to claim 1, characterized by, Analyzing the state vector using a reinforcement learning model and generating bias information for adjusting the air conditioning control command includes: Inputting the state vector into the reinforcement learning model; The reinforcement learning model determines an adjustment action for the air conditioning control instruction based on the current state vector and the learned strategy; and The determined adjustment action is converted into corresponding bias information.

6. The air conditioner control method according to claim 1, characterized by, The bias information includes a bias item and a bias value, wherein: The bias items include one or more of the following: temperature adjustment direction, air volume adjustment direction, mode selection, air supply area, The bias value includes one or more of the following: temperature adjustment range, air volume adjustment level, mode selection probability, and regional air supply ratio.

7. The air conditioner control method according to claim 1, characterized by, Feeding back the user's adjustment behavior of the air conditioning setting as a reward signal to the reinforcement learning model to optimize the model parameters of the reinforcement learning model and the generated bias information includes: monitoring in real time whether the correction triggers the user to adjust the air conditioning settings; If the modification does not trigger the adjustment behavior, a positive reward is granted; or If the correction triggers the adjustment behavior, a negative reward is granted.

8. The air conditioner control method according to claim 7, characterized by, Feeding back the user's adjustment behavior of the air conditioning setting as a reward signal to the reinforcement learning model to optimize the model parameters of the reinforcement learning model and the generated bias information also includes: The adjustment range of the air conditioner setting by the user is monitored in real time, wherein the greater the adjustment range, the greater the absolute value of the reward signal.

9. An air conditioner control device characterized by comprising: The method comprises: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein the execution of the computer program causes the method according to any one of claims 1 to 8 to be performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises instructions that, when executed, perform the method according to any one of claims 1-8.

11. A computer program product, characterised in that, A computer program comprising computer program elements, which, when executed by a processor, carry out the method according to any one of claims 1-8. A computer program comprising computer program elements, which, when executed by a processor, carry out the method according to any one of claims 1-8.

Images