Thermal comfort two-stage optimization evaluation method based on multi-modal fusion

Abstract

The invention discloses a thermal comfort two-stage optimization evaluation method based on multi-modal fusion. In the first-stage optimization, an attention mechanism is introduced into a general thermal comfort model formed by a neural network input by face infrared thermal imaging for improvement; and automatically extracting personalized feature genders in the RGB image by using a target detection algorithm, and carrying out multi-modal fusion on the personalized feature genders and temperature features synchronously extracted from the infrared image by the improved neural network model in the first optimization stage in an end-to-end manner. According to the method, the prediction deviation of a general model among member groups participating in training is solved, the accuracy of the thermal comfort model among specific training members and the generalization ability of the thermal comfort model for new individuals are effectively improved, frequent and high-cost updating of the model is avoided, and the thermal comfort model is further optimized. A new method is provided for multi-modal feature fusion in the field of thermal comfort evaluation, and a feasible technical scheme is provided for practical application of a personal thermal comfort system due to the lightweight feature of the method.

Description

Technical Field

[0001] The present invention relates to the technical field of building thermal comfort evaluation, and particularly relates to a two-stage optimized evaluation method for thermal comfort based on multimodal fusion. Background Technique

[0002] More than 80% of modern people's time is spent indoors. The thermal comfort of the indoor environment directly affects people's physical health, mental state, and work performance. As the core equipment for regulating indoor temperature, heating, ventilation, and air conditioning (HVAC) is the main equipment for adjusting the indoor thermal environment to meet thermal comfort. However, traditional HVAC systems usually rely on set-point control, which not only causes discomfort for most users but also results in energy waste. Accurately predicting the thermal sensation of occupants and using it as a benchmark for HVAC system control can create a healthy and comfortable personalized indoor environment while minimizing energy consumption as much as possible. The PMV model, which is widely used to evaluate thermal comfort, was proposed in the 1970s (Fanger P O, Breum N, Jerking E. Can colour and noise influence man's thermal comfort?[J]. Ergonomics, 1977, 20(1): 11-18). However, this model is mainly based on environmental parameter calculations and does not consider individual differences, so it cannot accurately reflect the true thermal sensations of each user in practical applications. In contrast, the PCM model (Personal thermal comfort model) based on artificial intelligence and data-driven provides a more accurate personalized thermal comfort assessment. Therefore, considering individual physiological parameters and personalized characteristics as the main indicators of thermal sensation will be more conducive to meeting the needs of specific users. There are diverse acquisition methods for individual physiological parameters and personalized characteristics. Among them, the acquisition of individual physiological parameters mainly includes contact and non-contact methods. The local skin temperature of the human body is collected through contact methods such as thermocouples or wearable bracelets to establish a high-performance thermal comfort model. Although skin temperature data can be directly obtained, it requires close contact with the human skin, which will seriously affect user comfort. In contrast, non-contact measurement methods, especially facial temperature monitoring based on infrared imaging technology, have become a more attractive option. By extracting the temperature of a fixed area of the infrared face and sorting the feature importance, the prediction accuracy can be improved. Similarly, in the modeling of personalized thermal comfort, the influence of personalized characteristics cannot be ignored. With the emergence of machine vision, the intelligent extraction of personalized characteristics has also attracted people's attention. Therefore, collecting the key individual physiological parameters and personalized characteristics required for thermal comfort assessment through non-contact means based on artificial intelligence has become the current mainstream method. With the increasing maturity of non-contact acquisition technology, significant progress has also been made in the construction of thermal comfort assessment models.In the research of the thermal comfort model through infrared images rich in facial temperature information, although the method of extracting skin temperature in key areas has shown significant potential in thermal comfort evaluation, any deviation in the multi-stage data processing process will lead to the risk that the output of the thermal comfort model does not match the actual situation. In contrast, the method of comprehensively learning infrared image data through end-to-end networks such as deep neural networks (DNNs) can abandon the manual feature extraction link and automatically extract feature points related to thermal sensation in infrared images.

[0003] However, there are the following technical problems in the existing technologies: First, the current thermal comfort model evaluation method usually only considers the overall prediction effect and ignores the application deviation of the model among members, and may not accurately reflect the thermal comfort needs of each individual in practice. Second, these deep learning models usually require a large amount of computing resources and the training process is time-consuming. Facing the frequent update tasks when new individuals are added, the application cost of the general model will increase. In addition, a large number of studies have proven that the fusion of personalized features and physiological factors is effective in improving the prediction effect of the thermal comfort model, but there is a gap in the technology of synchronously extracting personalized features and physiological factors from multi-modalities in a non-contact manner for fusion.

[0004] In summary, the existing thermal comfort models cannot meet the accurate prediction requirements of thermal comfort personalization in actual scenarios. Summary of the Invention

[0005] The purpose of the present invention is to propose a two-stage optimization evaluation method for thermal comfort based on multi-modal fusion. In the first stage, the generalization ability of the general model is effectively improved by improving the optimized attention mechanism of the general model for facial infrared images, avoiding the high-cost update of the model when facing new members. In the second stage, the YOLOv5 algorithm for object detection is used to automatically extract personalized features (gender) in RGB images and fuse them with temperature features in infrared images to further improve the generalization ability of the general model.

[0006] The present invention is implemented through the following technical solutions.

[0007] A two-stage optimization evaluation method for thermal comfort based on multi-modal fusion according to the present invention includes the following steps.

[0008] Step S1, Thermal comfort data collection: Design a thermal comfort experiment to collect facial infrared thermograms and individual subjective thermal comfort votes of volunteers. After processing the outliers of the infrared image data and thermal comfort votes, continue to label cold (-1), medium (0), and hot (1) tags one by one.

[0009] Step S2, Production of thermal comfort model dataset: The collected data above is first divided into a training member group and a test member group according to individuals, and then the data in the training member group is divided into a training set, a validation set, and a test set according to a ratio.

[0010] Step S3, Extraction of thermal comfort features in the infrared modality: The color change of pixels related to thermal comfort in the facial infrared thermal imaging in the training set is extracted by selecting a neural network (taking EfficientNet B1 as an example) to construct a preliminary thermal comfort model.

[0011] Step S4, Optimization of one-stage single-modal attention: The attention mechanism CBAM is introduced to improve the EfficientNet B1 network to construct a CBAM-EfficientNet B1 model that can capture the subtle changes in the thermal comfort state.

[0012] Step S5, Extraction of personalized features in the RGB modality: The gender features extracted by using the YOLOv5 algorithm in the RGB layer are used to obtain the classification results.

[0013] Step S6, Optimization of two-stage multi-modal fusion: The gender classification results are input into the CBAM-EfficientNet B1 model, and then a multi-modal thermal comfort model Gender+CBAM-EfficientNet B1 that distinguishes genders is optimized.

[0014] Step S7, Optimization of thermal comfort model evaluation: The two-stage optimized models are respectively evaluated for performance in the training member group and the test member group.

[0015] Further, in the step S2, when dividing the training member group and the test member group according to individuals, a rotation strategy will be used to ensure that each of the above volunteers becomes a test individual once to meet the individual cross-validation condition and prove the effectiveness of the optimized model.

[0016] Further, in the step S3, the EfficientNet neural network used is used to verify the effectiveness of the two-stage optimization evaluation method for the thermal comfort of multi-modal fusion, and it is also effective for optimizing other neural network models, such as the ResNet network and the MobileNet network.

[0017] Further, in the step S4, the CBAM attention mechanism is composed of a spatial attention SAM and a channel attention CAM. The CBAM-EfficientNet B1 model replaces the SE modules in the MBConv1 convolutional layer and the MBConv6 convolutional layer in the preliminary thermal comfort model with CBAM modules to provide more comprehensive feature weighting. The multi-dimensional weighting method enables the model to have more fine-grained feature selection on the infrared images of the face, highlighting the specific thermal regions of the human face while ignoring those unimportant backgrounds or irrelevant regions to improve the model performance.

[0018] Further, in the step S5, the gender discrimination results extracted from the RGB layer are output to the overall facial infrared training dataset to divide the facial infrared images by gender.

[0019] Further, in the step S6, the Gender+CBAM-EfficientNet B1 multi-modal thermal comfort model is composed of a male model and a female model under gender discrimination. Its application method requires visually extracting the personalized feature gender from the RGB layer, activating the thermal comfort model corresponding to the feature to wait for the input of the infrared image, and finally outputting an accurate thermal comfort judgment.

[0020] Further, in the steps S4 and S6, the two-stage optimization process is used to make up for the prediction deviation between training groups of the model and strengthen the generalization ability of the model, so as to solve the shortcomings of the current comprehensive thermal comfort model that can only be effective for a fixed group and the performance limitation of new members.

[0021] Further, in the step S6, the RGB modality and the infrared modality of the Gender+CBAM-EfficientNet B1 multi-modal thermal comfort model respectively adopt the lightweight network YOLOv5s and EfficientNet B1 as the basic networks, providing a feasible technical solution for the practical application of the personal thermal comfort system.

[0022] Further, in the step S7, the thermal comfort CBAM-EfficientNet B1 model composed of the infrared single modality in the optimization process can adjust the samples of the training individual group according to the prediction performance of specific individuals to pursue the best thermal comfort prediction for users.

[0023] The present invention has at least the following technical effects or advantages:

[0024] (1) The present invention proposes an optimization method for an end-to-end general thermal comfort evaluation model, which introduces the attention mechanism CBAM to improve the EfficientNet B1 network. It solves the prediction deviation among the participating training member groups of the general model, effectively improves the accuracy of the thermal comfort model among specific training members and the generalization ability for new individuals, and avoids frequent and costly updates of the model. It provides theoretical support for the construction of personalized thermal comfort evaluation models and new solutions for cross-individual needs.

[0025] (2) The present invention proposes a feature fusion method for a multi-modal thermal comfort model, which uses the object detection algorithm YOLOv5 to automatically extract personalized features in RGB images and deeply fuses them with the infrared image physiological feature extraction model CBAM-EfficientNet B1 to further optimize the thermal comfort model. It verifies the improvement of the thermal comfort model in individual generalization ability under gender fusion. It provides a new methodology for multi-modal feature fusion in the field of thermal comfort evaluation, and its lightweight feature also provides a feasible technical solution for the practical application of personal thermal comfort systems (PCS).

[0026] (3) The present invention proposes a method for selecting the thermal comfort model with the best prediction ability for new individuals, providing a theoretical basis for the algorithm research of matching suitable training individual data for users. It provides a development idea for the application of thermal comfort models in groups that cannot normally feedback thermal sensations (patients, infants, etc.). At the same time, it provides inspiration for the design of individual thermal comfort products that can adapt to more new users in temporarily occupied spaces, and provides ideas for more detailed thermal comfort prediction and application in multi-person scenarios with strong mobility such as hotels, restaurants, bookstores, and hospitals with high comfort requirements. Description of the Drawings

[0027] Figure 1 It is the overall framework flowchart of a two-stage optimization evaluation method for thermal comfort based on multi-modal fusion according to the present invention.

[0028] Figure 2 It is the design diagram of the data experiment time points of the two-stage optimized thermal comfort model according to the present invention.

[0029] Figure 3 It is the two-stage optimization strategy diagram of the multi-modal fusion thermal comfort model according to the present invention.

[0030] Figure 4 It is the schematic diagram of the rotation strategy of 5-fold cross-validation according to the present invention.

[0031] Figure 5 It is the accuracy schematic diagram of the two-stage optimization process of the multi-modal fusion thermal comfort model according to the present invention. Detailed implementation manners

[0032] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and through specific implementation methods.

[0033] A two-stage optimization evaluation method for thermal comfort based on multimodal fusion described in this embodiment includes the following steps.

[0034] Step S1, design an experiment to collect data required for thermal comfort evaluation. Recruit at least 20 volunteers, with an equal number of men and women of similar ages, and then conduct a formal experiment in pairs according to the experimental process as Figure 2 follows. This includes collecting human physiological data, and collecting air temperature and humidity at multiple locations. Among them, use infrared thermal imaging to record the RGB modality and infrared modality of the human face by taking pictures every 20 s for 20 s, and require a vote on thermal comfort every 3 minutes to meet the consistency in time sequence. After processing the abnormal values of the infrared image data and thermal comfort votes, then continue to label the cold (-1), medium (0), and hot (1) labels one by one. The final data distribution results are shown in Table 1.

[0035] Table 1 The amount of different thermal comfort voting data

[0036] Label name Cold (-1) Medium (0) Hot (1) Quantity (sheets) 2222 2401 2070

[0037] Step S2, production of the thermal comfort model data set. First divide the collected data into a training member group and a test member group according to a ratio of 4:1 of the total number of individuals, and then divide the data in the training member group into a training set, a validation set, and a test set according to a ratio of 7:1:2 to evaluate the prediction effect. When dividing the training member group and the test member group according to individuals, the rotation strategy of 5-fold cross-validation will be used. This strategy is as Figure 4 shown, ensuring that each of the above-mentioned volunteers becomes a test individual once to fully prove the effectiveness of the model performance improvement under the two-stage optimization.

[0038] Step S3, extraction of thermal comfort features in the infrared modality. Select a neural network (taking EfficientNet B1 as an example) to extract the pixel color changes related to thermal comfort in the facial infrared thermal imaging in the training set, and construct a preliminary thermal comfort model. In this process, first adjust the image to the input size of EfficientNet B1, and the adjustment formula is (1), I resized_IR is the adjusted image, and I input_IR represents the input image.

[0039] I resized_IR = Resize(I input_IR , (240, 240)) (1)

[0040] Then, features are extracted through multiple convolutional layers and MBConv modules. The output feature map F of the l-th layer l can be expressed as (2)

[0041] F l = MBConv(F l-1 , W l ) (2)

[0042] where W l is the weight of the l-th layer, and F l-1 is the output of the previous layer. After the extraction of thermal comfort features from the infrared image, global average pooling is performed to convert the feature map F l into a feature vector f, which is expressed by the formula (3)

[0043] f = GAP(F l ) (3)

[0044] Then, the feature vector f is input into the fully connected layer to obtain the thermal comfort feature h, as shown in Equation (4). Among them, W fc and b fc are the weight and bias of the fully connected layer, respectively.

[0045] h = W fc ·f + b fc (4)

[0046] Finally, the thermal comfort feature h is used to construct a preliminary thermal comfort model to be optimized, and the thermal comfort prediction y is finally output. Among them, W out and b out are the weight and bias of the output layer, and σ is the Swish activation function.

[0047] y = σ(W out ·h + b out ) (5)

[0048] Step S4, one-stage single-modal attention optimization:

[0049] The attention mechanism CBAM is introduced to improve the EfficientNet B1 network to construct a CBAM-EfficientNet B1 model that can capture subtle changes in the thermal comfort state. The CBAM attention mechanism is composed of a spatial attention SAM and a channel attention CAM. The formula for the weighted output of the feature map of the CAM module is (6)

[0050]

[0051] The SAM module performs pooling operations on the input feature map in the channel dimension, usually using average pooling (AvgPool) and max pooling (MaxPool). Average pooling calculates the average value of all channels at each spatial position, and then max pooling selects the maximum value at each spatial position across all channels. This is done to extract global spatial information from each pixel position. The formula for the output of the weighted feature map of this module is (7)

[0052]

[0053] The comprehensive calculation formula of CBAM first performs channel attention and then spatial attention operations to finally generate a weighted feature map. The calculation process can be expressed as (8)

[0054]

[0055] where F is the given input feature map, F′ is the output of the channel attention module which is the weighted feature map, F″ is the output of the spatial attention module which is the weighted feature map, and M c is a multi-layer perceptron (MLP) shared by the input, which obtains the channel attention weights through summation, and M s obtains the spatial attention weights through the Sigmoid activation function, represents an element-wise multiplication operation.

[0056] As Figure 3 In PART1, the CBAM-EfficientNet B1 model replaces the SE modules in the MBConv1 and MBConv6 convolutional layers of the preliminary thermal comfort model with CBAM modules, improving the feature extraction method in Equation (2) to provide more comprehensive feature weighting. The multi-dimensional weighting method enables the model to have more fine-grained feature selection on the infrared images of the face, highlighting specific thermal regions of the human face while ignoring unimportant backgrounds or irrelevant regions to improve the model performance.

[0057] Step S5, RGB modality personalized feature extraction:

[0058] The gender features extracted using the YOLOv5 algorithm in the RGB layer are used to obtain the classification results. As Figure 3 shown in PART2, the gender discrimination results extracted from the RGB layer are output to the overall face infrared training dataset, and the face infrared image data is divided using gender as a label, while keeping the data quantities of the two datasets relatively balanced.

[0059] Step S6, two-stage multi-modal fusion optimization:

[0060] The gender classification result is input into the CBAM-EfficientNet B1 model, and then the multi-modal thermal comfort model Gender+CBAM-EfficientNet B1 that distinguishes genders is optimized. As Figure 3 shown in PART3, the Gender+CBAM-EfficientNet B1 multi-modal thermal comfort model consists of a male model and a female model under gender distinction. Its application method requires visually extracting personalized feature genders from the RGB layer, then activating the thermal comfort model corresponding to the feature to wait for the input of the infrared image, and finally outputting an accurate thermal comfort judgment. In the two-stage optimization process, the prediction deviation between training groups of the model is compensated and the generalization ability of the model is strengthened to make up for the shortcomings of the current comprehensive thermal comfort model, which can only be effective for a fixed group and has limitations in the performance of new members.

[0061] Step S7, optimizing the evaluation of the thermal comfort model:

[0062] The two-stage separately optimized models are evaluated for performance in the training group of members and the test group of members. The performance accuracy evaluation results are as Figure 5 shown. In the optimization process, the thermal comfort CBAM-EfficientNet B1 model composed of infrared single modality can adjust the samples of the training individual group according to the predicted performance of specific individuals to pursue the best thermal comfort prediction for users.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A two-stage optimization evaluation method for thermal comfort based on multimodal fusion, characterized in that It includes the following steps: Step S1, Thermal comfort data collection: Design a thermal comfort experiment to collect the facial infrared thermal images and individual subjective thermal comfort votes of volunteers. After processing the outliers in the infrared image data and thermal comfort votes, continue to label cold, medium, and hot tags one by one in correspondence; Step S2, Thermal comfort model dataset production: First divide the collected data into a training member group and a test member group according to individuals, and then divide the data in the training member group into a training set, a validation set, and a test set according to a ratio; Step S3, Infrared modality thermal comfort feature extraction: By selecting the neural network EfficientNet B1, extract the pixel color changes related to thermal comfort in the facial infrared thermal images in the training set, and construct a preliminary thermal comfort model; Step S4, First-stage single-modal attention optimization: Introduce the attention mechanism CBAM to improve the neural network EfficientNet B1 to construct the CBAM-EfficientNetB1 model that can capture the subtle changes in the thermal comfort state; Step S5, RGB modality personalized feature extraction: Use the YOLOv5 algorithm to extract gender features in the RGB layer to obtain the classification results; Step S6, Second-stage multi-modal fusion optimization: Input the gender classification results into the CBAM-EfficientNetB1 model, and then optimize the multi-modal thermal comfort model Gender+CBAM-EfficientNet B1 that distinguishes genders; Step S7, Optimize the thermal comfort model evaluation: Evaluate the performance of the two-stage optimized models in the training member group and the test member group.

2. The two-stage optimized evaluation method for thermal comfort based on multimodal fusion according to claim 1, characterized in that In step S2, when dividing the training member group and the test member group according to individuals, use the rotation strategy to ensure that each of the above-mentioned volunteers becomes a test individual once to meet the individual cross-validation conditions and prove the effectiveness of the optimized model.

3. The two-stage optimization evaluation method for thermal comfort based on multimodal fusion according to claim 1, characterized in that In step S4, the CBAM attention mechanism is composed of a spatial attention SAM and a channel attention CAM. The CBAM-EfficientNet B1 model replaces the SE modules in the MBConv1 convolutional layer and the MBConv6 convolutional layer in the preliminary thermal comfort model with CBAM modules.

4. A two-stage optimization evaluation method for thermal comfort based on multimodal fusion according to claim 1, characterized in that In step S5, the gender discrimination results extracted from the RGB layer will be output to the overall facial infrared training dataset to divide the facial infrared images by gender.

5. A two-stage optimization evaluation method for thermal comfort based on multimodal fusion according to claim 1, characterized in that the In step S6, the Gender+CBAM-EfficientNet B1 multi-modal thermal comfort model is composed of a male model and a female model under gender discrimination. After visually extracting the personalized feature gender from the RGB layer, activate the thermal comfort model corresponding to the feature to wait for the input of the infrared image, and finally output an accurate thermal comfort judgment.

6. The two-stage optimized evaluation method for thermal comfort based on multimodal fusion according to claim 1, wherein the In step S6, the RGB modality and the infrared modality of the Gender+CBAM-EfficientNet B1 multi-modal thermal comfort model respectively use the lightweight network YOLOv5s and EfficientNet B1 as the basic networks.

7. A two-stage optimization evaluation method for thermal comfort based on multi-modal fusion according to claim 1, characterized in that In step S7, during the optimization process, the thermal comfort CBAM-EfficientNet B1 model composed of infrared single modality adjusts the samples of the training individual group according to the prediction performance of specific individuals to pursue the best thermal comfort prediction for users.

8. A two-stage optimization evaluation method for thermal comfort based on multimodal fusion according to claim 1, characterized in that In step S3, the neural network is a ResNet network or a MobileNet network.

Images