Multi-modal body language feature recognition and fusion based thermal comfort monitoring robot system and method

Abstract

The application discloses a thermal comfort monitoring robot system and method based on multi-modal body language feature recognition and fusion, and realizes individual thermal comfort monitoring and regulation in a dynamic indoor environment. The intelligent mobile platform is equipped with multiple types of environmental sensors and visual acquisition equipment, autonomously navigates and collects environmental parameters and human image data; a multi-modal feature recognition module is based on a deep learning network, performs human detection, posture recognition and identity maintenance on video streams frame by frame, and extracts multi-modal human features; a data processing and fusion module realizes cross-frame identity consistency through an algorithm, and fuses multi-modal human features and environmental parameters; a thermal comfort degree calculation module is based on a predicted average voting model, dynamically adjusts the metabolic rate parameter and incorporates a psychological correction factor, and calculates an individual thermal comfort index; a three-dimensional visualization and regulation module generates a comfort degree distribution map, and interfaces with a building environment automatic control system to realize dynamic regulation, and provides support for intelligent building environment management.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent buildings and indoor robots, in particular to a thermal comfort monitoring robot system and method based on multi-modal body language feature recognition and fusion. BACKGROUND

[0002] Indoor thermal comfort is an important part of building environmental quality, and has a direct impact on the health, work efficiency and life satisfaction of users. Studies have shown that more than 80% of human time is spent in indoor environments, and thermal environmental factors such as temperature, humidity, wind speed and radiant temperature have a significant effect on human comfort perception. Therefore, how to scientifically and accurately monitor and evaluate indoor thermal comfort has been a research focus in the fields of building environment and HVAC control for a long time.

[0003] The most widely used evaluation method at present is based on the predicted mean vote model. This model calculates the overall thermal comfort level of the group by measuring parameters such as temperature, humidity, wind speed, clothing thermal resistance and human metabolic rate. Although this method plays an important role in standardized evaluation, it is essentially a static model that is difficult to adapt to the complex scenarios of diverse personnel behavior and frequent state changes in actual indoor environments. At the same time, the PMV model mainly predicts based on group averages, ignoring individualized comfort differences caused by differences in gender, age, clothing, posture, emotion, etc. In addition, traditional methods usually rely on fixed-position environmental sensors, and the data obtained can only reflect the local point situation, and cannot reveal the distribution characteristics of the thermal environment in the entire space. Combined with the limited sampling frequency and lack of real-time performance, it is even more difficult to meet the needs of dynamic comfort regulation.

[0004] With the development of artificial intelligence, computer vision and mobile robot technology, researchers have begun to try to apply human feature recognition and multi-sensor fusion to thermal comfort monitoring. For example, mobile robots equipped with temperature and humidity sensors and cameras can improve the flexibility and coverage of data collection to some extent. However, existing methods still face many challenges in application: in complex environments with multiple people present, the recognition results are easily disturbed by personnel occlusion, posture overlap and changes in lighting, resulting in insufficient accuracy; at the same time, most existing researches still remain in the two-dimensional detection and static modeling stage, lacking three-dimensional dynamic visualization capabilities, and cannot provide intuitive and scientific decision-making basis for comfort management and energy optimization.

[0005] Therefore, the existing thermal comfort monitoring methods have obvious deficiencies in dynamicity, personalization, spatialization and real-time performance, and are difficult to meet the actual needs of intelligent building and green energy development.

[0006] It is to be understood that the information disclosed in the background section is only for the purpose of providing an understanding of the present application, and thus, can include information that is not prior art to those skilled in the art. SUMMARY

[0007] The main purpose of the present application is to overcome the defects existing in the background art, provide a thermal comfort monitoring robot system and method based on multi-modal body language feature recognition and fusion, realize real-time, accurate and personalized monitoring and visualized regulation of individual thermal comfort state in dynamic and multi-person indoor environment, so as to overcome the limitations of static, group and single-point measurement of traditional methods.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] In the first aspect of the present application, a thermal comfort monitoring robot system based on multi-modal body language feature recognition and fusion comprises:

[0010] An intelligent mobile platform carries multiple types of environmental sensors and visual acquisition equipment, and is used for autonomous navigation and acquisition of environmental parameters and human image data in a dynamic indoor environment;

[0011] A multi-modal feature recognition module performs frame-by-frame human detection, posture recognition and identity maintenance on a video stream based on a deep learning network, and extracts multi-modal human features such as gender, clothing category, thermal resistance, emotional state and behavior pattern;

[0012] A data processing and fusion module realizes cross-frame identity consistency maintenance through multi-stage identity matching and time sequence correction algorithm, and realizes cross-modal alignment and fusion of multi-modal human features and environmental parameters;

[0013] A thermal comfort degree calculation module introduces individual clothing thermal resistance parameters estimated by a clothing recognition module, combines posture and behavior recognition to estimate individual metabolic rate, dynamically adjusts individual metabolic rate parameters used in the model, and incorporates emotional state recognition results as a psychological correction factor into the model to dynamically calculate individual thermal comfort index;

[0014] A three-dimensional visualization and regulation module maps the thermal comfort index to a three-dimensional human mesh and a space map, generates individual and space comfort degree distribution map, and realizes dynamic regulation of comfort-oriented building environment automatic control system.

[0015] In the second aspect of the present application, a thermal comfort monitoring method based on multi-modal body language feature recognition and fusion uses the system, and the method comprises:

[0016] Autonomously cruise in a dynamic indoor environment through an intelligent mobile platform, and synchronously acquire environmental parameters and human image data;

[0017] Human body detection, posture recognition and cross-frame identity maintenance are performed on the video stream frame by frame, and gender, clothing category, thermal resistance, emotional state and behavior mode multi-modal human body features are extracted;

[0018] Identity consistency is maintained through multi-stage identity matching and timing correction, multi-modal human body features and environmental parameters are time-stamped and spatially aligned, and a fusion data vector is constructed;

[0019] Based on the predicted average voting model, the individual clothing thermal resistance parameters estimated by the clothing recognition module are introduced, the individual metabolic rate estimated by the posture and behavior recognition is combined, the individual metabolic rate parameters used in the model are dynamically adjusted, and the emotional state recognition result is taken as a psychological correction factor and is included in the model, and the individual thermal comfort index is dynamically calculated;

[0020] The thermal comfort index is mapped to a three-dimensional human body grid and a spatial map to generate individual and spatial comfort distribution maps;

[0021] The comfort data is fed back to the building environment automatic control system to realize dynamic regulation and control of comfort-oriented.

[0022] In a third aspect of the present application, a computer program product comprises a computer program which, when executed by a processor, implements the multi-modal body language feature recognition and fusion-based frame-by-frame processing thermal comfort monitoring method.

[0023] The present application has the following beneficial effects:

[0024] The present application is directed to the field of intelligent buildings and indoor environment control, and proposes a thermal comfort monitoring robot system and method based on multi-modal body language feature recognition and fusion. Specifically, the present application relies on a mobile robot platform, integrates deep learning, multi-sensor perception and three-dimensional modeling technology, and can realize real-time evaluation and spatial visualization of individual thermal comfort state in a complex, multi-person dynamic indoor environment; the system realizes multi-human posture fusion and identity maintenance across frames, effectively overcoming problems such as personnel occlusion and overlap, greatly improving the stability and accuracy of recognition; at the same time, the system comprehensively utilizes multi-modal human body features such as gender, clothing thermal resistance, emotion, behavior, and combines environmental parameters such as temperature, humidity, wind speed and CO2 concentration to construct a dynamic predicted mean vote model (PMV), significantly improving the accuracy of personalized thermal comfort estimation; on this basis, combined with three-dimensional human body grid modeling and SLAM spatial mapping, a multi-human comfort three-dimensional distribution map is generated to realize dynamic visualization and fine management of indoor thermal comfort state.

[0025] The application not only breaks through the limitations of traditional single-point measurement and static model in thermal comfort evaluation, but also realizes multi-human frame-by-frame dynamic monitoring and three-dimensional comfort visualization based on a robot platform for the first time, with the advantages of strong real-time performance, high accuracy, good adaptability and strong engineering feasibility; its application scenarios cover public buildings such as office buildings, research institutions, shopping malls, schools and airports, which can provide scientific basis for intelligent control of heating, ventilation and air conditioning systems, and has significant energy-saving potential and industrialization promotion value. Further, the application realizes frame-by-frame identification of multiple people, personalized thermal comfort calculation and three-dimensional dynamic visualization by integrating deep learning, multi-modal human feature recognition, multi-sensor environment perception and mobile robot platform, thereby providing more scientific and efficient technical support for intelligent building environment management and energy-saving regulation.

[0026] By integrating multi-sensor perception, deep learning recognition and three-dimensional modeling technology on a mobile robot platform, the application can realize frame-by-frame identification and cross-frame identity maintenance of multiple people in a complex indoor environment, effectively solving the problems of low recognition accuracy and poor stability caused by personnel occlusion, posture overlap and light changes in traditional methods; at the same time, the application dynamically fuses multi-modal features such as gender, clothing thermal resistance, posture, behavior and emotion with environmental parameters, breaks through the limitations of traditional prediction average voting model relying only on average parameters, and establishes a more accurate individualized thermal comfort calculation model; in addition, through three-dimensional human mesh modeling and space mapping, the application can generate a thermal comfort three-dimensional distribution map in real time, realizing the leap from single-point measurement to spatial dynamic visualization.

[0027] As a new type of intelligent monitoring method, the application establishes an innovative multi-modal body language integrated real-time thermal comfort monitoring (MRTM) system, which can realize real-time, dynamic and visual evaluation of multiple human thermal comfort states; the system not only improves the accuracy and individualization level of comfort prediction, but also provides fine-tuned basis for intelligent building heating, ventilation and air conditioning system regulation, thereby effectively promoting energy saving and green building development.

[0028] Compared with existing thermal comfort monitoring schemes based on single-point collection and static model prediction, the application has the following significant innovative advantages:

[0029] 1. Realizes frame-by-frame identification and cross-frame identity maintenance of multiple people in complex dynamic scenes, which can effectively overcome the identification errors caused by personnel occlusion, posture overlap and light changes, and greatly improve the stability and accuracy of real-time monitoring;

[0030] 2. Introducing multi-modal features such as gender, clothing thermal resistance, posture behavior, emotional state, and fusing them with environmental parameters to construct a dynamic thermal comfort calculation model, breaking through the limitations of traditional PMV model relying only on average parameters, and realizing individualized and differentiated comfort evaluation;

[0031] 3. Through three-dimensional human body grid modeling and space mapping, three-dimensional dynamic visualization of multi-human thermal comfort state is realized for the first time, which can intuitively present individual comfort state and form overall space comfort degree distribution diagram, thereby providing scientific and intuitive decision basis for building environment regulation and control;

[0032] 4. Based on a mobile robot platform and a lightweight deep learning model, the system has good real-time performance and deployability, can run without expensive hardware support, and has high engineering and industrial application potential.

[0033] In summary, the present application provides a comprehensive solution that can accurately and real-time perceive and visualize individual thermal comfort state, and dynamically and individually regulate and control building environment according to the individual thermal comfort state, thereby improving indoor environment quality and facilitating energy saving and consumption reduction.

[0034] Other benefits of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 Fig. 1 is a structural schematic diagram of a frame-by-frame processing thermal comfort monitoring robot system of the present application based on multi-modal body language feature recognition and fusion;

[0036] Figure 2 Fig. 2 is a technical process schematic diagram of the system of the present application;

[0037] Figure 3 Fig. 3 is a human multi-modal feature recognition and fusion schematic diagram of the system of the present application;

[0038] Figure 4 Fig. 4 is a thermal comfort degree prediction schematic diagram of the system of the present application;

[0039] Figure 5 Fig. 5 is an experimental test schematic diagram of the system of the present application;

[0040] Figure 6 Fig. 6 is an interaction schematic diagram of the system of the present application. DETAILED DESCRIPTION

[0041] The embodiments of the present application are described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present application and its applications.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] This invention aims to address the problems of low recognition accuracy, poor stability, large model limitations, and insufficient visualization in traditional thermal comfort monitoring methods in complex indoor environments. By integrating multi-sensor perception, deep learning recognition, and 3D modeling technology on a mobile robot platform, it achieves frame-by-frame recognition and cross-frame identity preservation for multiple people in complex indoor environments. This overcomes interference caused by personnel occlusion, posture overlap, and changes in lighting. Simultaneously, it dynamically fuses multimodal human characteristics such as gender, clothing thermal resistance, posture, behavior, and emotion with environmental parameters, breaking through the limitations of traditional predictive average voting models that rely solely on average parameters. It establishes a more accurate individualized thermal comfort calculation model and generates a real-time 3D thermal comfort distribution map through 3D human body mesh modeling and spatial mapping, achieving a leap from single-point measurement to dynamic spatial visualization. This invention establishes an innovative multimodal body language integrated real-time thermal comfort monitoring (MRTM) system, enabling real-time, dynamic, and visualized assessment of the thermal comfort status of multiple individuals. This not only improves the accuracy and personalization of comfort prediction but also provides refined data for the control of HVAC systems in intelligent buildings, effectively promoting energy conservation, emission reduction, and green building development.

[0044] See Figure 1 and Figure 2This invention provides a thermal comfort monitoring robot system based on multimodal body language feature recognition and fusion, including an intelligent mobile platform, a multimodal feature recognition module, a data processing and fusion module, a thermal comfort calculation module, and a three-dimensional visualization and control module. The intelligent mobile platform is equipped with multiple types of environmental sensors and visual acquisition devices for autonomous navigation in dynamic indoor environments and for collecting environmental parameters and human image data. The multimodal feature recognition module uses a deep learning network to perform frame-by-frame human detection, posture recognition, and identity preservation on the video stream, extracting multimodal human features such as gender, clothing type and thermal resistance, emotional state, and behavioral patterns. The data processing and fusion module achieves cross-frame identity consistency maintenance through multi-stage identity matching and temporal correction algorithms, and performs cross-modal alignment and fusion of multimodal human features and environmental parameters. The thermal comfort calculation module, based on a predictive average voting model, incorporates individual clothing thermal resistance parameters estimated by the clothing recognition module, combines posture and behavior recognition to estimate individual metabolic rate to dynamically adjust the individual metabolic rate parameters used in the model, and incorporates emotional state recognition results as a psychological correction factor into the model to dynamically calculate the individual thermal comfort index. The three-dimensional visualization and control module maps the thermal comfort index to a three-dimensional human body grid and spatial map, generates individual and spatial comfort distribution maps, and connects with the building environment automatic control system (including but not limited to the building automation system) to achieve comfort-oriented dynamic control.

[0045] In some embodiments, the multimodal feature recognition module specifically includes: a human body detection and pose recognition unit, which uses a SlowFast network and an OpenPose model to extract human body key points frame by frame; a cross-frame identity preservation unit, which, based on the AlphaPose and PoseFlow algorithms, improves the matching robustness when multiple people are occluded and their poses change by introducing a hybrid feature measurement method based on Euclidean distance and cosine similarity, and uses Kalman filtering to dynamically correct the trajectory to reduce noise and sudden action interference, thereby achieving continuous tracking of multiple human identities; and a multimodal feature extraction unit, which performs gender and emotion recognition through a convolutional neural network, performs clothing recognition and thermal resistance matching through a target detection network, and performs behavior recognition and metabolic rate estimation through pose sequence analysis.

[0046] In some embodiments, the cross-frame identity preservation unit is specifically configured to: perform key point matching using a hybrid metric of Euclidean distance and cosine similarity; perform temporal smoothing and noise suppression of the trajectory using a Kalman filter; and perform re-identification and identity recovery using appearance features and spatial constraints when occlusion or pose overlap occurs.

[0047] In some embodiments, the data processing and fusion module specifically includes: a multi-stage identity matching unit, which performs two-layer matching by combining the intersection-union ratio of the target detection box and the depth appearance features; a temporal correction unit, which dynamically smooths the position and key point trajectory through Kalman filtering; and a cross-modal alignment unit, which realizes the alignment and fusion of human features and environmental parameters through a unified timestamp and spatial coordinate system.

[0048] In some embodiments, the multi-stage identity matching unit is specifically configured to: perform spatial constraint matching using the cross-union ratio of detection boxes in the initial stage; introduce appearance features of clothing texture, contour and color distribution for high-dimensional embedding matching in the second stage; and perform normalized pose distance calculation and identity reassignment for unmatched targets.

[0049] In some embodiments, the thermal comfort calculation module is specifically configured to: dynamically adjust the thermal resistance and metabolic rate parameters of clothing to replace the fixed average value in the traditional PMV model; introduce emotion recognition results as a psychological correction factor to enhance individualized comfort prediction; output an individual thermal comfort index ranging from -3 to +3, and update its changes over time and space in real time.

[0050] In some embodiments, the three-dimensional visualization and control module is specifically configured as follows: mapping the thermal comfort index to a three-dimensional skeleton model using human body mesh reconstruction technology, and rendering the comfort distribution using color gradient; generating an overall spatial comfort distribution map using SLAM mapping technology to intuitively display the differences between hot and cold areas; and transmitting the comfort data to the building environment automatic control system in real time through a wireless communication interface to trigger dynamic control of HVAC.

[0051] In some embodiments, the intelligent mobile platform is specifically configured as follows: equipped with a lidar and inertial navigation unit to achieve autonomous mapping and path planning; integrated with temperature, humidity, wind speed, carbon dioxide, and thermal imaging sensors to comprehensively collect environmental parameters; and using the ROS2 operating system to achieve multi-sensor data synchronization and system collaborative control. Distributed processing can be adopted (detection is completed at the robot end, and fusion computing is completed at the cloud / edge end) to ensure real-time performance and computing power adaptability.

[0052] This invention also provides a thermal comfort monitoring method based on multimodal body language feature recognition and fusion, using the thermal comfort monitoring robot system of any of the foregoing embodiments. The method includes the following steps:

[0053] Autonomous navigation in dynamic indoor environments via an intelligent mobile platform, simultaneously collecting environmental parameters and human image data;

[0054] Perform human detection, pose recognition, and cross-frame identity preservation on the video stream frame by frame, and extract multimodal human features such as gender, clothing category and thermal resistance, emotional state and behavioral pattern;

[0055] By using multi-stage identity matching and temporal correction, cross-frame identity consistency is maintained. Multimodal human features and environmental parameters are time-stamped and spatially aligned to construct a fused data vector.

[0056] Based on the predictive average voting model, the individual clothing thermal resistance parameter estimated by the clothing recognition module and the individual metabolic rate estimated by posture and behavior recognition are introduced to dynamically adjust the individual metabolic rate parameter used in the model. The emotional state recognition result is also incorporated into the model as a psychological correction factor to dynamically calculate the individual thermal comfort index.

[0057] The thermal comfort index is mapped onto a 3D human body mesh and spatial map to generate individual and spatial comfort distribution maps.

[0058] Comfort data is fed back to the building environment automation system to achieve dynamic adjustment guided by comfort.

[0059] The thermal comfort monitoring robot system and method proposed in this invention, based on multimodal body language feature recognition and fusion, effectively overcomes the shortcomings of traditional thermal comfort monitoring methods in terms of dynamism, personalization, spatiality, and real-time performance by integrating mobile robot platforms, multi-sensor perception, deep learning, and 3D modeling technology. It achieves frame-by-frame recognition of individual thermal comfort status, cross-frame identity preservation, and multimodal feature fusion in complex multi-person indoor environments, significantly improving monitoring accuracy, stability, and personalization. Furthermore, it intuitively presents the spatial comfort distribution based on 3D dynamic visualization technology, providing a scientific basis for building energy conservation and fine-tuning, and has outstanding practical value and broad application prospects.

[0060] The following further describes specific embodiments of the present invention, algorithm examples, and experimental verification.

[0061] A thermal comfort monitoring robot system based on multimodal body language feature recognition and fusion, relying on a mobile robot platform and combining deep learning and multi-sensor fusion, enables real-time thermal comfort monitoring and 3D visualization of multiple people in complex indoor environments. Figures 1 to 6 As shown, the specific technical solution is as follows:

[0062] First, the robot is equipped with various environmental sensors, including those for temperature, humidity, wind speed, carbon dioxide concentration, and thermal imaging. It also acquires dynamic image information of people indoors using a high-definition binocular camera. The system utilizes an improved SlowFast network and OpenPose model to perform human detection and pose recognition frame by frame in the video, extracting key human features. Furthermore, it uses the PoseFlow algorithm to achieve cross-frame pose correlation, thereby maintaining the consistency of human identity and the continuity of actions.

[0063] Secondly, the system uses an improved AlphaPose algorithm to re-identify multiple people in complex scenes, extracting multimodal body language features including gender, clothing type and thermal resistance, posture and behavior, and emotional state. The acquired human feature information and environmental parameter data are fused through a multi-stage identity matching algorithm and input into an improved predictive average voting model and thermal voting model to achieve dynamic calculation of individual thermal comfort.

[0064] Furthermore, by combining human body mesh reconstruction technology with 3D modeling methods, the system maps the individual thermal comfort index calculated frame by frame onto the human skeleton and 3D mesh model, generating dynamic 3D visualization results of human thermal comfort. Simultaneously, by incorporating mapping technology, the system constructs an indoor space model in real time and overlays multiple individuals' thermal comfort data to form an overall comfort distribution map of the space.

[0065] Finally, the system interfaces with the building automation system via a wireless communication module, outputting real-time comfort information for individuals and the overall space, providing a basis for the control of the HVAC system, and realizing dynamic feedback and adaptive adjustment.

[0066] The specific implementation of the system is further detailed below.

[0067] System Architecture

[0068] The system architecture of this invention consists of three modules: a perception layer (mobile platform and environmental sensing), a cognition layer (multimodal recognition and computation), and an application layer (3D modeling and intelligent control), forming a complete technology chain, such as... Figure 1 As shown.

[0069] 1. Intelligent mobile platform and environmental perception architecture

[0070] This invention uses a mobile robot platform as its core, integrating autonomous navigation and mapping functions. The robot platform can achieve stable path planning and positioning in complex indoor environments, thus ensuring the coverage and flexibility of data collection. With the help of LiDAR, inertial navigation units, and the ROS2 operating system, the robot can achieve high-precision environmental modeling and autonomous navigation, avoiding the limitations of fixed sensor deployment.

[0071] In terms of environmental perception, the system is equipped with multiple types of sensors, including temperature and humidity sensors, anemometers, carbon dioxide sensors, air quality sensors, and infrared thermal imagers. These sensors can collect physical and air quality parameters of the indoor environment in real time, ensuring that the data comprehensively covers the input dimensions required for thermal comfort calculations. Multi-sensor data is aligned with human feature data through a unified timestamp management and synchronization mechanism, providing a solid environmental foundation for subsequent multimodal fusion. The innovation of this module lies in the deep integration of mobile robots and environmental perception units, realizing a mobile and scalable dynamic perception platform that overcomes the limitations of insufficient coverage by traditional fixed sensor points.

[0072] 2. Multimodal human feature recognition and fusion computing architecture

[0073] The system captures indoor video streams of people using a high-definition RGB-D camera and employs a deep learning network to perform frame-by-frame human detection and pose recognition. Specifically, it utilizes the SlowFast network and the OpenPose model to extract key human skeleton information and combines improved AlphaPose and Pose Flow algorithms to maintain identity across frames, ensuring continuity and accuracy in complex scenarios such as multiple people occluding, overlapping poses, and changes in lighting. Figure 3 As shown.

[0074] In terms of human feature recognition, the system can extract multimodal information such as gender, clothing type and thermal resistance, behavioral posture, and emotional state. Among them, clothing recognition results are matched with a thermal resistance database to estimate human heat dissipation conditions; behavioral recognition is combined with posture sequence analysis to infer metabolic rate levels; and emotion recognition is based on a facial expression deep neural network classifier to supplement the influence of individual psychological factors on thermal comfort.

[0075] In the fusion computing phase, a multi-stage identity matching and Kalman filtering algorithm is employed to align frame-by-frame features with environmental parameters in a temporal and spatial manner. Through this process, the system can construct an individualized thermal comfort calculation model, overcoming the limitations of traditional thermal sensation voting methods that rely solely on group average parameters, thus achieving dynamic and differentiated comfort assessment. The advantage of this module lies in the deep integration of multimodal features, encompassing both physical environmental parameters and individual differences in psychological and behavioral aspects, thereby significantly improving the accuracy and personalization of thermal comfort prediction.

[0076] 3. 3D Modeling and Visualization Architecture

[0077] In the results presentation and application phase, this invention utilizes human body mesh reconstruction technology and spatial modeling methods to map the individual thermal comfort index calculated frame by frame onto a three-dimensional human skeleton model, generating a visualized human comfort distribution map. Combined with the spatial mapping results, an overall indoor thermal comfort distribution map is formed, achieving multi-level comfort visualization from individual to group and from point location to space. This three-dimensional visualization intuitively demonstrates the dynamic evolution of indoor comfort states, providing researchers and building managers with a scientific basis for decision-making.

[0078] Furthermore, this system interfaces with the building automation system via a wireless communication interface, enabling real-time monitoring results to be fed back to the HVAC control system for comfort-oriented dynamic energy-saving regulation. This improves both the building's energy efficiency and the comfort experience of its occupants. The module's innovation lies in its pioneering integration of frame-by-frame dynamic thermal comfort calculation and 3D visualization results into intelligent building control, achieving a closed-loop application of "sensing-analysis-regulation" and providing a new path for the development of green buildings and smart cities.

[0079] Human detection, cross-frame identity preservation, and multimodal feature recognition algorithms

[0080] 1. Human body detection and posture recognition

[0081] This invention provides a method for thermal comfort monitoring through human body detection and posture recognition.

[0082] A joint system combining the SlowFast network and the OpenPose model is constructed to perform frame-by-frame analysis of RGB video streams acquired by a robot. The SlowFast network is a dual-channel video understanding architecture, where the "Slow" branch captures low-frequency spatial semantic features, and the "Fast" branch captures high-frequency motion changes, accurately modeling human motion dynamics in complex environments.

[0083] The SlowFast network is used to first identify motion patterns in the video clips, which enhances the contextual awareness of subsequent human detection.

[0084] The OpenPose model is used to perform two-dimensional keypoint prediction on the detected human body region, extracting 18 main skeletal points, including the head, torso, and limbs. Through frame-by-frame extraction of these skeletal keypoints, the system can generate a clear representation of human posture, providing foundational data for subsequent behavior recognition and thermal comfort calculations.

[0085] The embodiments of the present invention show that the method improves the average detection accuracy (mAP) by 7.3% in multi-person scenarios compared with the traditional single model, and still maintains a high accuracy when the lighting is complex and the movements of people are large.

[0086] 2. Maintaining User Identity Across Frames

[0087] This invention addresses the problems of frequent identity switching and false tracking in traditional methods by improving upon the AlphaPose and PoseFlow algorithms. In dynamic indoor environments, the system ensures both the accuracy of frame-by-frame detection of human bodies and the continuity of cross-frame tracking.

[0088] The AlphaPose model is used to generate single-frame human key points with high accuracy, and the Pose Flow algorithm is used to construct identity trajectories by temporal correlation of key points.

[0089] This invention introduces a hybrid feature measurement method based on Euclidean distance and cosine similarity into Pose Flow, which improves the matching robustness under multiple occlusions and pose changes.

[0090] Using Kalman filtering to dynamically correct the trajectory effectively reduces the interference of noise and sudden movements on the recognition results.

[0091] The embodiments of this invention demonstrate that the method can significantly reduce the ID switching rate (by approximately 35%) and increase the average identity retention time when there is cross-movement, partial obstruction, or close interaction among multiple people. The method ensures that the system can establish a continuous and stable comfort calculation curve for each individual, thereby meeting real-time monitoring requirements.

[0092] 3. Multimodal feature recognition

[0093] This invention enables individualized and differentiated thermal comfort estimation. The system not only relies on posture recognition but also extracts multimodal human features and fuses them with environmental parameters.

[0094] In some preferred embodiments, for gender recognition, the system employs a lightweight face classification model based on a convolutional neural network (CNN), and the system embeds a facial_expression_recognition module.

[0095] A hybrid deep learning CNN-extreme machine learning approach was employed. Tested on the MORPH-II dataset, it maintains high recognition accuracy (>95%) even with low-resolution video input, providing prior conditions for the individualized setting of clothing thermal resistance and metabolic rate parameters.

[0096] In some preferred embodiments, a `dress_type_recognition` module is embedded in the system for clothing identification and thermal resistance estimation. The system is trained on a YOLOx-Tiny object detection network and uses a deep image classification network to identify clothing types (long-sleeved, short-sleeved, jackets, etc.). The results are then matched with a clothing thermal resistance database in the ISO 9920 standard to infer an individual's insulation capacity. Under laboratory conditions, this method achieves an average identification accuracy of 92.1%, and the estimated thermal resistance parameter error is within ±0.05 clo.

[0097] In some preferred embodiments, for emotion recognition, the system uses Keras for facial emotion detection. The system utilizes facial expression features combined with a deep neural network classifier to identify emotional states such as pleasure, calmness, and tension. Research shows that emotional state has a significant impact on thermal perception; for example, pleasant emotions often reduce sensitivity to environmental hot and cold stimuli. The introduction of this module makes thermal comfort prediction closer to an individual's actual feelings.

[0098] In some preferred embodiments, for behavior recognition, the system infers the metabolic rate level of an individual through posture sequence features. Specifically, when an individual is in a standing, walking, or sitting position, the metabolic rate (unit: met) is calculated by combining the amplitude of their movements. This method realizes the extension from static parameters to dynamic parameters, thereby more accurately reflecting the thermal comfort level of the human body in different activity states. The robot collects individual data in an indoor environment and outputs video data according to the previous experimental settings. The MRTM framework uses Faster R-CNN for human behavior detection and an hourglass network for posture estimation. The framework also includes two modules - a posture flow generator (PF-Builder) and a posture flow non-maximum suppression (PF-NMS) for cross-frame posture association to generate a posture flow. The system associates similar postures across frames by maximizing the overall confidence within the time series, while calculating spatial similarity and posture matching. The system measures the similarity of human bounding boxes in adjacent frames through feature extraction and matching algorithms, thereby achieving accurate and stable recognition of multi-body language features and posture estimation within the video. The system defines a soft matching function for cross-frame posture distance and intra-frame posture distance according to formula (1).

[0099] (1)

[0100] This invention assumes two human poses, P1 and P2, each containing N keypoints. The method described will maximize the matching accuracy across frames. In formula (1) and These represent the coordinates of the nth keypoint in poses P1 and P2, respectively. (Set) {1,2, ..., N} represents all keypoints, where N is the total number of keypoints in each human pose. The neighborhood centered on the key point and These are the confidence scores for key points in behaviors P1 and P2, respectively.

[0101] Formula (2) represents the spatial similarity between key points.

[0102] (2)

[0103] Formula (3) is obtained by combining formula (1) and formula (2).

[0104] (3)

[0105] when ={ , When λ is constant, these parameters can be determined in a data-driven manner. A time dimension is introduced to measure the probability that the pose distance between two intersecting frames points to the same person. Features are extracted using bounding boxes. and , respectively denoted as and The size threshold is 10%. and The similarity between them is evaluated based on the standard percentage of correct key points. express and The similarity score between them. Formula (4) represents the pose distance between P1 and P2.

[0106] (4)

[0107] Through the above-mentioned multimodal feature recognition and fusion, this system can construct an individualized thermal comfort profile, realizing the transformation from "group average prediction" to "individual differentiated assessment", and providing a solid foundation for the next step of dynamic comfort modeling.

[0108] Data processing and fusion

[0109] 1. Multi-stage identity matching

[0110] In dynamic indoor scenes, single-frame recognition results are easily affected by human interaction, occlusion, and overlap, leading to discontinuities and misidentifications of target identities. To address this, this system introduces a multi-stage identity matching method, performing layer-by-layer verification and integration of frame-by-frame detection results. In the initial stage, the system first calculates the intersection-union ratio (IUGR) between adjacent frames using the position and size information of the target detection bounding box, serving as a spatial constraint for candidate identity matching. Building upon this, deep appearance features, including human clothing texture, contour, and color distribution, are further introduced, and matching is performed using the cosine similarity of high-dimensional embedding vectors, improving robustness under conditions of similar human appearance or partial occlusion. This two-layer matching mechanism ensures the accuracy of identity assignment and significantly reduces the mismatch rate during multi-human interaction. This system uses the Multi-Stage Identity Matching (MSIM) algorithm to combine various information modules to achieve real-time online generation of 3D thermal meshes and behavior tracking.

[0111] In some preferred embodiments, the features used to identify a person include: gesture ( ), bounding box ( The system matches features based on gender, behavior, emotion, and identity. It applies a Kalman filter to refine the detected features in the current video frame, generating smoother trajectories. The system then calculates the matching matrix. The first stage of matching is performed, where the identity embedding of frame t is matched against all existing embeddings. Equation (5) represents Matching rules. When yes At that time, is and The threshold that is exactly the same as the trajectory proposed by the qth human.

[0112] (5)

[0113] For individuals whose identities remain unknown due to indoor environmental occlusion, the intersection of bounding boxes serves as the positional constraint, while the normalized pose distance serves as the shape constraint. First, all bounding boxes are scaled proportionally, and the center point of each box is determined. Then, the normalized pose vector is calculated by subtracting the center point from the coordinates of each keypoint. Finally, the normalized pose distance is obtained. Match trajectories that are not highly similar to those in the previous frame, lower the threshold, and repeat the steps. If no match is found, update the trajectory segment and assign a new ID. Equation (6) is the merged distance matrix.

[0114] (6)

[0115] 2. Timing Correction and Data Stabilization

[0116] After identity matching is completed, the system needs to maintain the continuity of the trajectory and the stability of the data. To this end, this study introduces a Kalman filter to perform temporal correction on the human body position and key point trajectory. The Kalman filter can predict the target state at the next moment based on historical states (position, velocity), and then update it by combining the current detection results, thus maintaining the stability and smoothness of the trajectory even under short-term frame drops, sudden action changes, or severe occlusion. Furthermore, for the identity switching problem in multi-person scenarios, the system comprehensively considers the Kalman prediction results and appearance similarity to achieve adaptive trajectory correction, effectively reducing identity loss and duplicate assignment. Experiments show that this method can reduce the user switching rate by approximately 35%, thereby ensuring the continuity and availability of individual feature data.

[0117] 3. Cross-modal alignment and fusion

[0118] After achieving individual identity preservation and trajectory stabilization, the system further aligns and fuses cross-modal data. First, all human feature data and environmental sensor data are synchronized using a unified timestamp mechanism to ensure a one-to-one correspondence between human state and environmental conditions at the same point in time. Second, by constructing a spatial coordinate system, the individual's spatial location is projected and matched with environmental parameters, thereby achieving data alignment in the spatial dimension. Finally, the system integrates multimodal human features (gender, posture, clothing, emotion, behavior, etc.) and environmental parameters (temperature, humidity, wind speed, CO2, PM2.5, etc.) into a complete input vector, which is then passed to the thermal comfort calculation model. Through this fusion process, the system can achieve joint modeling of individual human differences and dynamic environmental changes, providing high-precision data support for personalized and dynamic thermal comfort prediction. Cross-modal alignment can be achieved, for example, through ROS timestamp synchronization and SLAM coordinate mapping.

[0119] Thermal comfort calculation and modeling

[0120] 1. Optimize the predicted average vote

[0121] Traditional predictive average voting models are widely used in thermal comfort research, but they are based on the assumption of steady-state thermal balance and mainly rely on group average parameters, such as uniform clothing thermal resistance and metabolic rate, making it difficult to reflect individual differences and dynamic changes. To address this issue, this system improves upon the classic model. First, it introduces individual clothing thermal resistance parameters estimated by the clothing recognition module to replace the fixed average value, reflecting the differences in insulation capacity under different clothing conditions. Second, it combines posture and behavior recognition modules to estimate individual metabolic rates, dynamically adjusting metabolic levels to avoid prediction bias caused by differences in activity states. Finally, emotion recognition results are incorporated into the model as a psychological correction factor, taking into account the impact of psychological feelings on thermal comfort while maintaining physical layer thermal balance calculations. Emotion recognition results can serve as an auxiliary decision-making reference or fine-tune the PMV output to adapt to the thermal balance model. Through these improvements, the model can not only calculate the individual's thermal comfort index under specific environmental conditions but also dynamically reflect the comfort fluctuations caused by changes in activity and emotion.

[0122] This system reconstructs the dynamic thermal comfort multi-mesh by substituting all occupant characteristics and environmental features into Fanger's PMV model in formulas (7) and (8), and combining body-mesh estimation and three-dimensional key point estimation. The accurate dynamic thermal comfort results are directly mapped onto the three-dimensional mesh, and the dynamic thermal comfort multi-mesh is reconstructed.

[0123] (7)

[0124] (8)

[0125] in, , , , W and These refer to clothing insulation, clothing surface area factor, metabolic rate (W / m²), external work (W / m²), and convective heat transfer coefficient, respectively.

[0126] 2. Individualized thermal comfort modeling

[0127] After parameter correction, the system establishes an individualized thermal comfort curve for each monitored individual. This process first fuses frame-by-frame detected features such as clothing, behavior, and emotions with data collected from environmental sensors, including temperature, humidity, wind speed, and carbon dioxide concentration, to form a multi-dimensional input vector. Subsequently, an improved formula or a machine learning-based regression model is used to calculate the individual's thermal comfort index (range -3 to +3), where -3 represents "cold," +3 represents "hot," and 0 represents "neutral." Unlike traditional methods based on group averages, this system's individualized modeling can track the comfort state of each user in real time and maintain continuity across different time periods and spatial locations. For example, when there are people in suits working in the same space and people in sportswear exercising, the system can derive the thermal comfort values ​​for each individual, accurately reflecting their different perceptions of the environment. Figure 4 As shown in the figure, this method effectively improves the personalization and differentiation of thermal comfort prediction.

[0128] 3. Three-dimensional dynamic visualization modeling and distribution

[0129] To visually represent the spatial distribution of thermal comfort, this system combines 3D human body mesh reconstruction and indoor mapping techniques to generate 3D comfort visualization models for both individuals and the overall space. At the individual level, the system maps the calculated thermal comfort index onto the surface of the 3D human skeleton mesh, rendering it using color gradients (e.g., blue for cold, red for hot) to create a comfort distribution map of the human body surface. At the spatial level, the comfort information of each individual is overlaid on an indoor map constructed using map modeling technology, generating a dynamic comfort distribution map of the entire space. This distribution map visually reflects the differences in temperature and human perception across different areas; for example, areas near windows may feel warmer due to higher radiant temperatures, while areas near vents may feel cooler. This 3D visualization not only provides researchers with an intuitive tool for presenting comfort but also offers decision support for building management systems, achieving a leap from "point measurement" to "area distribution."

[0130] Experimental verification

[0131] Experimental Environment and Configuration

[0132] To verify the feasibility and advancement of this system, it was deployed in the lobby of a university research building. The experimental area, approximately 143.2 square meters in area and 4.5 meters in height, exhibits typical characteristics of an open public space. The uneven distribution of people in this environment, along with multi-directional walking, interaction, and short-term gatherings, effectively simulates real-world application scenarios. For hardware, the Turtlebot4 mobile robot platform was selected, equipped with a LiDAR for mapping and autonomous navigation. Environmental sensors included temperature and humidity sensors, wind speed sensors, carbon dioxide sensors, and an infrared thermal imager for comprehensive indoor environmental parameter collection. The visual acquisition component used a binocular camera at 30 fps and 1080p resolution for human detection and 3D modeling. The computing unit consisted of an Intel NUC and a Raspberry Pi 4, running the ROS2 operating system and interacting with and storing data with a host computer via Wi-Fi. Figure 5 As shown.

[0133] Experimental process

[0134] To verify the feasibility and effectiveness of the system of this invention, the experimental process was designed into three stages: data acquisition, feature recognition and fusion, and thermal comfort calculation and visualization. Each stage is interconnected, forming a complete monitoring and verification process. The first stage, data acquisition, involved the robot using SLAM technology to complete spatial mapping and path planning in the hall, and then autonomously moving within the experimental area according to a preset navigation route. Environmental sensors on the robot collected real-time environmental parameters such as temperature, humidity, wind speed, CO2 concentration, and PM2.5; an RGB-D camera and an infrared thermal imager simultaneously acquired human video streams and depth data. To ensure the diversity and representativeness of the experimental data, the experiment was conducted at different times (morning, afternoon, and night), covering different indoor population densities and lighting conditions. A total of 30 volunteers participated in the experiment, representing different genders, ages, and clothing types. Some volunteers remained seated during the experiment, while others walked, communicated, or engaged in short-term activities to simulate diverse behavioral scenarios in real architectural spaces.

[0135] The second phase, feature recognition and fusion, involves frame-by-frame processing of the video stream after data acquisition. First, the SlowFast network and OpenPose model are used for human detection and pose recognition, extracting key point skeleton information. Then, improved AlphaPose and Pose Flow algorithms maintain cross-frame identity consistency, ensuring continuous and stable trajectories even during occlusion or movement of individuals. Next, the system further extracts multimodal features such as gender, clothing type and thermal resistance, emotional state, and behavioral patterns. Clothing recognition results are compared with the ISO 9920 standard database to obtain corresponding thermal resistance parameters; behavioral recognition, combined with skeleton sequence analysis, is used to estimate metabolic rate levels; and emotion recognition uses facial features and a deep classification model for classification. Simultaneously, all human feature data and environmental parameters are synchronized through a unified timestamp mechanism and spatially aligned in the SLAM coordinate system, ensuring a one-to-one correspondence between the features of each participant and the environmental conditions of their location.

[0136] The third stage: Thermal comfort calculation and visualization. Based on feature recognition and fusion, the system uses multimodal features and environmental parameters as input, feeding them into an improved PMV model for individualized thermal comfort calculation. Unlike traditional methods, this model dynamically adjusts clothing thermal resistance, metabolic rate, and psychological correction factors to generate a more accurate comfort index. The calculation results are mapped in real time onto a 3D human body mesh model and overlaid on a spatial map created by SLAM, forming a visualization of individual and overall comfort distribution. During the experiment, volunteers were also asked to periodically fill out thermal comfort polls, comparing their results with the system's predictions to assess the accuracy of the predictions. Figure 6 As shown.

[0137] The entire experiment achieved a complete closed loop, from the robot's autonomous data collection to multimodal feature extraction, and then to individualized thermal comfort calculation and 3D visualization. This process not only verified the system's recognition accuracy and calculation precision in complex dynamic scenarios but also demonstrated its application potential for supporting energy-saving optimization in real-world building environments.

[0138] Experimental Results and Analysis

[0139] Experimental results show that the system demonstrates superior performance in multimodal feature recognition, personalized thermal comfort prediction, and 3D visualization, with multimodal feature recognition accuracy exceeding 90%. Gender recognition accuracy reached 94.6%, clothing recognition and thermal resistance estimation accuracy reached 91.7%, emotion recognition accuracy reached 92.3%, and multi-person behavior recognition accuracy reached 93.8%, maintaining a high overall level and ensuring the reliability of input data. In thermal comfort prediction, the system's calculation results based on the improved model achieved a consistency of 93.5% with volunteer thermal comfort voting, significantly outperforming traditional single-point measurement methods (approximately 82%), and dynamically reflecting individual comfort changes caused by differences in clothing, behavior, and emotions. In 3D dynamic visualization modeling, the system successfully mapped individual comfort indices to human body grids and indoor maps, clearly presenting the differences in temperature between different areas; for example, areas near glass curtain walls were warmer while areas near ventilation openings were cooler. In summary, the system not only significantly improves accuracy and personalization in identification and prediction, but also demonstrates intuitive spatial visualization and significant energy-saving potential in application, verifying its feasibility and advanced nature in smart buildings and green energy-saving management.

[0140] In summary, this invention provides a thermal comfort monitoring robot system and method based on multimodal body language feature recognition and fusion, establishing an innovative multimodal body language integrated real-time thermal comfort monitoring (MRTM) system to achieve real-time, dynamic, and visual assessment of the thermal comfort status of multiple individuals. By integrating multi-sensor perception, deep learning recognition, and 3D modeling technologies on a mobile robot platform, this invention achieves frame-by-frame recognition and cross-frame identity preservation for multiple individuals in complex indoor environments, solving the problems of low recognition accuracy and instability caused by personnel occlusion, posture overlap, and lighting changes in traditional methods. Simultaneously, this invention dynamically fuses multimodal human features such as gender, clothing thermal resistance, posture, behavior, and emotion with environmental parameters, overcoming the limitations of traditional predictive average voting models that rely solely on average parameters, and establishing a more accurate individualized thermal comfort calculation model. Furthermore, this invention, through 3D human body mesh modeling and spatial mapping, can generate a real-time 3D thermal comfort distribution map, achieving a leap from single-point measurement to dynamic spatial visualization. As a novel intelligent monitoring method, this invention can not only improve the accuracy and personalization of comfort prediction, but also provide a refined basis for the control of HVAC systems in intelligent buildings, thereby effectively promoting energy conservation, emission reduction and green building development.

[0141] Compared with existing thermal comfort monitoring schemes based on single-point data acquisition and static model prediction, this invention has the following outstanding features and significant advantages:

[0142] 1. This invention enables frame-by-frame recognition and cross-frame identity preservation of multiple human bodies in complex dynamic scenes, effectively overcoming recognition errors caused by human occlusion, posture overlap and lighting changes, and significantly improving the stability and accuracy of real-time monitoring.

[0143] 2. This invention introduces multimodal features such as gender, clothing thermal resistance, posture behavior, and emotional state, and integrates them with environmental parameters to construct a dynamic thermal comfort calculation model. This breaks through the limitation of the traditional PMV model, which only relies on average parameters, and realizes individualized and differentiated comfort assessment.

[0144] 3. This invention, through three-dimensional human body mesh modeling and spatial mapping, has for the first time realized the three-dimensional dynamic visualization of the thermal comfort state of multiple people. It can not only intuitively present the individual comfort state, but also form an overall spatial comfort distribution map, providing a scientific and intuitive decision-making basis for building environment control.

[0145] 4. This invention is based on a mobile robot platform and a lightweight deep learning model, which has good real-time performance and deployability. It can run without the need for expensive hardware support and has high potential for engineering and industrial applications.

[0146] In summary, this invention significantly outperforms existing technologies in terms of real-time performance, accuracy, personalization, and spatial visualization. It can provide reliable support for intelligent building environmental monitoring, energy-saving control of HVAC systems, and the development of green buildings, demonstrating outstanding technological advancement and broad application prospects.

[0147] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0148] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0149] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0150] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0151] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0152] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0154] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0156] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0157] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0158] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0159] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A thermal comfort monitoring robot system based on multi-modal body language feature recognition and fusion, characterized in that, The application relates to a thermal comfort prediction system based on multi-modal feature recognition and fusion, comprising the following steps: An intelligent mobile platform is provided with multiple types of environmental sensors and visual acquisition equipment, which is used for autonomous navigation and acquisition of environmental parameters and human image data in a dynamic indoor environment; A multi-modal feature recognition module is used for frame-by-frame human body detection, posture recognition and identity maintenance based on a deep learning network, and multi-modal human body features such as gender, clothing category, thermal resistance, emotional state and behavior mode are extracted; The multi-modal feature recognition module comprises a human body detection and posture recognition unit which uses a SlowFast network and an OpenPose model to extract human body key points frame by frame; a cross-frame identity maintenance unit which uses an AlphaPose and PoseFlow algorithm, introduces a hybrid feature measurement method based on Euclidean distance and cosine similarity to improve the matching robustness when multiple people are blocked and postures change, uses Kalman filtering to dynamically correct the trajectory to reduce noise and sudden action interference, and realizes continuous tracking of multiple human bodies; and a multi-modal feature extraction unit which uses a convolutional neural network to respectively identify gender and emotion, uses a target detection network to identify clothing and match thermal resistance, and uses posture sequence analysis to identify behavior and estimate metabolic rate; wherein the cross-frame identity maintenance unit is configured to: match key points through hybrid measurement of Euclidean distance and cosine similarity; use a Kalman filter to perform time sequence smoothing and noise suppression on the trajectory; and when blocking or posture overlapping occurs, re-identify and recover the identity through appearance features and spatial constraints; A data processing and fusion module is used for realizing cross-frame identity consistency maintenance through multi-stage identity matching and time sequence correction algorithms, and aligning and fusing multi-modal human body features and environmental parameters across modalities; A thermal comfort calculation module is used for dynamically adjusting individual metabolic rate parameters used in the model based on a predicted average voting model, introducing individual clothing thermal resistance parameters estimated by a clothing identification module, combining posture and behavior recognition to estimate individual metabolic rate, and introducing emotional state recognition results as a psychological correction factor into the model to dynamically calculate individual thermal comfort indexes; the thermal comfort calculation module is configured to: dynamically adjust clothing thermal resistance and metabolic rate parameters to replace fixed average values in a traditional PMV model; introduce emotional recognition results as a psychological correction factor to enhance individualized comfort prediction; output individual thermal comfort indexes in the range of -3 to +3, and update the indexes in real time according to changes in time and space; A three-dimensional visualization and regulation module is used for mapping thermal comfort indexes to a three-dimensional human body grid and a space map, generating individual and space comfort distribution maps, and connecting a building environment automatic control system to realize dynamic regulation and control of comfort orientation; the three-dimensional visualization and regulation module is configured to: map thermal comfort indexes to a three-dimensional skeleton model through human body grid reconstruction technology, and use color gradient rendering to render comfort distribution; generate an overall space comfort distribution map by combining SLAM mapping technology to visually display cold and hot area differences; and transmit comfort data to the building environment automatic control system in real time through a wireless communication interface to trigger dynamic regulation and control of a heating ventilation air conditioning system.

2. The system of claim 1, wherein, The data processing and fusion module comprises: A multi-stage identity matching unit combines the bounding box intersection over union and deep appearance features for double-layer matching; A timing correction unit performs dynamic smoothing on the position and key point trajectories through Kalman filtering; A cross-modal alignment unit aligns and fuses the human features and environmental parameters through unified timestamps and spatial coordinate systems.

3. The system of claim 2, wherein, The multi-stage identity matching unit is configured to: In the initial stage, use the bounding box intersection over union for spatial constraint matching; In the second stage, introduce appearance features such as clothing texture, contour, and color distribution for high-dimensional embedding matching; Calculate the normalized posture distance and reassign the identity for unmatched targets.

4. The system of any one of claims 1 to 3, wherein, The intelligent mobile platform is configured to: Carry a laser radar and an inertial navigation unit to realize autonomous mapping and path planning; Integrate temperature and humidity, wind speed, carbon dioxide, and thermal imaging sensors to comprehensively collect environmental parameters; Realize multi-sensor data synchronization and system collaborative control through the ROS2 operating system.

5. A frame-by-frame processing thermal comfort monitoring method based on multi-modal body language feature recognition and fusion, using the system according to any one of claims 1 to 4, characterized in that, The method includes: Autonomously cruise in a dynamic indoor environment through an intelligent mobile platform, and synchronously collect environmental parameters and human image data; Perform human detection, posture recognition, and cross-frame identity maintenance on video streams frame by frame, and extract multi-modal human features such as gender, clothing category, thermal resistance, emotional state, and behavior pattern; Realize cross-frame identity consistency maintenance through multi-stage identity matching and timing correction, synchronize multi-modal human features and environmental parameters with timestamps, and spatially align them to construct a fusion data vector; Based on the predicted average voting model, introduce the individual clothing thermal resistance parameters estimated by the clothing recognition module, combine the individual metabolic rate estimated by the posture and behavior recognition to dynamically adjust the individual metabolic rate parameters used in the model, and include the emotional state recognition results as psychological correction factors into the model to dynamically calculate the individual thermal comfort index; Map the thermal comfort index to a three-dimensional human mesh and a spatial map to generate individual and spatial comfort distribution maps; Feedback the comfort data to the building environment automatic control system to realize dynamic regulation and control guided by comfort.

6. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, realizes the frame-by-frame processing thermal comfort monitoring method based on multi-modal body language feature recognition and fusion as claimed in claim 5.

Images