A portable urban thermal environment intelligent perception method and device based on AR

Abstract

The present invention discloses a portable AR-based intelligent perception method and device for urban thermal environment. The method comprises the following steps: using AR glasses to collect panoramic RGB street view images from the user's perspective, and synchronously obtaining time, geographic location, and meteorological parameters; using a generative adversarial network to map the RGB images and environmental parameters into a panoramic infrared thermal map, and obtaining the radiation temperature of each pixel; projecting the panoramic thermal map onto a unit sphere centered on the user, and calculating the long-wave radiation flux by integrating the spherical angles; and calculating the short-wave radiation flux based on a solar radiation model using the solar altitude angle and other factors; fusing the total radiation to calculate the average radiation temperature, and finally superimposing the thermal map and / or the average radiation temperature on a real or virtual environment through AR glasses or a smart terminal to achieve spatialized heat load visualization.

Description

Technical Field

[0001] The present invention relates to environmental thermal parameter perception and human thermal comfort assessment technology, and in particular to a portable urban thermal environment intelligent perception method and device based on AR (augmented reality). Background Art

[0002] The impact of outdoor thermal environments on human health and comfort is gaining increasing attention. Thermal comfort, a comprehensive indicator of human perception of environmental thermal conditions, is widely used in fields such as urban planning, architectural design, public health, and smart mobility. Rapid, accurate, and personalized acquisition of human thermal comfort indicators in complex urban environments has become a key research topic in urban microclimate perception and human-factor intelligent systems in recent years.

[0003] Currently, there are two main methods for obtaining thermal comfort: one is calculation based on conventional meteorological data and thermal comfort models, such as calculating indicators like the Universal Thermal Climate Index (UTCI) or Physiological Equivalent Temperature (PET) using parameters such as air temperature, humidity, wind speed, and radiant temperature; the other is field measurement using fixed sensing equipment (such as weather stations and human thermal comfort monitoring platforms). While these methods have a sound theoretical foundation in scientific research and engineering practice, they generally suffer from limitations such as limited spatial coverage, poor real-time performance, high deployment costs, and a lack of user interaction. These limitations make them difficult to meet the needs of assessing the dynamic changes in the local thermal environment and human thermal responses in densely built-up areas.

[0004] To address these issues, researchers are exploring the feasibility of using mobile devices to capture images and combine them with infrared sensors for environmental thermal perception. However, these current approaches still face significant technical bottlenecks, such as limited spatial viewing angles, low temperature estimation accuracy, and a lack of systematic coupling with human thermal comfort indicators. For example, a single-frame infrared image typically only reflects the surface temperature distribution at a specific point in time and direction, making it difficult to restore the human body's true full-space radiation environment, thus affecting the accuracy of calculations of key parameters such as mean radiant temperature (Tmrt). Furthermore, these systems generally lack mobile, human-computer interaction-friendly presentation methods, making it difficult to intuitively assist users in identifying thermal risk areas or comfortable paths.

[0005] Currently, some thermal imaging devices are available on the market for portable thermal environment sensing. Representative products include FLIR's FLIR ONE thermal imaging accessory and its professional-grade infrared thermal imaging camera. The FLIR ONE connects to a smartphone via a docking station for image capture and infrared visualization, making it suitable for temperature visualization in everyday scenarios. However, these devices suffer from low thermal sensitivity and measurement accuracy, making them incapable of capturing subtle temperature variations in complex urban environments. In contrast, FLIR's professional infrared thermal imaging cameras offer higher resolution and measurement accuracy, making them suitable for thermal monitoring tasks at the research or engineering level. However, these devices are large, cumbersome to operate, and expensive, making them unsuitable for daily mobile measurements or personalized urban thermal environment assessments, limiting their practicality and potential for widespread deployment. Therefore, neither portable docking devices nor professional infrared cameras offer efficient, flexible, and interactive thermal environment assessment solutions at both the urban and human scales. They also lack a comprehensive thermal parameter estimation process, preventing the scientific calculation and analysis of structured metrics such as mean radiant temperature, UTCI, and PET.

[0006] In the study of thermal environment assessment with higher precision, existing technologies have attempted to perform parameter collection and thermal comfort analysis through multi-point sensor networks. CN105371897A discloses an "outdoor thermal comfort monitoring system and its monitoring method", which collects data such as air temperature, humidity, wind speed and black globe temperature by deploying multiple environmental parameter sensors in active urban areas, and transmits them to the central processor in the control room, and calculates thermal comfort indicators based on a preset model. Although this method has a certain degree of accuracy in static, multi-point regional thermal perception, its deployment and maintenance costs are high, its flexibility is insufficient, and it is difficult to cope with short-term climate changes or the heterogeneity of local thermal environments under complex urban geometric structures, and it is difficult to meet the real-time and individual application requirements.

[0007] Furthermore, existing research and products generally overlook the potential of integrating AR technology with thermal sensing systems. Most current mobile infrared devices lack the ability to overlay thermal comfort assessment results on the user's field of view in real time, nor can they leverage spatial perception technology to guide users in identifying heat-risk areas or comfortable pathways within the city. Furthermore, these systems lack the ability to transfer algorithms from real street scenes to virtual spaces (such as digital twin cities, simulated neighborhoods, and immersive design scenarios), limiting their future application in urban design, heat island simulation, and human-computer interaction visualization.

[0008] In summary, existing technologies still have obvious shortcomings in image fusion analysis capabilities, spatial perception coverage, interactive visualization expression and virtual environment adaptability.

[0009] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0010] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a portable urban thermal environment intelligent perception method and device based on AR.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] In a first aspect, a portable urban thermal environment intelligent perception method based on AR comprises the following steps:

[0013] S1. Use AR glasses to collect panoramic RGB street view images from the user's perspective, obtain time and geographic location information, and call meteorological data sources to supplement air temperature and solar radiation intensity parameters;

[0014] S2. Map the RGB image and environmental parameters into a panoramic infrared thermal image through a generative adversarial network to obtain the radiation temperature distribution of each pixel;

[0015] S3. Project the panoramic heat map onto a unit sphere centered on the user, and calculate the longwave radiation flux received by the observation point by integrating the spherical angle. Calculate the shortwave radiation flux based on the solar radiation model, taking into account the solar altitude angle, shadow state, and sky view factor.

[0016] S4. Combine the total amount of longwave and shortwave radiation to calculate the average radiation temperature;

[0017] S5. Through AR glasses or smart terminal screens, the heat map and / or the average radiation temperature calculated based on the heat map is superimposed on the real scene or virtual environment to achieve spatial heat load visualization.

[0018] Furthermore, step S2 further includes: identifying the surface material category in the RGB image based on the deep learning model, and calibrating the surface temperature according to the material emissivity;

[0019] The surface material recognition uses a semantic segmentation network to achieve pixel-level classification and assigns a preset emissivity value based on the material category;

[0020] The emissivity value is set according to the thermal radiation characteristics of the material and is used to calibrate the radiant brightness output by the generative adversarial network to the actual surface temperature.

[0021] Furthermore, in step S2:

[0022] The generative adversarial network uses U-Net as the generator backbone, the encoder adopts a residual pre-training module, and the decoder combines multi-scale upsampling and skip connections; the discriminator adopts a local block discriminant structure to judge the authenticity of the local area of ​​the heat map, and generates the physical consistency and spatial authenticity of the heat map by combining L1 regression loss, perceptual loss and adversarial loss constraints.

[0023] Among them, the L1 regression loss constrains pixel-level temperature differences; the perceptual loss extracts high-level feature similarity through a pre-trained image recognition model; and the adversarial loss guides the generator to output a thermal distribution image that conforms to the actual thermal distribution characteristics.

[0024] Furthermore, in step S3:

[0025] The long-wave radiation flux calculation adopts the conformal integration method to map the panoramic heat map pixels to the unit spherical coordinates centered on the user, and realizes the full-view radiation energy calculation through the physical integration of longitude φ and latitude θ.

[0026] Furthermore, in step S3:

[0027] The shortwave radiation flux calculation is based on the solar radiation geometric model. By analyzing the sine value of the solar altitude angle, the binary state of the shadow mask and the surface albedo, the following components are calculated: the direct radiation component of the area directly exposed to sunlight; the scattered radiation component of the area with visible sky; and the ground reflected radiation component.

[0028] Furthermore, the solar altitude angle is determined by the acquisition time and the geographical location through an orbital parameter model;

[0029] The sky viewing factor is obtained by counting the pixel ratio of the sky in the low temperature area in the panoramic thermal map pixels.

[0030] Furthermore, step S4 also includes: combining environmental parameters such as air temperature, wind speed and humidity parameters, and outputting a structured thermal comfort index through a thermal comfort model; step S5 also includes: superimposing the thermal comfort index on a real scene or a virtual environment.

[0031] Furthermore, step S5 includes two modes:

[0032] AR mode: Overlays the real-time generated heat map onto the real scene through pseudo-color mapping, and supports head tracking interaction;

[0033] VR mode: Load the heat map generation algorithm in a virtual environment or digital twin model to simulate the heat distribution under given parameters.

[0034] Furthermore, the meteorological data is obtained in real time by calling a public interface through the Internet, or is approximately matched offline through a built-in typical day database combined with time and location information.

[0035] The second aspect is a portable urban thermal environment intelligent sensing device based on AR, comprising:

[0036] AR glasses with an integrated camera module for capturing panoramic RGB street view images from the user's perspective;

[0037] an intelligent terminal, communicatively connected to the AR glasses, comprising a processor and a memory;

[0038] The processor is configured to perform:

[0039] (a) obtaining the panoramic RGB street view image, current time and geographic location information, and calling a meteorological data source to supplement air temperature and solar radiation intensity parameters;

[0040] (b) Using a generative adversarial network, the RGB image and environmental parameters are mapped into a panoramic infrared thermal image to obtain the radiation temperature distribution of each pixel;

[0041] (c) Project the panoramic heat map onto a unit sphere centered on the user and calculate the longwave radiation flux by integrating the spherical angles. Also, calculate the shortwave radiation flux based on the solar radiation geometry model, combined with the solar altitude angle, shadow state, and sky view factor.

[0042] (d) Calculate the mean radiation temperature by combining the total longwave and shortwave radiation;

[0043] (e) controlling the AR glasses or smart terminal screen to superimpose the heat map and / or the average radiation temperature calculated based on the heat map on the real scene or virtual environment to achieve spatialized heat load visualization.

[0044] The present invention has the following beneficial effects:

[0045] This invention presents a portable AR-based intelligent urban thermal environment sensing device and method. By integrating AR glasses with a smart mobile terminal (such as a mobile phone or tablet), and combining panoramic heat map generation using a generative adversarial network with spherical integral radiance calculation, it achieves portable, high-precision intelligent urban thermal environment sensing. This device breaks away from the traditional reliance of fixed sensor networks for thermal comfort assessment. Instead, it captures RGB street view images from the user's perspective to drive thermal radiation field reconstruction. Lightweight edge computing rapidly completes the entire process from image acquisition to thermal comfort index output in near real time. Using AR augmented reality technology, the thermal map and indicators are overlaid on the real scene in real time, enabling interactive visualization of spatial heat loads. Furthermore, the device supports algorithm migration to virtual environments such as digital twins, combining personalized thermal risk warnings with urban planning-level thermal assessment capabilities. Compared to simulation tools such as ENVI-met, this device can achieve ±5°C accuracy in mean radiant temperature estimation without complex modeling, significantly improving spatial coverage efficiency and real-time interactive experience in complex urban scenarios. This provides efficient technical support for smart city management, public health services, and other fields.

[0046] Compared with the prior art, the significant technical advantages of the present invention are specifically reflected in the following aspects:

[0047] 1. More portable and adaptable evaluation method: This system eliminates traditional sensor modules and relies solely on image acquisition and edge computing, significantly improving portability and making it suitable for a variety of use scenarios, including daily travel for individual users and field visits by researchers.

[0048] 2. Image-driven, wide spatial coverage: Leveraging the field of view and panoramic image acquisition capabilities of AR glasses, efficient thermal field restoration of complex street environments can be achieved, overcoming the limitations of traditional infrared acquisition equipment, which have limited coverage and lack of street view semantics.

[0049] 3. Enhanced visualization and interactivity of results: AR displays integrate thermal maps with real-world scenes, allowing users to obtain thermal environment perception feedback without switching devices or viewing angles, making the assessment process more intuitive and real-time.

[0050] 4. Broad application potential: This invention can be used not only to assess the thermal comfort experience of ordinary users in their daily travel, but can also be expanded to professional fields such as urban planning, smart city management, and emergency high-temperature monitoring, and has good market application prospects.

[0051] In summary, this invention focuses on key aspects such as image acquisition, thermal estimation, and augmented reality display, constructing an intelligent assessment system based on mobile devices, using images as input and thermal comfort as output. This system significantly outperforms existing technologies in terms of structural design, assessment accuracy, and user experience, demonstrating its innovative potential and promising application prospects.

[0052] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall flow chart of the portable urban thermal environment intelligent perception method based on AR of the present invention.

[0054] Figure 2 This is a system block diagram of the AR-based portable urban thermal environment intelligent sensing device of the present invention.

[0055] Figure 3 This is a flowchart of a method for implementing portable thermal comfort assessment according to an embodiment of the present invention.

[0056] Figure 4 Schematic diagram of the hardware of a portable urban thermal environment intelligent sensing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0059] To address the shortcomings of existing thermal environment perception and thermal comfort assessment technologies, such as limited portability, limited accuracy, poor real-time performance, and a lack of interactive visualization, this paper proposes a portable AR-based intelligent urban thermal environment perception method and device. This portable system, constructed using AR glasses and a smartphone, captures street imagery and uses deep learning and generative adversarial networks to estimate thermal parameters and thermal comfort indicators, which are then visualized using AR technology. This method combines portability with high accuracy, enabling near-real-time processing and intuitive presentation of results, making it particularly suitable for individualized and city-level thermal environment assessments.

[0060] See Figures 1 to 4 The embodiment of the present invention provides a portable urban thermal environment intelligent perception method based on AR, comprising the following steps:

[0061] Step S1: Use AR glasses to collect panoramic RGB street view images from the user's perspective, obtain time and geographic location information, and call meteorological data sources to supplement air temperature and solar radiation intensity parameters.

[0062] In some embodiments, the meteorological data is obtained in real time by calling a public interface through the Internet, or is approximately matched offline through a built-in typical day database combined with time and location information.

[0063] Step S2: Map the RGB image and environmental parameters into a panoramic infrared thermal image through a generative adversarial network to obtain the radiation temperature distribution of each pixel.

[0064] In some embodiments, step S2 also includes: identifying the surface material category in the RGB image based on a deep learning model, and calibrating the surface temperature according to the material emissivity; wherein, the surface material identification uses a semantic segmentation network to achieve pixel-level classification, and assigns a preset emissivity value based on the material category; wherein, the emissivity value is set according to the thermal radiation characteristics of the material, and is used to calibrate the radiation brightness output by the generative adversarial network to the actual surface temperature.

[0065] In some embodiments, in step S2: the generative adversarial network uses U-Net as the generator backbone, the encoder adopts a residual pre-training module, and the decoder combines multi-scale upsampling and jump connections; the discriminator adopts a local block discriminant structure to perform authenticity judgment on the local area of ​​the heat map, and generates the physical consistency and spatial authenticity of the heat map by combining L1 regression loss, perceptual loss and adversarial loss constraints; wherein, the L1 regression loss constrains pixel-level temperature differences; the perceptual loss extracts high-level feature similarity through a pre-trained image recognition model; the adversarial loss guides the generator to output a thermal distribution image that conforms to the real thermal distribution characteristics.

[0066] Step S3: Project the panoramic heat map onto a unit sphere centered on the user, and calculate the long-wave radiation flux received by the observation point by integrating the spherical angle; and calculate the short-wave radiation flux based on the solar radiation model, combined with the solar altitude angle, shadow state, and sky view factor.

[0067] In some embodiments, in step S3: the long-wave radiation flux calculation adopts the isometric integration method to map the panoramic heat map pixels to the unit spherical coordinates centered on the user, and realizes the full-view radiation energy calculation by physical integration of longitude φ and latitude θ.

[0068] In some embodiments, in step S3: the shortwave radiation flux calculation is based on the solar radiation geometric model, and by analyzing the sine value of the solar altitude angle, the binary state of the shadow mask and the surface albedo, the following are calculated: the direct radiation component of the direct sunlight area; the scattered radiation component of the visible sky area; and the ground reflected radiation component.

[0069] In some embodiments, the solar altitude angle is determined by the acquisition time and geographic location through an orbital parameter model; the sky viewing factor is obtained by counting the pixel ratio of the sky corresponding to the low temperature area in the panoramic thermal map pixels.

[0070] Step S4: merging the total amount of long-wave and short-wave radiation to calculate the average radiation temperature.

[0071] In some embodiments, step S4 also includes: combining environmental parameters such as air temperature, wind speed and humidity parameters, and outputting a structured thermal comfort index through a thermal comfort model; step S5 also includes: superimposing the thermal comfort index on a real scene or a virtual environment.

[0072] Step S5: Using AR glasses or smart terminal screens, the heat map and / or the average radiation temperature calculated based on the heat map is superimposed on the real scene or virtual environment to achieve spatialized heat load visualization.

[0073] In some embodiments, step S5 includes two modes: AR mode: superimposing the real-time generated heat map on the real scene through pseudo-color mapping, supporting head tracking interaction; VR mode: loading the heat map generation algorithm in the virtual environment or digital twin model to simulate the heat distribution under given parameters.

[0074] See Figures 1 to 4 , an embodiment of the present invention also provides an AR-based portable urban thermal environment intelligent perception device, comprising: AR glasses, an integrated camera module, for collecting panoramic RGB street view images from a user's perspective; an intelligent terminal, communicatively connected to the AR glasses, comprising a processor and a memory; the processor is configured to execute: (a) obtaining the panoramic RGB street view image, current time and geographic location information, and calling a meteorological data source to supplement air temperature and solar radiation intensity parameters; (b) mapping the RGB image and environmental parameters into a panoramic infrared thermal image through a generative adversarial network to obtain the radiation temperature distribution of each pixel; (c) projecting the panoramic thermal map onto a unit sphere centered on the user, and calculating the long-wave radiation flux by integrating the spherical angle; and calculating the short-wave radiation flux based on a solar radiation geometric model combined with the solar altitude angle, shadow state, and sky view factor; (d) calculating the average radiation temperature by fusing the total long-wave and short-wave radiation; and (e) controlling the screen of the AR glasses or the intelligent terminal to superimpose the heat map and / or the average radiation temperature calculated based on the heat map on a real scene or a virtual environment to realize spatialized heat load visualization.

[0075] This invention, a portable AR-based urban thermal comfort assessment system and method, exhibits significant technical advantages in many aspects. Combining AR glasses and a smart mobile terminal, it combines excellent mobility and portability, eliminating the high deployment costs and inflexibility limitations of traditional fixed sensor equipment and meeting the needs of personalized urban thermal environment assessment. By leveraging image-sensor fusion technology, combined with a deep learning model to identify surface materials and calibrate temperature, and using a generative adversarial network to convert RGB images into panoramic infrared thermal maps, the system then calculates the radiant flux through methods such as spherical angle integration. This effectively improves the accuracy of surface temperature and radiation parameter estimation, effectively capturing thermal distribution characteristics in complex urban environments and achieving accuracy close to that of mainstream simulation methods without the need for intensive modeling. Furthermore, the system utilizes a lightweight network architecture and supports model quantization and device-side deployment optimization, achieving near-real-time processing. Image acquisition and thermal comfort estimation can be completed in 2-5 seconds, meeting the time requirements of most urban pedestrian environments. It also supports real-time on-site deployment and automatic processing, demonstrating excellent system robustness. In addition, combined with augmented reality technology, heat maps and thermal comfort indicators can be superimposed on real scenes in real time, and heat distribution can be simulated in a virtual environment to provide users with intuitive spatial heat load visualization, enhancing interactivity and practicality. It is suitable for various scenarios such as urban microclimate monitoring and human thermal environment response assessment, and has outstanding advantages in the intersection of intelligent environmental monitoring and ergonomics.

[0076] The following further describes specific embodiments of the present invention, its algorithm examples and experimental verification.

[0077] This embodiment provides a compact thermal comfort assessment system and method with image perception and augmented reality capabilities. These systems address existing thermal comfort assessment technologies, which suffer from poor portability, limited spatial coverage, insufficient estimation accuracy, and a lack of real-time visual interaction. Based on head-mounted AR glasses and a mobile computing platform, the system integrates image acquisition, thermal map estimation, and interactive display, supporting personalized, spatialized, and visual assessments of the urban outdoor thermal environment.

[0078] like Figure 2 As shown in the figure, the system architecture uses AR glasses as the front-end device, with an integrated high-resolution camera to capture RGB street view images from the user's perspective, and simultaneously transmits the image data wirelessly to a connected smartphone. The phone's built-in image processing and thermal estimation algorithms convert the raw RGB images into radiant heat maps. Based on scene semantic analysis and surface property modeling, the system infers the surface temperature distribution of each area. The system further calculates thermal comfort indicators such as PET or UTCI based on the visually estimated average radiant temperature and matching background parameters from a meteorological database, generating structured and spatially characterized thermal information.

[0079] To enhance the user experience, this system incorporates augmented reality (AR) technology, overlaying thermal maps and thermal comfort results on the user's street view in real time via AR glasses. This allows users to perceive the heat load characteristics of their surroundings while observing them naturally, thereby enabling spatial awareness of thermal environmental conditions and guiding their behavior. Furthermore, this system supports algorithm execution in virtual environments, loading virtual street views or city digital twin models to simulate and analyze thermal environmental responses under planned scenarios, providing technical support for urban design, outdoor activity simulation, and thermal health education.

[0080] 1. Hardware structure:

[0081] The system consists of a portable evaluation terminal consisting of AR glasses and a smartphone / tablet. The AR glasses are equipped with a high-definition camera lens to capture real-time street view images from the user's first-person perspective. The smartphone or tablet serves as the main computing platform, receiving image data via a wireless connection and performing image preprocessing, thermal map inference, and thermal comfort calculation. Image acquisition and AR display functions are implemented on the glasses, while all core algorithms run on the mobile device, achieving lightweight and high-performance distributed collaboration.

[0082] 2. Image processing and heat map estimation:

[0083] The system processes the collected RGB panoramic images through built-in image processing algorithms, including image dedistortion, semantic segmentation, material recognition, and surface property estimation. Based on physical models and machine learning methods (such as deep network training to predict surface radiation temperature from RGB images), the radiation temperature distribution of each object in the street scene is estimated. In this patent, the calculation of the key indicator average radiation temperature depends on the integration of the whole-sphere radiation environment, including long-wave radiation from different directions. Compared with traditional single-frame infrared images that can only estimate the field of view in one direction, panoramic thermal maps achieve radiation angle integration that conforms to physical laws by mapping the image to the unit sphere, especially having significant advantages in radiation calculation in street canyons and complex occlusion areas.

[0084] 3. Calculation of thermal comfort index:

[0085] Based on thermal map data and the user's location, the system preliminarily estimates average radiant temperature and thermal comfort indicators such as PET and UTCI. The system can utilize background meteorological parameters (such as temperature and wind speed) from existing meteorological databases as supplementary input to improve the completeness and applicability of these estimates. All calculations are performed on the mobile device, ensuring processing speed and scalability.

[0086] 4. Interaction and visualization:

[0087] The system presents results through a mobile app, supporting augmented reality (AR) mode, overlaying real-time estimated heat maps onto the actual street view seen on the user's phone, allowing users to intuitively visualize the distribution of heat loads across different spatial areas. Furthermore, the system supports loading the model and performing thermal comfort prediction analysis within virtual scenarios (such as VR environments and digital twin neighborhoods), expanding its application in urban design, outdoor activity simulation, and other scenarios.

[0088] Figure 3 This flowchart illustrates a method for portable thermal comfort assessment. The method includes generating panoramic infrared thermal images using a generative adversarial network and integrating the longwave radiation flux at the observation point based on this thermal image. This provides the core technical support for achieving high-spatial-resolution radiation estimation at street-level scale, significantly different from traditional point-by-point measurements and single-frame thermal image presentation. Furthermore, a solar radiation model is used to estimate shortwave radiation and calculate the mean radiation temperature, thereby constructing a complete thermal radiation flux model. To improve estimation accuracy and performance, a deep learning model is used to identify surface materials in the image and perform temperature calibration based on different emissivities, thereby enhancing the visual realism and accuracy of the thermal image. Similarly, the system supports the further calculation of comprehensive thermal comfort indicators such as PET and UTCI. In most application scenarios, mean radiation temperature serves as a widely applicable indicator of environmental thermal perception, and subsequent indicator calculations can be flexibly configured according to specific needs.

[0089] Figure 4 The hardware structure of the device is demonstrated. The system consists of AR glasses and a mobile device (such as a smartphone). By combining image acquisition, deep learning inference, and physical modeling, it automatically estimates the heat load distribution and thermal comfort indicators of the user's street environment.

[0090] The overall operation process of the system is as follows:

[0091] 1. System composition and startup method

[0092] like Figure 2 As shown, a user puts on AR glasses equipped with a camera module and launches a dedicated mobile app. The glasses capture real-time RGB images of the street scene from the user's perspective and transmit them wirelessly to the mobile phone. The system automatically records the current time and GPS coordinates and accesses external meteorological data sources to obtain environmental parameters such as air temperature and total solar radiation at the corresponding location. All image processing and thermal comfort estimation are performed by the mobile phone, and the final results are displayed on the glasses or mobile phone screen via the AR interface.

[0093] 2. Convert RGB image to thermal image

[0094] 1. Image acquisition and preprocessing:

[0095] After the first-person RGB street view images captured by AR glasses are transmitted to the mobile device, image dedistortion and illumination normalization are first performed to reduce the interference of lens distortion and ambient light differences on subsequent reasoning.

[0096] 2. Surface material recognition model:

[0097] like Figure 2 As shown, in order to further improve the physical accuracy of surface temperature estimation, the system introduces an image semantic segmentation model based on a deep neural network to identify surface materials in street view images. The model can adopt any of the classic DeepLabV3+, PSPNet, HRNet and other architectures. After training on large-scale street view images and material annotation datasets (such as Cityscapes, ADE20K), it can achieve pixel-level segmentation of typical material areas in street view images. Based on the thermal radiation characteristics of each type of material, the system assigns its corresponding infrared emissivity parameter (ε), which is used to calibrate the conversion of radiation brightness into actual surface temperature. Typical values ​​of emissivity ε are shown in Table 1 below (the numerical values ​​refer to relevant literature on self-radiation heat and thermal engineering standards):

[0098] Table 1

[0099]

[0100] In this embodiment, the system generally adopts the median value or high-confidence measured value of each type of material as the default parameter, and reserves a user-defined modification entry to adapt to special materials or local correction needs.

[0101] 3. Heatmap Generation Model (IRGAN, an information retrieval model based on Generative Adversarial Networks (GANs)):

[0102] After material recognition, the image is fed into our proprietary generative adversarial network (IRGAN) as input. Using a GAN framework, it maps RGB street scene images and environmental parameters into high-resolution infrared thermal images under supervised conditions, thereby predicting the radiant temperature of each surface pixel in the scene. The IRGAN network inputs include:

[0103] RGB street view images (RGB values ​​will be normalized before input);

[0104] Time information (month, day, hour, and minute, which is used to calculate the solar altitude angle through the formula after input and serves as the condition vector of the model);

[0105] GPS coordinates (latitude and longitude, used for the calculation of the above solar altitude angle);

[0106] Real-time ambient temperature (°C), (provided by public meteorological data sources);

[0107] Solar radiation intensity (W / m²), (provided by public meteorological data sources)

[0108] This non-image information is fed into the network's conditional modules through concatenation or mapping, participating in multimodal learning. The paired infrared thermal and RGB image dataset used for IRGAN model training is primarily self-collected. The paired street scene images, captured simultaneously with a FLIR infrared camera and a standard visible light camera, cover outdoor scenes under varying time of day, weather, lighting conditions, and urban conditions. The original images undergo data augmentation (random adjustments to brightness, contrast, and hue) and are then synthesized into an extended set to improve model generalization.

[0109] The IRGAN backbone uses a U-Net architecture, with the encoder using a pre-trained ResNet backbone and the decoder combining multi-scale upsampling and skip connections. The discriminator uses a PatchGAN to perform a block-by-block comparison between the local structure of the predicted heat map and the actual heat map, improving the spatial consistency and realism of the generated image. The loss function design combines the dual goals of physical consistency and visual consistency, mainly including:

[0110] L1 regression loss is used to constrain the difference between the generated heat map and the real heat map in each pixel temperature value, keeping the overall value close:

[0111]

[0112] in is the true radiation temperature of the i-th pixel, To predict the temperature, N is the total number of pixels;

[0113] Perceptual loss is used to constrain the similarity between generated images and real images in the high-level semantic feature space, represented by the intermediate features extracted by pre-trained image recognition models (such as VGG):

[0114]

[0115] in, Represents the features extracted at layer j, and M is the number of selected feature layers

[0116] Adversarial loss, used to guide the generator to produce more "realistic" heat maps, fooling the discriminator so that it cannot distinguish between real and fake images

[0117]

[0118] Among them, x is the input RGB image and conditional data, y is the real thermal image, G(x) is the generator output, and D is the discriminator output confidence

[0119] 3. Mean radiant temperature (T mrt) and UTCI estimation

[0120] 1. Longwave radiation integral:

[0121] The system regards the generated panoramic heat map as a unit sphere projection map centered on the user, and uses the isotropic integration method to calculate the long-wave radiation from all directions to obtain the total long-wave radiation flux at the current observation point:

[0122]

[0123] Where: L i – The amount of long-wave radiation emitted by a point in the panoramic image that reaches the observer

[0124] φ,θ – the longitude and latitude coordinates of the point

[0125] T i – Surface temperature at that point (°C)

[0126] n – the total number of pixels in the panorama

[0127] 2. Shortwave radiation estimation:

[0128] In this paper, the shortwave radiation estimation adopts the SOLWEIG (SOlar and LongWave Environmental Irradiance Geometry) model, which is widely used in urban outdoor thermal comfort research, as its theoretical basis. Based on the principles of solar radiation geometry and thermal radiation transmission, this model can calculate the shortwave radiation value received by the human body at any time and space position, taking into account the conditions of surface and building shielding:

[0129]

[0130] where K dir , K diff and G represent direct solar radiation, diffuse solar radiation, and global solar radiation, respectively. This data can be obtained in two ways: if the system is connected to the Internet and public data is available, real-time solar radiation intensity data (unit: W / m²) can be automatically obtained from the open interface of the local observatory or meteorological station; if it is used offline, the built-in typical day database can be used to obtain historical representative radiation values ​​by combining date and location information for approximate estimation. hiThe shadow mask is a binary matrix, where the value of direct sunlight is 1 and the value of shadowed area is 0. This value can be manually entered by the user to reflect whether the user is in the direct sunlight area. α represents the average albedo of the surface, which is used to estimate the shortwave radiation reflected by the ground. The typical average albedo in an urban environment is 0.15. sin(η) represents the sine value of the solar altitude angle η, which is used to perform vertical projection correction on the direct radiation flux. φ is the sky view factor (SVF), which is used to estimate the contribution of the visible sky to the diffuse radiation. In this system, the SVF value is obtained by analyzing the panoramic thermal map generated in the previous step: Based on the numerical statistics of a large number of collected infrared images and existing literature records, it can be approximately considered that the temperature T of all pixels is i The area with a temperature <5℃ corresponds to the sky area. Based on the statistical ratio of sky pixels, the SVF value can be derived.

[0131] In an embodiment of the present invention, the current solar altitude angle η can be derived from the Earth orbit parameter model based on the image acquisition time and geographic location information:

[0132]

[0133] Where φ is the solar declination, is the latitude of the observation point, and h is the hour angle, calculated by converting local time to longitude. The system then combines the calculated direct, scattered, and reflected shortwave radiation to form the total shortwave radiation input for the human exposure location, which is then used to estimate thermal comfort indicators such as average radiant temperature.

[0134] 3. Calculation of average radiant temperature and thermal comfort index:

[0135] Traditional methods for measuring mean radiation temperature usually rely on black globe thermometers or fixed thermal environment sensors. These devices have limited measurement viewing angles, high deployment and maintenance costs, and insufficient flexibility. However, the present invention uses panoramic RGB images and the infrared thermal images generated by them, and through spherical coordinate reprojection, simulates the radiation energy received by a human body in a street scene from a full spherical viewing angle (360° × 180°). This method relies only on portable image acquisition modules and lightweight inference algorithms to achieve high-precision estimation of mean radiation temperature, giving thermal comfort assessment technology unprecedented scalability and universality. According to the sum of long-wave and short-wave radiation fluxes S str , human body emissivity ε p (typically 0.97) and the Stefan-Boltzmann constant σ (5.67 x 10-8 W / m 2 k 4 ), calculate the average radiation temperature around the user (T mrt ):

[0136]

[0137] Based on this, combined with air temperature, wind speed, and relative humidity parameters (which can be obtained from meteorological data sources or user input), a thermal comfort model (such as UTCI or PET) is used to estimate human heat load and output a heat stress level and comfort index. This process is highly adaptable and can reflect the thermal environmental impacts caused by micro-scale street geometry and material heterogeneity.

[0138] 4. AR / VR Real-time Visualization

[0139] In AR mode, the system will overlay thermal map information on the mobile phone camera screen, use pseudo-color mapping to show the heat load status of different spatial areas, support head perspective tracking and layer interaction, allowing users to perceive thermal environment differences in real street scenes.

[0140] In VR mode, the system can deploy the heat map estimation model in a digital twin block or virtual reality scene, simulate the outdoor thermal environment distribution under given meteorological parameters, and support extended applications in scenarios such as urban design, heat island analysis, or outdoor activity simulation.

[0141] This implementation method comprehensively utilizes image intelligent perception, physical modeling and AR visualization technology to achieve portable, high-precision, spatial perception and interactive display of the urban thermal environment, and has good practicality and promotion prospects.

[0142] Experimental example

[0143] To verify the feasibility and accuracy of the system and method described in this invention in a real urban environment, the applicant conducted field experiments in a region, selecting several typical urban blocks as test samples. During the experiments, sensors, infrared cameras, and other equipment were used to collect data, and algorithms were developed to run the data.

[0144] At the street level, a large number of paired panoramic RGB street view images and corresponding infrared thermal images at corresponding time points were collected, covering different time periods (morning, noon, and afternoon) and different street types (high-rise canyons, open squares, and green space boundaries). This served as a real-world dataset for training and testing the GAN model. The image conversion model described in this paper enables the generation of infrared thermal maps from street view RGB images. These maps incorporate environmental parameters such as time, geographic location, solar radiation, and air temperature, providing fundamental information for surface temperature prediction under diverse meteorological conditions. In terms of temperature prediction accuracy, the model performs well in estimating surface temperatures in complex urban thermal environments. Compared to measured infrared imagery, the system-generated maps achieve a mean absolute error (MAE) of 3.27°C and a root mean square error (RMSE) of 4.61°C. Visually, the infrared maps generated by the model accurately capture key spatial distribution features of the urban thermal environment, revealing the true characteristics of hotspot distribution, sunlight intensity, and material thermal response. In contrast, although current mainstream urban microclimate simulation software (such as ENVI-met) can provide relatively detailed local thermal environment modeling results (MAE ±2–4°C), its accuracy is highly dependent on input parameters (such as ground material, vegetation distribution, building geometry, etc.) and grid configuration, and the modeling process is complex and the simulation calculation is time-consuming.

[0145] For thermal comfort assessment, this invention uses high-precision mobile sensors to simultaneously measure air temperature and globe temperature on-site, verifying the mean radiant temperature estimated by the proposed system. Using the proposed radiation integration calculation process and numerical modeling approach, the error between the estimated mean radiant temperature and the measured value is within acceptable limits, with a mean absolute error of 4.70°C and a root mean square error of 5.83°C. Compared to existing mainstream simulation methods such as ENVI-met (which typically have an RMSE of ±5-7°C for mean radiant temperature estimation), this invention achieves near-standard accuracy without requiring extensive modeling. It also supports real-time deployment and automated processing on-site, demonstrating its superior precision and robustness in practical applications.

[0146] In summary, the embodiments of the present invention propose a portable AR-based intelligent perception method and device for urban thermal environment, which uses AR glasses and smart phones to build a portable terminal. Through image-sensor fusion technology, combined with deep learning and generative adversarial networks, high-precision panoramic infrared thermal map generation and radiation parameter estimation are achieved, and accuracy close to that of mainstream simulation methods can be achieved without high-intensity modeling. The present invention adopts a lightweight network structure and supports model quantization and terminal deployment optimization, and has the technical foundation for expansion in the direction of high frame rate real-time performance. In addition, the algorithm operation and thermal map rendering results in the design of the present invention can support a "near real-time" processing mode, that is, image acquisition and thermal comfort estimation are completed within 2-5 seconds, which meets the time requirements for heat load perception and response in most urban pedestrian environments. Therefore, even if 60FPS continuous rendering has not yet been fully achieved under existing hardware conditions, this system can still effectively support the practical needs of daily thermal comfort assessment.

[0147] Compared with the prior art, the present invention has the following technical advantages and effects:

[0148] 1. More portable and adaptable evaluation method: This system eliminates traditional sensor modules and relies solely on image acquisition and edge computing, significantly improving portability and making it suitable for a variety of use scenarios, including daily travel for individual users and field visits by researchers.

[0149] 2. Image-driven, wide spatial coverage: Leveraging the field of view and panoramic image acquisition capabilities of AR glasses, efficient thermal field restoration of complex street environments can be achieved, overcoming the limitations of traditional infrared acquisition equipment, which have limited coverage and lack of street view semantics.

[0150] 3. Enhanced visualization and interactivity of results: AR displays integrate thermal maps with real-world scenes, allowing users to obtain thermal environment perception feedback without switching devices or viewing angles, making the assessment process more intuitive and real-time.

[0151] 4. Broad application potential: This invention can be used not only to assess the thermal comfort experience of ordinary users during daily travel, but can also be expanded to professional fields such as urban planning, smart city management, and emergency high-temperature monitoring, and has good market application prospects.

[0152] In summary, the present invention focuses on key links such as image acquisition, thermal estimation, and AR augmented reality display, and constructs an intelligent assessment system with mobile devices as the core, images as input, and thermal comfort as output. It is significantly superior to existing technologies in structural design, assessment accuracy, and user experience, and has good innovation and application prospects.

[0153] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0154] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0155] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0156] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc or a read-only optical disc (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0160] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0161] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0162] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0163] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0164] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0165] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A portable urban thermal environment intelligent perception method based on AR, characterized in that: The following steps are involved: S1. Use AR glasses to collect panoramic RGB street view images from the user's perspective, obtain time and geographic location information, and call meteorological data sources to supplement air temperature and solar radiation intensity parameters; S2. Map the RGB image and environmental parameters into a panoramic infrared thermal image through a generative adversarial network to obtain the radiation temperature distribution of each pixel; S3, projecting the panoramic heat map onto a unit sphere centered on the user, and calculating the long-wave radiation flux received by the observation point by integrating the spherical angles; Based on the solar radiation model, the shortwave radiation flux is calculated by combining the solar altitude angle, shadow state and sky view factor; S4. Combine the total amount of longwave and shortwave radiation to calculate the average radiation temperature; S5. Through AR glasses or smart terminal screens, the heat map and / or the average radiation temperature calculated based on the heat map is superimposed on the real scene or virtual environment to achieve spatial heat load visualization.

2. The method according to claim 1, wherein Step S2 also includes: identifying the surface material category in the RGB image based on the deep learning model, and calibrating the surface temperature according to the material emissivity; The surface material recognition uses a semantic segmentation network to achieve pixel-level classification and assigns a preset emissivity value based on the material category; The emissivity value is set according to the thermal radiation characteristics of the material and is used to calibrate the radiant brightness output by the generative adversarial network to the actual surface temperature.

3. The method according to claim 1 or 2, wherein: In step S2: The generative adversarial network uses U-Net as the generator backbone, the encoder adopts a residual pre-training module, and the decoder combines multi-scale upsampling and skip connections. The discriminator adopts a local block discriminant structure to judge the authenticity of local areas of the heat map. The physical consistency and spatial authenticity of the generated heat map are constrained by combining L1 regression loss, perceptual loss and adversarial loss. Among them, the L1 regression loss constrains pixel-level temperature differences; the perceptual loss extracts high-level feature similarity through a pre-trained image recognition model; and the adversarial loss guides the generator to output a thermal distribution image that conforms to the actual thermal distribution characteristics.

4. The method according to any one of claims 1 to 2, characterized in that In step S3: The long-wave radiation flux calculation adopts the conformal integration method to map the panoramic heat map pixels to the unit spherical coordinates centered on the user, and realizes the full-view radiation energy calculation through the physical integration of longitude φ and latitude θ.

5. The method according to any one of claims 1 to 2, characterized in that In step S3: The shortwave radiation flux calculation is based on the solar radiation geometric model. By analyzing the sine value of the solar altitude angle, the binary state of the shadow mask and the surface albedo, the following components are calculated: the direct radiation component of the area directly exposed to sunlight; the scattered radiation component of the area with visible sky; and the ground reflected radiation component.

6. The method according to claim 5, wherein: The solar altitude angle is determined by the acquisition time and the geographical location through an orbital parameter model; The sky viewing factor is obtained by counting the pixel ratio of the sky in the low temperature area in the panoramic thermal map pixels.

7. The method according to any one of claims 1 to 2, characterized in that Step S4 also includes: combining environmental parameters such as air temperature, wind speed and humidity parameters, and outputting a structured thermal comfort index through a thermal comfort model; step S5 also includes: superimposing the thermal comfort index on a real scene or a virtual environment.

8. The method according to any one of claims 1 to 2, characterized in that Step S5 includes two modes: AR mode: Overlays the real-time generated heat map onto the real scene through pseudo-color mapping, and supports head tracking interaction; VR mode: Load the heat map generation algorithm in a virtual environment or digital twin model to simulate the heat distribution under given parameters.

9. The method according to any one of claims 1 to 2, characterized in that: The meteorological data is obtained in real time by calling a public interface through the Internet, or is approximately matched offline through a built-in typical day database combined with time and location information.

10. A portable urban thermal environment intelligent sensing device based on AR, characterized in that: include: AR glasses with an integrated camera module for capturing panoramic RGB street view images from the user's perspective; an intelligent terminal, communicatively connected to the AR glasses, comprising a processor and a memory; The processor is configured to perform: (a) obtaining the panoramic RGB street view image, current time and geographic location information, and calling a meteorological data source to supplement air temperature and solar radiation intensity parameters; (b) Using a generative adversarial network, the RGB image and environmental parameters are mapped into a panoramic infrared thermal image to obtain the radiation temperature distribution of each pixel; (c) Project the panoramic heat map onto the unit sphere centered on the user and calculate the longwave radiation flux by integrating the spherical angles. The shortwave radiation flux is calculated based on the solar radiation geometry model combined with the solar altitude angle, shadow state and sky view factor; (d) Calculate the mean radiation temperature by combining the total longwave and shortwave radiation; (e) controlling the AR glasses or smart terminal screen to superimpose the heat map and / or the average radiation temperature calculated based on the heat map on the real scene or virtual environment to achieve spatialized heat load visualization.

Images