Outdoor mixed reality illumination compensation method and system based on multi-source environment perception and real-time weather data

Abstract

The invention provides an outdoor mixed reality illumination compensation method and system based on multi-source environment perception and real-time weather data, and the method comprises the steps: obtaining multi-source data: providing original data of'space-time reference + environment characteristics' for all subsequent illumination calculation, solving the problems of incomplete coverage and insufficient precision of a single data source (such as only a camera), and obtaining the multi-source data; the data is a data basis for real-time illumination compensation; main light source resolving: the'spatio-temporal data 'is converted into'virtual main light source parameters' through a physical algorithm, it is ensured that the direction and intensity of a virtual main light source (sun / moon) are completely consistent with those of the real world, and the problems of'shadow direction error 'and'illumination angle splitting' are solved; through GPS space-time positioning, network weather API data and local visual sensor fused illumination compensation, a real-time environment illumination shadow relationship matches a virtual world with a real world, a more immersive mixed reality experience is presented, and a split feeling is reduced.

Description

Technical Field

[0001] This invention relates to the field of illumination compensation technology, specifically to an outdoor mixed reality illumination compensation method and system based on multi-source environmental perception and real-time weather data. Background Technology

[0002] Existing Technology 1: Pre-baked Lightmaps. Technical Principle: Textures are baked onto virtual objects using a preset lighting environment. Technical Defects: Inability to handle sudden weather changes: When clouds obscure the real light, the virtual object remains bright, appearing to "float" in the air. Incorrect Shadow Direction: As the sun's angle changes over time, the shadows of virtual buildings differ in direction from those of real trees. Color Temperature Mismatch: The cool tones of dawn and the warm tones of dusk are not reflected in real-time on the virtual object.

[0003] Existing Technology 2: Brightness Matching Based on Camera Image. Technical Principle: Adjusts the lighting of mixed reality content based on the ambient brightness identified by the camera. Technical Limitations: Unable to resolve camera occlusion bias: When the camera is occluded in different environments, the lighting identified will deviate from the real-world lighting, resulting in a mismatch in virtual objects. Nighttime Recognition Issues: If virtual objects are simply rendered black at night without a main light source, they will be difficult to see; if they are too bright, they will resemble "will-o'-the-wisps." Bias Exists in Special Weather Conditions: For example, in snowy weather, reflectivity increases significantly. If virtual objects are rendered based on the reflection of ordinary ground, the bottom will appear very dark and completely unrealistic.

[0004] In outdoor MR scenes, "lighting consistency" is a core factor determining whether users will feel "out of immersion." Imagine the real world is overcast, while your virtual palace is cast in harsh shadows by bright sunlight. Or, in snowy weather, the ground's reflectivity jumps dramatically from 0.2 (grass / soil) to 0.8 (snow). If virtual objects are rendered using the same ground reflectivity, their bottoms will appear extremely dark and unrealistic. Similarly, at night, without the sun as the primary light source, the lighting environment becomes extremely complex (moonlight + streetlights + landscape lighting). If virtual objects are simply rendered black, they will be difficult to see; if they are too bright, they will resemble "will-o'-the-wisps." Currently, outdoor MR applications typically use pre-baked lightmaps or rely solely on simple brightness matching from the camera image, which cannot solve these problems. This disconnect instantly destroys the sense of immersion. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide an outdoor mixed reality lighting compensation method and system based on multi-source environmental perception and real-time weather data to solve the problems mentioned in the background. The present invention has a novel structure and uses lighting compensation fused from GPS spatiotemporal positioning, network meteorological API data and local visual sensors to match the real-time ambient lighting and shadow relationship between the virtual world and the real world, presenting a more immersive mixed reality experience and reducing the sense of disconnect.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data, wherein the compensation method includes:

[0007] S1. Multi-source data acquisition: Provides raw data of "spatiotemporal reference + environmental features" for all subsequent illumination calculations, solving the problem of incomplete coverage and insufficient accuracy of a single data source (such as only cameras), and is the "data foundation" for real-time illumination compensation;

[0008] S2. Main Light Source Calculation: The "spatiotemporal data" is converted into "virtual main light source parameters" through physical algorithms to ensure that the direction and intensity of the virtual main light source (sun / moon) are completely consistent with the real world, thus solving the problems of "incorrect shadow direction" and "disjointed lighting angle".

[0009] S3. Ambient Light Intensity Estimation: Through "camera visual analysis + meteorological data fusion", the "average brightness" and "color temperature" of the real environment are accurately obtained, solving the problems of "color temperature of virtual objects not matching the real environment" (such as cool color tone in the morning vs. warm color tone in virtual objects) and "brightness deviation causing virtual objects to be too bright / too dark".

[0010] S4. Dynamic correction of ambient light reflectance: The ambient light reflectance parameter of virtual objects is dynamically adjusted according to "weather type + visual characteristics" to solve the problem of "inconsistency between the reflection of virtual objects and the real environment under special weather conditions" (such as the bottom of virtual objects being too dark in snowy weather, or the lack of wet reflection in rainy weather).

[0011] S5, Lighting Model Fusion: Integrates "main light source parameters, ambient light parameters, and reflectivity parameters" into the lighting model (shader) of the rendering engine to achieve collaborative work of multiple parameters and avoid lighting logic chaos caused by the independent action of a single parameter;

[0012] S6. Real-time rendering: The parameters after the lighting model is fused are applied to virtual objects to generate an image that "seamlessly matches the lighting of the real environment" and superimposes it onto the real scene. This is the "immersive experience" that users ultimately perceive.

[0013] Furthermore, according to step S1, the key technical details of the multi-source data acquisition include:

[0014] (1) Data synchronization mechanism: The device timestamp (UTC) is used to align the GPS positioning, 6DoF attitude and camera frame to ensure that the multi-source data at the same time point corresponds to the same environmental state (error ≤ 50ms).

[0015] (2) Data caching strategy: Locally cache the meteorological data of the last 5 minutes. When the network is interrupted, the cached data is automatically used and decayed over time (e.g., the accuracy is reduced by 10% every 30 seconds) to avoid system failure;

[0016] (3) Data validity verification: GPS data and meteorological data.

[0017] Furthermore, according to step S2, the refinement steps for calculating the main light source include:

[0018] (1) Basic parameter preprocessing: time transformation, coordinate transformation and attitude calibration;

[0019] (2) Calculation of the position of the main light source;

[0020] (3) Adjust the characteristics of the main light source based on meteorological data: According to the real-time weather type, correct the "shadow parameters" and "light intensity attenuation" of the main light source to ensure that it is consistent with the real environment light.

[0021] Furthermore, according to step S3, the refinement steps for estimating ambient light intensity include:

[0022] (1) Image preprocessing: downsampling, noise reduction and region segmentation;

[0023] (2) Calculation of ambient average brightness: histogram analysis, HDR estimation and weighted fusion;

[0024] (3) Environmental color temperature extraction and equipment color deviation correction: color temperature calculation, equipment color deviation correction and meteorological data fusion.

[0025] Furthermore, according to step S4, the detailed steps for dynamically correcting the ambient light reflectance include:

[0026] (1) Determination of reflectivity correction trigger conditions: Determine whether correction needs to be initiated and the correction mode by dual determination of "meteorological data + visual characteristics";

[0027] (2) Dynamic adjustment of reflectivity parameters: ground reflectivity weight enhancement, virtual object bottom shadow correction, and object surface reflectivity adaptation;

[0028] (3) Real-time verification and iterative optimization: visual feedback verification and frame rate adaptation.

[0029] Furthermore, according to step S5, the refinement steps of the lighting model fusion include:

[0030] (1) Input parameter classification and standardization: All input parameters are classified according to "lighting dimension" and converted into a standardized format (0-1 range) that the rendering engine can recognize.

[0031] (2) Shader fusion logic: Based on the extension of the "Phong Lighting Model", it integrates multiple parameters to calculate the final pixel color;

[0032] (3) Mode switching and parameter priority definition.

[0033] Furthermore, the mode switching includes: mode switching logic, daytime sunny mode, nighttime mode and snow mode, and the parameter priority is: local visual fine-tuning parameters (such as exposure compensation under the shade of trees) > ambient light parameters > main light source parameters > basic default parameters (to ensure that local environmental deviations are corrected first).

[0034] Furthermore, according to step S6, the detailed steps of real-time rendering include:

[0035] (1) Preparation before rendering: updating material parameters, diffuse texture, normal texture, transparency parameters and virtual / real position calibration;

[0036] (2) Core processing in rendering: real-time lighting calculation, per-pixel lighting, ambient occlusion (AO), real-time shadow generation, shadow mapping, shadow filtering and rain and snow particle rendering;

[0037] (3) Post-rendering optimization and MR overlay: Image post-processing and MR overlay display.

[0038] Furthermore, the image post-processing includes: anti-aliasing (MSAA 4x), tone mapping, and virtual-real color fusion; the MR overlay display includes: depth fusion and display synchronization.

[0039] An outdoor mixed reality illumination compensation system based on multi-source environmental perception and real-time weather data, the system comprising:

[0040] Main Light Source Tracking Module: In outdoor MR scenes, the sun is the only natural main light source. This module uses physical algorithms to force the virtual main light source to be completely aligned with the spatial position and lighting characteristics of the real sun, solving the problems of "incorrect shadow direction" and "disjointed lighting angle".

[0041] Night mode and artificial light source compensation module: After sunset, there is no main sun light source, and the scene lighting consists of "moonlight + artificial light source (streetlight, landscape light)". This module uses multiple algorithms to simulate complex low light environment and solves the problem of "virtual objects being too dark to see / too bright like will-o'-the-wisps at night".

[0042] Weather-based diffuse reflection controller: The "diffuse reflection intensity" and "ambient light ratio" vary greatly under different weather conditions (sunny / cloudy / rainy / foggy). This module dynamically adjusts rendering parameters based on meteorological data to solve the problem of "disconnect between weather and lighting atmosphere" (such as virtual objects still having hard shadows on cloudy days).

[0043] Special Weather Mode: Snowy weather presents two major optical problems: "high ground reflectivity" and "rain and snow obstruction". This module uses a dual judgment of "meteorological data + visual verification" to optimize the rendering logic and solve the problems of "the bottom of virtual objects is too dark in snowy weather" and "no rain and snow interaction".

[0044] Normal-based procedural snow accumulation: Virtual objects in snowy weather need to have synchronized "snow accumulation effects". Traditional solutions require artists to manually draw snow textures. This module achieves automatic snow accumulation through "software shaders + surface normal detection", solving the problem of "static snow accumulation effects that do not match the actual snow depth".

[0045] Vision-based fine-tuning (HDR environment detection): Global meteorological data and main light source algorithms cannot cover "local occlusion" (such as when a user walks into the shade of a tree or building). This module analyzes real-time camera images to correct local lighting deviations and solve the problem of "correct global lighting but local fragmentation" (such as virtual objects remaining bright under the shade of a tree).

[0046] The beneficial effects of this invention are:

[0047] 1. This invention enhances user experience: the fusion of lighting and visuals that more closely resembles the real environment in the virtual world provides users with a more realistic and immersive virtual experience.

[0048] 2. This invention is adaptable to various environments: regardless of day or night, sunny or snowy weather, the method can switch between different modes to perform lighting and visual fusion between the virtual world and the real world according to the current real-time environmental conditions, and has a wide range of environmental adaptability.

[0049] 3. This invention improves accuracy: Through multi-source fusion—that is, by combining geographic information (GPS), meteorological data (API), and local vision (Camera) for triple calibration—it provides more accurate real-time lighting environment data.

[0050] 4. Compared with the prior art, the present invention adopts a lighting compensation scheme that integrates GPS spatiotemporal positioning, network meteorological API data and local visual sensors. The system first calculates the precise azimuth angle of the sun through GPS and time; then it calls real-time meteorological data to obtain cloud thickness and visibility coefficient; finally, it combines the ambient light spherical harmonic coefficient captured by the camera to dynamically generate a "virtual sun" and "ambient skylight", which drives the rendering engine in real time, so that the brightness, shadow direction and color temperature of the virtual ruins are seamlessly integrated with the real environment. Attached Figure Description

[0051] Figure 1 This is a system control diagram of the outdoor mixed reality illumination compensation system based on multi-source environmental perception and real-time weather data according to the present invention.

[0052] Figure 2 This diagram illustrates the steps of the outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to the present invention. Detailed Implementation

[0053] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0054] Please see Figures 1 to 2 This invention provides a technical solution: an outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data, the compensation method comprising:

[0055] S1. Multi-source data acquisition:

[0056] Core positioning

[0057] It provides raw data of "spatiotemporal reference + environmental features" for all subsequent illumination calculations, solving the problem of incomplete coverage and insufficient accuracy of a single data source (such as only cameras), and is the "data foundation" for real-time illumination compensation.

[0058] Input data source classification

[0059] Based on the data source, it is divided into two categories: "device-built-in sensor data" and "network API data," which are further detailed below:

[0060] Data source type Data Items How to obtain Data format / units frequency acquisition Core Uses Built-in sensors Geographic location information MR device GPS / BeiDou module Latitude and longitude (e.g., 116.4°E, 39.9°N) 1 time / second (dynamically adjusted; when stationary, it decreases to 1 time / 3 seconds). Calculation of the main light source (sun / moon position) Built-in sensors Current UTC time Device system time (network calibration) Timestamp (e.g., 2025-11-21T14:00:00Z) Real-time synchronization (with GPS time calibration) Calculation of the main light source (the sun's position changes over time) Built-in sensors Device attitude information (6DoF) Gyroscope + Accelerometer + Magnetometer Position (x, y, z) + Attitude (roll / pitch / yaw) 10 times / second The virtual light source is matched to the user's viewing angle (to avoid light source direction deviation). Built-in sensors Camera footage Front / Rear RGB Cameras for MR Devices Image frame (resolution such as 1280×720, RGB format) 30 frames per second Ambient light intensity estimation and visual verification (e.g., snow detection). Built-in sensors Depth data (optional) Depth camera / SLAM module Point cloud data (x, y, z coordinates) / depth map 15 frames per second 3D position estimation of artificial light sources and interactive features for rain and snow occlusion Network API data Real-time weather data Calling third-party weather APIs (such as Gaode / Moji Weather Open Platform) JSON format (weather type, cloud cover, visibility, snowfall) Once every 3 minutes (triggers automatic refresh when weather changes). Main light source shadow adjustment, ambient light reflectivity correction (e.g., snowfall determination).

[0061] Key technical details

[0062] 1. Data synchronization mechanism: The device timestamp (UTC) is used to align the GPS positioning, 6DoF attitude, and camera frame to ensure that multi-source data at the same point in time correspond to the same environmental state (error ≤ 50ms).

[0063] 2. Data caching strategy: Locally cache the weather data for the most recent 5 minutes. When the network is interrupted, the cached data is automatically used and decays over time (e.g., the accuracy is reduced by 10% every 30 seconds) to avoid system failure.

[0064] 3. Data validity verification:

[0065] GPS data: If the positioning accuracy is >10 meters (such as in the transition zone between indoor and outdoor areas), the valid positioning data of the previous 3 seconds will be used automatically.

[0066] Meteorological data: If the API returns an error (such as a timeout), the "Cloudy" parameter will be used by default (neutral weather to avoid extreme deviations).

[0067] Exception handling

[0068] Sensor failure: If GPS fails, the device's built-in "location estimation" (estimates the current location based on 6DoF motion trajectory) will be enabled. If the error exceeds 50 meters, the user will be prompted to "ensure that the GPS signal is normal".

[0069] Network interruption: After the weather data cache is exhausted, it automatically switches to "pure vision mode" (which only relies on camera images to estimate ambient light, reducing accuracy but ensuring basic functions).

[0070] S2, Calculation of the main light source:

[0071] Core positioning

[0072] By using physical algorithms to convert "spatiotemporal data" into "virtual main light source parameters", the direction and intensity of the virtual main light source (sun / moon) are completely consistent with the real world, solving the problems of "incorrect shadow direction" and "disjointed lighting angle".

[0073] Input data

[0074] Outputs from the multi-source data acquisition stage: latitude and longitude, UTC time, device 6DoF attitude, and real-time meteorological data (weather type, cloud cover).

[0075] Preset parameters: Solar constant (1361 W / m², used for basic light intensity calculation), lunar illumination intensity (approximately 0.25 lux during a full moon, used as the main light source at night).

[0076] Detailed steps

[0077] Step 1: Basic parameter preprocessing

[0078] Time conversion: Convert UTC time to "local solar time" (taking into account time zones and daylight saving time) to eliminate the impact of time deviation on the sun's position;

[0079] Coordinate transformation: Convert the device's GPS geodetic coordinates (latitude and longitude) to "celestial coordinate system" (right ascension and declination) to meet the input requirements of the SPA algorithm;

[0080] Attitude calibration: Correct the sun azimuth angle based on the device's 6DoF "pitch angle" (e.g., when the user looks up / down, the virtual sun direction must match the real sun's position in the user's field of vision);

[0081] Step 2: Calculate the position of the main light source (day and night modes)

[0082] model Core Algorithm Calculate output parameters Example of computational logic (document example) Daytime mode SPA (Solar Position Algorithm) Solar altitude angle (vertical angle), solar azimuth angle (horizontal angle), solar irradiance (light intensity) Longitude 116.4°E, Latitude 39.9°N, Time 2025-11-21 14:00 → Altitude 45°, Azimuth 210° Night mode Lunar Positioning Algorithm Moon altitude angle, azimuth angle, and lunar phase coefficient (full moon = 1.0, new moon = 0.0) Lunar calendar 15th (full moon) → Moon altitude angle 30°, azimuth angle 180°, light intensity 0.25 lux

[0083] Step 3: Adjust the characteristics of the main light source based on meteorological data

[0084] Based on the real-time weather type, adjust the "shadow parameters" and "light intensity attenuation" of the main light source to ensure consistency with the real-world lighting conditions.

[0085] Sunny (cloud cover < 20%):

[0086] Shadow parameters: Shadow sharpness = 0.9 (clear, hard shadow), Shadow length = calculated based on the sun's altitude angle (the lower the altitude angle, the longer the shadow).

[0087] Light intensity correction: Solar irradiance × 0.9 (slight atmospheric attenuation);

[0088] Partly cloudy (20% ≤ cloud cover ≤ 80%)

[0089] Shadow parameters: Shadow sharpness = 0.3 (blurred soft shadow), shadow length × 1.2 (cloud scattering causes shadow boundary diffusion);

[0090] Light intensity correction: Solar irradiance × 0.4 (cloud cover reduces light intensity);

[0091] Cloudy (cloud cover > 80%):

[0092] Shadow parameters: Turn off hard shadows and only retain the "Ambient Occlusion (AO)" effect (simulating diffuse shadows without a clear main light source);

[0093] Light intensity correction: Solar irradiance × 0.1 (due to heavy cloud cover, the main light source is almost invisible).

[0094] Key technical details

[0095] "User viewpoint adaptation" for solar azimuth: The calculated "absolute azimuth" (e.g., 210°, with true north as 0°) is converted into "device relative azimuth" (based on the device's yaw angle of 6DoF) to ensure that the virtual sun's position in the user's field of view is consistent with the real sun when the user rotates the device;

[0096] Dynamic adjustment of moon phase light intensity: Moon phase coefficient × 0.25 lux (0.25 lux during full moon, 0 lux during new moon), and the moonlight is set to "cool tone" by default (color temperature 6500K, simulating the characteristics of real moonlight).

[0097] Exception handling

[0098] Polar night / polar day regions: determined by latitude and longitude (e.g., north of 66.5°N in summer), during polar day, the daytime mode is used but the light intensity is reduced (×0.6), and during polar night, the artificial light source mode is switched directly to nighttime mode.

[0099] When the sun is completely blocked (e.g., during a heavy rain): the main light source is automatically turned off, and "ambient light" is used as the main light source to avoid a complete disconnect between the virtual main light source and the real environment.

[0100] S3. Ambient light intensity estimation:

[0101] Core positioning

[0102] By combining "camera visual analysis and meteorological data fusion", the "average brightness" and "color temperature" of the real environment can be accurately obtained, solving the problems of "color temperature of virtual objects not matching that of the real environment" (such as cool colors in the morning vs. warm colors in the virtual environment) and "brightness deviation causing virtual objects to be too bright / too dark".

[0103] Input data

[0104] Outputs from the multi-source data acquisition stage: real-time frame images from cameras and meteorological data (cloud cover rate).

[0105] Output of the main light source calculation stage: Basic light intensity of the main light source (used for cross-validation);

[0106] Device parameters: camera white balance parameters, screen color deviation calibration data (factory preset).

[0107] Detailed steps

[0108] Step 1: Image preprocessing (to improve analysis accuracy)

[0109] Downsampling: Scaling the camera frame image (e.g., 1280×720) to 64×64 pixels (reducing computation while preserving the global brightness trend and avoiding interference from local bright spots);

[0110] Noise reduction: Gaussian filtering (5×5 convolution kernel) is used to remove image noise (such as street light spots and leaf reflections) to avoid brightness calculation errors;

[0111] Region segmentation: Divide the 64×64 image into "sky region" (upper 1 / 3) and "ground region" (lower 2 / 3), and calculate the brightness separately (because the brightness difference between the sky and the ground is large, processing them separately is more accurate).

[0112] Step 2: Calculation of average ambient brightness

[0113] Histogram analysis: Statistically analyze the pixel brightness histograms (0-255) of the "sky area" and "ground area", and take the brightness value with a "cumulative probability of 70%" as the "effective brightness" (to avoid the extreme effects of overly bright / overly dark pixels);

[0114] HDR estimation: If the camera supports HDR, it fuses image frames with different exposures to calculate "dynamic range brightness" (such as the real-world ambient brightness range of 0.1-10000 lux), rather than a single brightness value;

[0115] Weighted fusion: The final average brightness is calculated based on a weight of "30% for the sky area + 70% for the ground area" (because users' perspectives are more focused on the ground).

[0116] Step 3: Ambient color temperature extraction and equipment color deviation correction

[0117] Color temperature calculation: Analyze the proportions of the RGB three channels in the image (e.g., B>R for cool tones, R>B for warm tones), and map the real ambient color temperature (e.g., 5000K in the morning, 2700K at dusk) through the "Planck blackbody radiation curve".

[0118] Device color shift correction: Read the device screen's "color shift calibration data" (e.g., the screen is reddish by default, and the R channel gain is too high), calculate the RGB gain matrix (e.g., R gain = 0.9, G gain = 1.0, B gain = 1.1), and use it to counteract the effect of screen color shift on virtual content;

[0119] Meteorological data fusion: If the cloud cover is >80% (overcast), the calculated color temperature value will be corrected to "neutral color temperature (5500K)" by 10% (because the color temperature is more uniform in overcast environments, avoiding local color deviation).

[0120] Key technical details

[0121] Brightness unit conversion: Converting image pixel brightness (0-255) to actual physical brightness (lux) requires combining the camera's "exposure time, ISO sensitivity" parameters (e.g., pixel brightness 200 → corresponding actual brightness 500 lux).

[0122] Dynamic threshold adjustment: When the ambient brightness is below 50 lux at night, the sensitivity of the brightness calculation is automatically increased (e.g., a brightness value with a cumulative probability of 60% is used as the effective brightness) to avoid calculation failure due to excessively low light.

[0123] Exception handling

[0124] Camera obstruction: If the camera is detected to be obstructed (e.g., covered by a finger, causing a sharp drop in image brightness of 90%), the system will automatically use the effective brightness / color temperature data from the previous 2 seconds and prompt the user that "the camera is obstructed and the light compensation accuracy has decreased".

[0125] Extreme lighting (such as strong midday light): Image pixels are overexposed (brightness 255 percentage > 30%). Enable "multi-exposure fusion" mode to calculate the true brightness by combining low-exposure frame images to avoid underestimation of brightness caused by overexposure.

[0126] S4. Dynamic correction of ambient light reflectance:

[0127] Core positioning

[0128] The "ambient light reflectivity parameter" of virtual objects is dynamically adjusted based on "weather type + visual characteristics" to solve the problem of "inconsistency between the reflection of virtual objects and the real environment under special weather conditions" (such as the bottom of virtual objects being too dark in snowy weather or the lack of wet reflection in rainy weather).

[0129] Input data

[0130] Outputs from the multi-source data acquisition stage: camera frame images, meteorological data (weather type, snowfall / rainfall);

[0131] Outputs of the ambient light intensity estimation step: average ambient brightness and color temperature;

[0132] Virtual object model data: Basic reflectivity of object surface material (e.g., stone reflectivity 0.1, metal reflectivity 0.8).

[0133] Detailed steps

[0134] Step 1: Determine the trigger condition for reflectivity correction

[0135] By combining meteorological data and visual characteristics, it is determined whether a correction needs to be initiated and the correction mode accordingly.

[0136] Target Weather Pattern Meteorological data conditions Visual feature conditions (camera image) Correction mode High reflectivity in snowy weather Weather = Snowfall / Snow Accumulation, Snow depth > 2cm In the ground area (lower 2 / 3 of the image), white pixels account for >60%. High reflectivity ground rendering mode Rainy days with damp reflections Weather = Rainfall, Rainfall > 1 mm / h In the ground area (lower 2 / 3 of the image), the percentage of specular reflection pixels is greater than 20% (e.g., road surface reflections). Wet surface reflection mode Dust storms reflect light at low altitudes Weather = Dust storm, visibility <1km Overall image saturation decreased by more than 30%. Low reflectivity attenuation mode

[0137] Step 2: Dynamic adjustment of reflectivity parameters (taking snowy weather as an example)

[0138] 1. Increased ground reflectivity:

[0139] Normal mode: Ground ambient light reflection weight = 0.2 (only a small amount of reflection);

[0140] Snow mode: Ground reflectivity weight is increased to 0.8, and the reflected light color tone is set to "light blue" (color temperature 7000K, simulating the bluish reflectivity of real snow).

[0141] 2. Correction of the bottom shadow of virtual objects:

[0142] Shadow opacity: Reduced from 0.8 in normal mode to 0.2 (lighter shadows in snow).

[0143] Shadow tone: Overlay 10% of the blue channel (to avoid shadows appearing gray, and to match the shadows of real snow).

[0144] 3. Adaptation to object surface reflectivity:

[0145] The "ambient light reflectance coefficient" of virtual objects is adjusted according to the material type: for example, stone from 0.1→0.3, metal from 0.8→0.9 (to simulate the enhanced reflectivity of objects in a strong reflective environment like snow).

[0146] Step 3: Real-time verification and iterative optimization

[0147] Visual feedback verification: After correction, the "difference in reflection between virtual objects and real reference objects (such as real snow)" is monitored in real time through the camera. If the difference is greater than 20% (such as the virtual object reflecting too much light), the reflection weight is automatically fine-tuned (reduced by 5%).

[0148] Frame rate adaptation: If the reflection calculation causes the rendering frame rate to drop by more than 10fps (such as the specular reflection of complex materials), the "reflection calculation precision" will be automatically reduced (such as changing from pixel-by-pixel reflection to region-by-region reflection) to balance precision and performance.

[0149] Key technical details

[0150] Calculation of reflected light direction: Combining the "direction of ambient light (such as sky light, ground reflection)" and the "surface normal of virtual object", the "Phong lighting model" is used to calculate the direction of reflected light to ensure that the reflection position is consistent with the real physical laws;

[0151] Differentiated processing for multiple materials: A uniform reflectivity is not used for all objects. For example, the reflectivity correction range of virtual cloth (+0.1) is much lower than that of virtual metal (+0.3), which is consistent with the characteristics of real materials.

[0152] Exception handling

[0153] False trigger detection: If a white wall is mistakenly identified as snow (weather data shows no snowfall, but the visual characteristics match), the system will automatically check the weather data for the past 3 minutes. If there is no snowfall, the snow mode will be disabled and the system will switch to the "normal high reflectivity mode".

[0154] Extreme reflectivity (such as icy roads): The reflectivity weight is set to a maximum of 0.9 (to avoid excessive reflection that could cause virtual objects to be "overexposed"), while adding a "reflective blur effect" (to simulate the slight roughness of a real ice surface).

[0155] S5, Lighting Model Blending:

[0156] Core positioning

[0157] By integrating the "main light source parameters, ambient light parameters, and reflectivity parameters" into the lighting model (shader) of the rendering engine, the collaborative work of multiple parameters is achieved, avoiding the lighting logic chaos caused by the independent action of a single parameter.

[0158] Input data

[0159] Main light source solution output: main light source direction, intensity, and shadow parameters;

[0160] Ambient light intensity estimation output: average ambient brightness, color temperature, RGB gain matrix;

[0161] Ambient light reflectance correction output: reflectance weight, reflected light hue, object surface reflectance coefficient;

[0162] Module Six (Visual Fine-tuning) Output: Local exposure compensation coefficient (such as the darkening coefficient under the shade of a tree).

[0163] Detailed steps

[0164] Step 1: Input parameter classification and standardization

[0165] All input parameters are categorized by "lighting dimension" and converted into a standardized format (range 0-1) that the rendering engine can recognize:

[0166] Light Dimension Parameters Standardization Processing Example Main light source parameters Principal light intensity, elevation angle, azimuth angle Presence intensity 1.0 (sunny day) → Normalized to 1.0, 0.4 (partly cloudy) → 0.4 Ambient light parameters Ambient light intensity, color temperature, exposure compensation factor Ambient light intensity 0.3 (sunny day) → 0.3, 0.8 (cloudy day) → 0.8; Color temperature 2700K → normalized to 0.2 (warm), 6500K → 0.8 (cool) reflectivity parameters Ground reflectivity weight, object reflectance coefficient Reflectivity weight: 0.2 (normal) → 0.2, 0.8 (snowy weather) → 0.8 Equipment calibration parameters RGB gain matrix R gain 0.9 → 0.9, G gain 1.0 → 1.0, B gain 1.1 → 1.1

[0167] Step 2: Shader Blending Logic

[0168] Based on the extension of the "Phong Lighting Model", the final pixel color is calculated by integrating multiple parameters. The core formula is as follows:

[0169] Final pixel color = ambient light component + diffuse component + specular component + emissive component.

[0170] Parameter fusion logic for each component:

[0171] Ambient light component: Ambient light intensity × Color temperature correction factor × RGB gain matrix (to compensate for device color shift);

[0172] Diffuse component: Principal light intensity × Diffuse coefficient (base value of object material) × Shadow opacity (output of principal light source solution);

[0173] Specular reflection component: reflectivity weight × object surface reflectivity × ambient light intensity (simulating ambient reflection);

[0174] Self-emitting component: Activated in night mode, intensity = 0.3 × ambient light (avoid excessive self-emitting).

[0175] Step 3: Mode Switching and Parameter Priority Definition

[0176] Mode switching logic: Automatically switch lighting models based on "time (day / night) + weather (sunny / snowy / rainy)".

[0177] Sunny daytime: Primarily based on the "diffuse reflection component of the main light source" (accounting for 60%);

[0178] Night mode: Primarily based on "ambient light component + self-emission component" (totaling 80%).

[0179] Snow mode: Primarily based on "specular reflection component (ground reflection)" (accounting for 50%);

[0180] Parameter priority: local visual fine-tuning parameters (such as exposure compensation under the shade of trees) > ambient light parameters > main light source parameters > basic default parameters (to ensure that local environmental deviations are corrected first).

[0181] Key technical details

[0182] Shader real-time compilation: Lighting model fusion is achieved through "dynamic shader compilation". Different shader code snippets are automatically loaded in different modes (such as loading "high reflectivity shader snippets" in snow mode) to avoid redundant calculations.

[0183] Parameter interpolation transition: When switching modes (such as from sunny to cloudy), the parameters are transitioned through "0.5-second linear interpolation" to avoid abrupt changes in lighting effects (such as shadows suddenly becoming blurry).

[0184] Exception handling

[0185] Parameter conflict: For example, if "main light source intensity = 1.0 (sunny day)" but "local exposure compensation = 0.5 (under the shade)", the system will automatically calculate "main light intensity × exposure compensation coefficient = 0.5" to prioritize meeting the local environmental requirements.

[0186] Model loading failure: If a shader fragment of a certain mode fails to load, it will automatically fallback to the "basic lighting model" (only retaining ambient light + diffuse components) to ensure that the core functions are available.

[0187] S6, Real-time Rendering:

[0188] Core positioning

[0189] Applying the parameters of the merged lighting model to virtual objects generates an image that "seamlessly matches the lighting of the real environment" and overlays it onto the real scene, which is the "immersive experience" that users ultimately perceive.

[0190] Input data

[0191] Lighting model fusion output: final shader parameters (ambient light, diffuse reflection, specular reflection, emissive components);

[0192] Multi-source data acquisition and output: device 6DoF pose (for virtual object position calibration), depth camera point cloud (for virtual-real depth fusion).

[0193] Virtual object data: 3D model vertex data, material textures (such as stone textures, metal textures).

[0194] Detailed steps

[0195] Step 1: Preparation before rendering (parameter and position calibration)

[0196] Material parameter update: Writes shader parameters to the material properties of the virtual object in real time.

[0197] Diffuse texture: Overlay a "color temperature correction texture" (such as overlaying a warm yellow tone onto a diffuse texture at dusk);

[0198] Normal texture: In snow mode, "micro-bumps" are overlaid on the normal texture of the virtual object surface (simulating the roughness of snow cover);

[0199] Transparency parameter: Shadow opacity is adjusted according to the output of the main light source (e.g., shadow opacity = 0.2 on a sunny day, which means 80% opacity).

[0200] Virtual-to-real position calibration: Based on the device's 6DoF orientation, adjust the "world coordinates" and "view matrix" of the virtual object to ensure that the virtual object's position in real space is fixed (e.g., the virtual terracotta warrior always stands on the real ground and does not float with the user's movement).

[0201] Step 2: Core processing during rendering (lighting and shadow calculations)

[0202] Real-time illumination calculation:

[0203] Per-pixel lighting: For each pixel of a virtual object, the final color is calculated according to the lighting model formula (ensuring accurate lighting details, such as the highlights at the edges of the object).

[0204] Ambient Occlusion (AO): Calculates the contact area between virtual objects and the real environment (such as the ground and walls), and adds local shadows (such as the dark area where the bottom of a virtual terracotta warrior contacts the ground to avoid a "floating" feeling);

[0205] Shadows are generated in real time:

[0206] Shadow Map: Generates shadow maps of virtual objects based on the direction of the main light source. The resolution is adjusted according to device performance (e.g., 1024×1024 for high-performance devices, 512×512 for low-performance devices).

[0207] Shadow filtering: The "PCF (Percentage Closer Filter)" algorithm is used to soften shadow edges (avoiding shadows appearing as "pixel blocks" and improving realism);

[0208] Rain and snow particle rendering: In snow / rain mode, a transparent rain and snow particle model is generated and superimposed between the virtual and real images. The "occlusion effect" (particles disappear under the eaves) is achieved based on the position of virtual objects (such as building eaves).

[0209] Step 3: Post-rendering optimization and MR overlay

[0210] Image post-processing:

[0211] Anti-aliasing (MSAA 4x): Eliminates the "jagged edges" of virtual objects, matching the smoothness of the real image;

[0212] Tone mapping: Compresses a virtual image of high dynamic range (HDR) to the dynamic range (LDR) of the device screen to avoid overexposure or underexposure;

[0213] Virtual-Real Color Blending: Adjust the "saturation and contrast" of the virtual image to match the color style of the real camera image (e.g., reduce the saturation of the virtual image when the real image is too dark).

[0214] MR overlay display:

[0215] Deep fusion: Using point cloud data from a depth camera, the depth values ​​of virtual objects are aligned with the depth values ​​of the real environment (e.g., real trees are in front of a virtual building, and the pixels of the virtual building are obscured by the trees).

[0216] Display synchronization: Ensure that the rendering frame rate of the virtual image (≥30fps) is consistent with the device screen refresh rate to avoid "screen stuttering" that disrupts the immersive experience.

[0217] Key technical details

[0218] Performance optimization strategy: Use "Level of Detail" (LOD): use low-poly models and low-resolution textures for virtual objects in the distance, and use high-poly models and high-resolution textures for objects in the foreground to balance image quality and frame rate.

[0219] User viewpoint adaptation: Adjust the rendering range of virtual objects according to the user's "field of view (FOV)" to ensure that the virtual objects are always within the user's field of view (e.g., when the user turns their head, the rendering area of ​​the virtual objects follows the viewpoint change in real time).

[0220] Exception handling

[0221] Frame rate drop: If the rendering frame rate is less than 25fps, automatically reduce the rendering precision (such as turning off anti-aliasing and reducing shadow map resolution) to ensure smooth gameplay.

[0222] Virtual-real depth misalignment: If the depth camera data fails, it automatically switches to "image-based depth estimation" (estimates depth by comparing the foreground / background of the real image) to prevent virtual objects from "penetrating" real objects;

[0223] Abnormal screen color shift: If the RGB gain matrix fails, the device's factory default color shift calibration data will be automatically loaded to ensure that the colors of virtual objects do not have extreme deviations (such as all red or all blue).

[0224] An outdoor mixed reality illumination compensation system based on multi-source environmental perception and real-time weather data, the system comprising:

[0225] (1) Main light source tracking module: In outdoor MR scenes, the sun is the only natural main light source. This module uses physical algorithms to force the virtual main light source to be fully aligned with the spatial position and lighting characteristics of the real sun, thus solving the problems of "wrong shadow direction" and "disjointed lighting angle".

[0226] Input data:

[0227] Basic positioning data: latitude and longitude collected by the GPS of the MR device (e.g., in the document example: 116.4°E longitude, 39.9°N latitude), and current UTC time (e.g., 2025-11-21 14:00).

[0228] Auxiliary calibration data: Device 6DoF orientation (to ensure that the virtual light source direction matches the user's viewing angle).

[0229] Core algorithm / execution steps:

[0230] Solar position calculation: Using SPA (Solar Position Algorithm), input latitude, longitude, and time to calculate two key parameters of the true sun:

[0231] Solar altitude angle (vertical angle, document example output 45°);

[0232] Solar azimuth (horizontal angle, document example output 210°);

[0233] Virtual light source mapping: converts the calculated elevation / azimuth angles into a "virtual sun" direction vector that the rendering engine can recognize;

[0234] Forced Shadow Alignment: By binding the shadow direction of virtual objects to the "virtual sun" vector through rendering logic, it ensures that the virtual shadow is absolutely parallel to the shadow direction of objects in the real environment (such as tourists and trees).

[0235] Output result:

[0236] The direction vector and basic value of the illumination intensity of the virtual main light source (directional light);

[0237] The projection angle parameter of the shadow of a virtual object.

[0238] Key technical features:

[0239] It is based entirely on physical laws, with no subjective parameter adjustments.

[0240] It works in conjunction with the subsequent "diffuse reflection controller" (e.g., enhancing shadow sharpness on sunny days and reducing shadow hardness on cloudy days).

[0241] (2) Night mode and artificial light source compensation module: After sunset there is no main sun light source, the scene lighting is composed of "moonlight + artificial light source (street lamp, landscape lamp)". This module simulates complex low light environment through multi-algorithm fusion to solve the problem of "virtual objects are too dark to see / too bright like will-o'-the-wisps at night";

[0242] Input data,

[0243] Spatiotemporal data: GPS-calculated sunset time (to determine whether to enter night mode), current UTC time (used for moon phase calculation);

[0244] Visual data: real-time camera footage (extracting highlighted areas), depth camera / SLAM point cloud data (estimating the 3D position of the light source);

[0245] Ambient threshold: Preset ambient brightness threshold (e.g., automatically triggering night mode when it is below 50 nits).

[0246] Core algorithm / execution steps:

[0247] Moonlight simulation sub-step,

[0248] Call the lunar phase positioning algorithm: calculate the moon's azimuth and phase (full moon / new moon / crescent moon) based on the time.

[0249] Moonlight generation parameters: output "weak cool-toned parallel light" (intensity 0.1~0.2) during full moon, and turn off moonlight effect during new moon, retaining only artificial light source.

[0250] Artificial point light source reconstruction sub-steps:

[0251] Highlight area extraction: Threshold segmentation is performed on the camera image to filter pixel clusters with brightness values ​​>200 (range 0-255) (i.e., light sources such as streetlights and spotlights).

[0252] 3D position estimation: Combine depth camera / SLAM point cloud data to calculate the relative coordinates (distance, height) of the bright pixel cluster in the real 3D space.

[0253] Virtual light source generation: In the rendering engine, a "virtual point light source" is created at the corresponding 3D coordinates and matched with the brightness (e.g., street light brightness is set to 0.8~1.0) and color temperature (warm yellow street light is set to 2700K).

[0254] Self-illuminating material activation steps:

[0255] Upon detecting night mode, the "self-illumination channel" of the virtual object's material is automatically triggered:

[0256] Virtual buildings (such as palaces): Lanterns and windows automatically generate "candlelight / light self-illumination effect" (brightness 0.3~0.5, warm color tone);

[0257] Virtual cultural relics (such as terracotta warriors): If there is a decorative light source (such as a handheld torch), they will simultaneously activate self-illumination.

[0258] Output results

[0259] The direction, intensity, and color temperature parameters of parallel moonlight;

[0260] List of 3D coordinates, intensity, and color temperature of an artificial virtual point light source;

[0261] The on / off state of the virtual object's self-illuminating material, and its brightness / color temperature parameters.

[0262] Key technical features

[0263] No manual switching of the light source is required; it is fully driven automatically by environmental sensing.

[0264] Artificial light sources are matched with real light sources in terms of "position, brightness, and color temperature" (e.g., when a user approaches a real street lamp, the virtual object is illuminated by the street lamp and produces a side highlight).

[0265] (3) Weather-based diffuse reflection controller: The "diffuse reflection intensity" and "ambient light ratio" vary greatly under different weather conditions (sunny / cloudy / rainy / foggy). This module dynamically adjusts the rendering parameters through meteorological data to solve the problem of "disconnect between weather and lighting atmosphere" (such as virtual objects still having hard shadows on cloudy days).

[0266] Input data,

[0267] Meteorological API data: Real-time weather type (Sunny / Clear, Cloudy, Rain / Fog), cloud cover;

[0268] Basic lighting data: The basic intensity of the virtual main light source output by Module 1.

[0269] Core algorithm / execution steps:

[0270] The "weather-parameter" mapping logic directly drives the rendering engine parameter configuration. The specific logic table is as follows:

[0271] Weather type Directional Light Intensity Ambient Light Intensity Shadow Opacity Special effects configuration Clear 1.0 (Strong Main Beam) 0.3 (weak ambient light) 0.8 Enhance shadow sharpness and precisely align with the sun's azimuth. Cloudy 0.4 (weak main light) 0.8 (strong ambient light) 0.3 Reduce shadow hardness and increase ambient occlusion weight. Rain / Fog 0.2 (Extremely weak main light) 0.9 (Extremely strong ambient light) 0.1 1. Enable Volumetric Fog effect; 2. Added mirror reflection to virtual objects (simulating reflection from a wet surface).

[0272] Output results

[0273] The rendering engine's "directional light intensity" and "ambient light intensity" parameters;

[0274] The "Opacity" and "Sharpness" parameters for shadows;

[0275] The on / off status and intensity parameters of special weather effects (volume fog, specular reflection).

[0276] Key technical features

[0277] The parameter mapping is based on real weather optical properties (such as reflection from wet surfaces in rainy weather and light scattering in foggy weather).

[0278] It works in conjunction with the "main light source tracking system" to ensure that the intensity of the main light does not deviate from the actual position of the sun.

[0279] (4) Special weather mode: There are two special optical problems in snowy weather: "high ground reflectivity" and "rain and snow obstruction". This module uses "meteorological data + visual verification" for dual judgment and optimizes the rendering logic to solve the problems of "the bottom of virtual objects is too dark in snowy weather" and "no rain and snow interaction".

[0280] Input data,

[0281] Meteorological API data: Snowfall status (whether it is snowing), snow depth;

[0282] Visual verification data: Camera footage (the lower half of the histogram is predominantly white, confirming that there is snow on the ground);

[0283] Spatial data: user location and 3D outlines of virtual objects (such as building eaves) acquired by depth cameras / SLAM.

[0284] Core algorithm / execution steps:

[0285] High-reflectivity ground rendering sub-step,

[0286] Mode Trigger: When the meteorological API determines "snowfall" and visual verification shows "the ground is white", the "high reflectivity ground rendering mode" is activated.

[0287] Reflection weight adjustment: The rendering engine increases the calculation weight of "ground reflection upwards" from 0.2 in normal mode to 0.8;

[0288] Shadow optimization: Reduce the "opacity" of the shadow at the bottom of the virtual object from 0.5 to 0.2 and overlay a blue tone (to simulate the bluish characteristics of shadows in real snow).

[0289] Rain and snow cover and interactive sub-steps,

[0290] Mask generation: Utilize a depth camera to detect the user's head position and generate a "virtual transparent mask cylinder" (covering the user's field of view).

[0291] Occlusion detection: When a user looks up at a virtual building, if the eaves of the virtual building are above the "occlusion cylinder", the eaves will automatically "occlude" the virtual rain and snow particles (that is, the rain and snow particles disappear below the eaves, simulating the real occlusion effect).

[0292] Particle optimization: If heavy snowfall is detected, increase the density of virtual rain and snow particles, but do not track individual particles (balancing performance and realism).

[0293] Output results

[0294] High reflectivity mode switch, ground-up reflectivity weighting parameters;

[0295] The transparency and hue parameters of the shadow at the bottom of the virtual object;

[0296] The 3D range of the virtual rain and snow mask, the occlusion on / off state, and the rain and snow particle density parameters.

[0297] Key technical features

[0298] Based on the dual judgment of "meteorological data + visual verification", false triggers are avoided (such as mistakenly judging a white wall as snow).

[0299] For the first time, "virtual objects interact with rain and snow particles through occlusion," enhancing the immersive experience of the space.

[0300] (5) Normal-based procedural snow accumulation: Virtual objects in snowy weather need to synchronize with the "snow accumulation effect". Traditional solutions require artists to manually draw snow textures. This module achieves automatic snow accumulation through "software shader + surface normal detection", solving the problem of "static snow accumulation effect and mismatch with real snow depth".

[0301] Input data,

[0302] Meteorological API data: snow depth (e.g., 5cm, 10cm), duration of snowfall;

[0303] Model data: Surface normal data of virtual objects (built into the 3D model, the positive Y-axis represents the "upward face").

[0304] Core algorithm / execution steps

[0305] Snow cover parameter reading: Obtain the current snow depth from the meteorological API and convert it into "snow thickness coefficient" (e.g., 5cm corresponds to coefficient 0.5, 10cm corresponds to coefficient 1.0).

[0306] Surface normal detection: Traverse all surfaces of the virtual object in the rendering pipeline and filter out the surfaces whose normal direction is in the positive Y-axis direction (such as roofs, the top of stone lions, tabletops, and other planes that are prone to snow accumulation).

[0307] Snow texture overlay:

[0308] The selected surfaces are automatically overlaid with "white snow texture" (the texture detail varies with the snow thickness coefficient: the higher the coefficient, the coarser the texture).

[0309] Adjust the snow's "height offset" according to the snow thickness coefficient (when the coefficient is 1.0, the snow layer is offset upwards by 10mm on the object's surface to simulate a real accumulation effect).

[0310] Dynamic updates: If the weather API returns "snow depth increased", the snow thickness coefficient and texture details will be improved in real time.

[0311] Output results

[0312] "Snowflake switch" on each surface of the virtual object (only the positive Y-axis surface is active);

[0313] The superposition intensity and roughness parameters of snow texture;

[0314] Snow layer height offset value.

[0315] Key technical features

[0316] Fully automated: No artistic intervention is required; the snow effect dynamically changes with the actual snow depth.

[0317] Physical plausibility: Snow accumulation only on the "upward-facing side" conforms to real physical laws (e.g., no snow accumulation on the sides of walls or the ground);

[0318] Example effect: When it snows in real time, the rooftops and stone lion heads of the virtual Yuanmingyuan are automatically covered with snow, seamlessly blending with the real snow scene.

[0319] (6) Vision-based fine-tuning (HDR environment detection): Global meteorological data and main light source algorithms cannot cover "local occlusion" (such as when a user walks into the shade of a tree or the shadow of a building). This module analyzes the real-time image from the camera to correct local lighting deviations and solve the problem of "correct global lighting but local fragmentation" (such as virtual objects under the shade of a tree still being bright).

[0320] Input data,

[0321] Visual data: The current frame image captured in real time by the camera;

[0322] Preliminary lighting data: "Global lighting parameters" (main light intensity, ambient light intensity, etc.) output from modules one through five.

[0323] Core algorithm / execution steps.

[0324] Image downsampling: Scaling the current frame image from the camera to 64×64 pixels (reducing computation while preserving global brightness trends);

[0325] Average brightness calculation: The average brightness (Luminance) of a 64×64 image is calculated through image histogram analysis.

[0326] Brightness change detection: Compare the "current average brightness" with the "previous frame average brightness". If the difference is greater than 30% (determined as "entering the shade / shadow"), trigger "exposure compensation".

[0327] Application of compensation curve: Generate an "exposure compensation curve" based on the brightness difference to darken the overall brightness of the virtual object (if the brightness drops by 50%, the brightness of the virtual object will be darkened by 40% to 50% simultaneously) to prevent "virtual objects from glowing in the dark".

[0328] Output results

[0329] The "exposure compensation factor" for virtual objects (e.g., 0.6 represents darkening by 40%).

[0330] The duration of the compensation curve's effect (e.g., after the user leaves the shade, the global illumination parameters are restored within 1 second).

[0331] Key technical features

[0332] Millisecond-level response: Only 50~100ms is needed from detecting a sudden change in brightness to applying compensation, with no significant delay;

[0333] Localized precise correction: Only adjusts virtual objects in "local shadows" without affecting global lighting parameters (e.g., virtual objects outside the shade of trees still maintain normal brightness).

[0334] The foregoing has shown and described the basic principles and main features of the present invention and its advantages. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0335] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data, characterized in that: The compensation method includes: S1. Multi-source data acquisition: Provides raw data of "spatiotemporal reference + environmental features" for all subsequent illumination calculations, solving the problem of incomplete coverage and insufficient accuracy of a single data source (such as only cameras), and is the "data foundation" for real-time illumination compensation; S2. Main Light Source Calculation: The "spatiotemporal data" is converted into "virtual main light source parameters" through physical algorithms to ensure that the direction and intensity of the virtual main light source (sun / moon) are completely consistent with the real world, thus solving the problems of "incorrect shadow direction" and "disjointed lighting angle". S3. Ambient Light Intensity Estimation: Through "camera visual analysis + meteorological data fusion", the "average brightness" and "color temperature" of the real environment are accurately obtained, solving the problems of "color temperature of virtual objects not matching that of the real environment" (such as cool colors in the morning vs. warm colors in virtual objects) and "brightness deviation causing virtual objects to be too bright / too dark". S4. Dynamic correction of ambient light reflectance: The ambient light reflectance parameter of virtual objects is dynamically adjusted according to "weather type + visual characteristics" to solve the problem of "inconsistency between the reflection of virtual objects and the real environment under special weather conditions" (such as the bottom of virtual objects being too dark in snowy weather, or the lack of wet reflection in rainy weather). S5, Lighting Model Fusion: Integrates "main light source parameters, ambient light parameters, and reflectivity parameters" into the lighting model (shader) of the rendering engine to achieve collaborative work of multiple parameters and avoid lighting logic chaos caused by the independent action of a single parameter; S6. Real-time rendering: The parameters after the lighting model is fused are applied to virtual objects to generate an image that "seamlessly matches the lighting of the real environment" and superimposes it onto the real scene. This is the "immersive experience" that users ultimately perceive.

2. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S1, the key technical details of multi-source data acquisition include: (1) Data synchronization mechanism: The device timestamp (UTC) is used to align the GPS positioning, 6DoF attitude and camera frame to ensure that the multi-source data at the same time point corresponds to the same environmental state (error ≤ 50ms). (2) Data caching strategy: Locally cache the meteorological data of the last 5 minutes. When the network is interrupted, the cached data is automatically used and decayed over time (e.g., the accuracy is reduced by 10% every 30 seconds) to avoid system failure; (3) Data validity verification: GPS data and meteorological data.

3. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S2, the refinement steps of the main light source calculation include: (1) Basic parameter preprocessing: time transformation, coordinate transformation and attitude calibration; (2) Calculation of the position of the main light source; (3) Adjust the characteristics of the main light source based on meteorological data: According to the real-time weather type, correct the "shadow parameters" and "light intensity attenuation" of the main light source to ensure that it is consistent with the real environment light.

4. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S3, the refinement steps for estimating ambient light intensity include: (1) Image preprocessing: downsampling, noise reduction and region segmentation; (2) Calculation of ambient average brightness: histogram analysis, HDR estimation and weighted fusion; (3) Environmental color temperature extraction and equipment color deviation correction: color temperature calculation, equipment color deviation correction and meteorological data fusion.

5. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S4, the detailed steps for dynamically correcting the ambient light reflectance include: (1) Determination of reflectivity correction trigger conditions: Determine whether correction needs to be initiated and the correction mode by dual determination of "meteorological data + visual characteristics"; (2) Dynamic adjustment of reflectivity parameters: ground reflectivity weight enhancement, virtual object bottom shadow correction, and object surface reflectivity adaptation; (3) Real-time verification and iterative optimization: visual feedback verification and frame rate adaptation.

6. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S5, the refinement steps of the lighting model fusion include: (1) Input parameter classification and standardization: All input parameters are classified according to "lighting dimension" and converted into a standardized format (0-1 range) that the rendering engine can recognize. (2) Shader fusion logic: Based on the extension of "Phong Lighting Model", it integrates multiple parameters to calculate the final pixel color; (3) Mode switching and parameter priority definition.

7. The outdoor mixed reality illumination compensation method based on multi-source environmental perception and real-time weather data according to claim 6, characterized in that: The mode switching includes: mode switching logic, daytime sunny mode, nighttime mode and snow mode. The parameter priority is: local visual fine-tuning parameters (such as exposure compensation under the shade of trees) > ambient light parameters > main light source parameters > basic default parameters (to ensure that local environmental deviations are corrected first).

8. The outdoor mixed reality illumination compensation method and system based on multi-source environmental perception and real-time weather data according to claim 1, characterized in that: According to step S6, the refinement steps of real-time rendering include: (1) Preparation before rendering: updating material parameters, diffuse texture, normal texture, transparency parameters and virtual / real position calibration; (2) Core processing in rendering: real-time lighting calculation, per-pixel lighting, ambient occlusion (AO), real-time shadow generation, shadow mapping, shadow filtering and rain and snow particle rendering; (3) Post-rendering optimization and MR overlay: Image post-processing and MR overlay display.

9. The outdoor mixed reality illumination compensation method and system based on multi-source environmental perception and real-time weather data according to claim 8, characterized in that: The image post-processing includes: anti-aliasing (MSAA 4x), tone mapping, and virtual-real color fusion; the MR overlay display includes: depth fusion and display synchronization.

10. An outdoor mixed reality illumination compensation system based on multi-source environmental perception and real-time weather data implemented according to the compensation method described in claim 1, characterized in that: The system includes: Main Light Source Tracking Module: In outdoor MR scenes, the sun is the only natural main light source. This module uses physical algorithms to force the virtual main light source to be completely aligned with the spatial position and lighting characteristics of the real sun, solving the problems of "incorrect shadow direction" and "disjointed lighting angle". Night mode and artificial light source compensation module: After sunset, there is no main sun light source, and the scene lighting consists of "moonlight + artificial light source (streetlight, landscape light)". This module uses multiple algorithms to simulate complex low light environment and solves the problem of "virtual objects being too dark to see / too bright like will-o'-the-wisps at night". Weather-based diffuse reflection controller: The "diffuse reflection intensity" and "ambient light ratio" vary greatly under different weather conditions (sunny / cloudy / rainy / foggy). This module dynamically adjusts rendering parameters based on meteorological data to solve the problem of "disconnect between weather and lighting atmosphere" (such as virtual objects still having hard shadows on cloudy days). Special Weather Mode: Snowy weather presents two major optical problems: "high ground reflectivity" and "rain and snow obstruction". This module uses "meteorological data + visual verification" for dual judgment and optimizes the rendering logic to solve the problems of "the bottom of virtual objects is too dark in snowy weather" and "no rain and snow interaction". Normal-based procedural snow accumulation: Virtual objects in snowy weather need to have synchronized "snow accumulation effects". Traditional solutions require artists to manually draw snow textures. This module achieves automatic snow accumulation through "software shaders + surface normal detection", solving the problem of "static snow accumulation effects that do not match the actual snow depth". Vision-based fine-tuning (HDR environment detection): Global meteorological data and main light source algorithms cannot cover "local occlusion" (such as when a user walks into the shade of a tree or building). This module analyzes real-time images from the camera to correct local lighting deviations and solve the problem of "correct global lighting but local fragmentation" (such as virtual objects remaining bright under the shade of a tree).

Images