Infrared night vision AR intelligent glasses based on multispectral fusion

Abstract

The invention discloses infrared night vision AR (Augmented Reality) intelligent glasses based on multispectral fusion, and relates to the technical field of infrared intelligent application. High-precision scene perception in a low-light / light-free environment is realized by fusing dual-channel data of low-light imaging and infrared thermal imaging and combining cross-modal feature matching and dynamic pose compensation; an environment detail structure is restored by a low-light image enhancement technology, thermal feature targets such as organisms are accurately recognized through thermal imaging target segmentation, and a three-dimensional environment model with space geometric information and target attributes is constructed through heterogeneous feature fusion; and dynamically adjusting the display transparency of the key target in the AR view field through a virtual information layer adaptive mixing mechanism driven by a hot target attribute, and finally outputting a virtual-real fused enhanced view, thereby breaking through the illumination limitation of the traditional AR glasses, and improving the visual effect of the AR glasses. And immersive augmented reality experience with panoramic detail perception and hot target accurate positioning is provided for the user in a dark environment.

Description

Technical Field

[0001] This invention relates to the field of infrared intelligent application technology, specifically to an infrared night vision AR smart glasses based on multispectral fusion. Background Technology

[0002] As described in the published patent CN218350626U, "A Head-Mounted Glasses for Low-Light Photography and Display," augmented reality (AR) technology is a technique that calculates the position and angle of images emitted by a light engine system (also known as a projector or optical engine) in real time and adds corresponding images. Augmented reality technology can overlay the virtual world onto the real world on a screen and allow for interaction. It can simulate and overlay entity information (such as visual information, sound, or touch) that is difficult to experience in the real world within its temporal and spatial range through computers, applying virtual information to the real world. Because augmented reality technology enables interaction between the virtual and real worlds, it is currently widely used in augmented reality devices, such as AR glasses, which can project virtual images onto the human eye, achieving the overlay of virtual and real images.

[0003] Current AR glasses are mainly used in well-lit environments, such as during the day. Therefore, the use of AR glasses is limited and difficult to use in low-light / no-light environments.

[0004] As described in the published patent CN115100556B, "A method, apparatus and electronic device for augmented reality based on image segmentation and fusion", the night vision device is a night external aiming device with an image intensifier as the core component. When it is working, it does not use an infrared searchlight to illuminate the target. Instead, it uses the light reflected by the target under weak light to enhance the image on the fluorescent screen to a visible image that can be perceived by the human eye in order to observe and aim at the target.

[0005] Currently, commonly used night vision devices mainly include low-light night vision devices and infrared thermal imaging night vision devices. Weak natural light is emitted from the surface of the target and enters the low-light night vision device. Under the action of the high-power objective lens, it is focused on the photocathode surface of the image intensifier (which coincides with the back focal plane of the objective lens), exciting photoelectrons. Under the action of the intensifier, a distant target illuminated only by weak natural light is transformed into a visible light image suitable for human observation. Infrared thermal imaging night vision devices rely on the infrared radiation of the target itself to form a "thermal image", hence they are also called "thermal imagers".

[0006] Low-light night vision devices can help users see their surroundings, but it is difficult to find targets. Infrared thermal imaging night vision devices are helpful for users to find targets with infrared features, but it is difficult to see environmental details.

[0007] As described in the published patent "AR Glasses" with announcement number CN109143590B, the background technology With the development of technology, AR (Augmented Reality) glasses are increasingly appearing in daily life. AR glasses are a combination of glasses, a camera, a microprocessor, and a miniature projection device. They can overlay virtual data onto real-time images captured by the camera and display the images in front of the human eye through a miniature projection device, thus enabling a variety of applications.

[0008] Currently, AR glasses on the market do not have night vision capabilities, so they can only be used in well-lit areas at night, which limits their application environment and affects the user experience.

[0009] In summary, existing AR smart glasses have the problem of being inconvenient to use in low-light / no-light environments. Summary of the Invention

[0010] To overcome the shortcomings mentioned above, this invention aims to provide a technical solution for infrared night vision AR smart glasses based on multispectral fusion that can solve the aforementioned problems.

[0011] To achieve the above objectives, the present invention provides the following technical solution: An infrared night vision AR smart glasses based on multispectral fusion includes the following steps: S100: Acquires ambient visible light images through a low-light imaging unit and simultaneously acquires thermal radiation images through an infrared thermal imaging unit; S200: Enhance the visible light image to improve details in dark areas, and perform target segmentation on the thermal radiation image to identify thermal target features, generating an optimized visible light image and a thermal target identification map respectively. S300: Extract spatial geometric features from the optimized visible light image, extract thermal attribute features from the thermal target identification map, and achieve heterogeneous feature fusion through cross-modal feature matching to generate a fused feature map containing environmental structure and target information; S400: Construct a three-dimensional environment model based on the fused feature map, and combine it with the real-time pose data of the AR glasses to spatially register the environment model with the user's field of view through a dynamic pose compensation mechanism; S500: Generate a virtual information layer according to task requirements and perform adaptive transparency blending with the optimized visible light image, wherein the blended transparency is dynamically adjusted based on thermal target properties; S600: The mixed image is output to the user's field of view through an optical projection system to form an augmented reality view with superimposed virtual information.

[0012] As a further aspect of the present invention: step S100 includes: S110: The smart glasses are provided with a first imaging channel and a second imaging channel. The first imaging channel is equipped with a low-light sensor, and the second imaging channel is equipped with an infrared sensor. S120: Acquire an ambient visible light image through a low-light sensor in the first imaging channel, the low-light sensor being configured to respond under illumination conditions below the human eye's visibility threshold; S130: Acquire thermal radiation images through an infrared sensor in the second imaging channel, the infrared sensor being configured to sense the temperature radiation characteristics of a living organism or mechanical device; S140: Dynamically triggers dual-channel acquisition mode based on ambient light intensity. The infrared sensor is disabled when the ambient light intensity is higher than a set threshold. When the ambient light intensity is lower than the set threshold, dual-channel acquisition is started simultaneously; S150: Performs spatiotemporal alignment on the acquired dual-modal images, compensating for parallax shift caused by differences in optical paths through feature point matching.

[0013] As a further aspect of the present invention: step S200 includes: S210: Perform multi-scale detail enhancement processing on the visible light image to improve the recognizability of dark area texture features while suppressing image noise; S220: Perform adaptive threshold segmentation on the thermal radiation image to identify target areas with significant thermal radiation characteristics based on temperature gradient distribution; S230: Establish a processing priority mapping between visible light images and thermal radiation images, where: When the ambient illuminance is below the first threshold, priority is given to ensuring the real-time generation of thermal target identification maps; When a moving target is detected, the dynamic range compression efficiency of the visible light image is improved first. S240: Perform pixel-level association registration between the processed visible light image and the thermal target identification map to generate an optimized visible light image and thermal target identification map with spatial correspondence.

[0014] As a further aspect of the present invention: step S300 includes: S310: Extract a set of spatial geometric features from the optimized visible light image, wherein the spatial geometric features include edge structure descriptors and texture distribution feature vectors; S320: Extract a set of thermal attribute features from the thermal target identification map, wherein the thermal attribute features include a temperature gradient vector and a semantic description of hot spot contours; S330: Construct a mapping model between the visible light feature space and the infrared feature space, and achieve feature-level association matching through cross-modal similarity measurement; S340: Generate a fused feature map based on the matching results, where: Spatial geometric features constitute the environmental structure layer, characterizing the physical contours and surface properties of the scene; Thermal attribute features constitute the target information layer, marking the location and attributes of targets with significant thermal radiation characteristics; The two layers of features are superimposed at the pixel level through a spatial transformation matrix.

[0015] As a further aspect of the present invention: step S400 includes: S410: Generate a sparse feature point cloud based on the fused feature map, and calculate the pose of the AR glasses through visual odometry. The AR glasses are equipped with an extrinsic parameter matrix composed of a camera and an inertial measurement unit. S420: Integrates angular velocity and acceleration data from the inertial measurement unit to construct a six-degree-of-freedom motion prediction model; S430: Perform online calibration of multi-source sensors. When the difference between the motion prediction model of the inertial measurement unit and the pose of the AR glasses calculated by the visual odometry exceeds a preset threshold, the camera-IMU extrinsic parameter matrix is ​​optimized based on the reprojection error of the feature points in the environmental structure layer. When environmental texture is missing, stable thermal features in the thermal target information layer are used as an auxiliary localization reference. S440: Through a sliding window for human-computer interaction, it jointly optimizes the 3D environment model and real-time pose data, and outputs the registered augmented reality spatial coordinate system.

[0016] As a further aspect of the present invention: step S500 includes: S510: Generate a virtual information layer according to the task mode selected by the user, wherein the task mode includes at least industrial inspection mode and security monitoring mode; S520: Establish rules for mapping thermal target attributes to transparency. In industrial inspection mode, transparency is positively correlated with target temperature; In security monitoring mode, transparency is negatively correlated with the intensity of target movement; S530: Perform dynamic range compression processing on the optimized visible light image to generate a basal layer image; S540: The virtual information layer is superimposed on the base layer image, and the pixel-level transparency coefficient α is calculated in real time based on the mapping rule, where: α=λ(P,M), and the transparency coefficient is jointly determined by the target attribute vector P and the task mode parameter M; S550: Eliminates the flickering effect of virtual-real fusion caused by image jitter through time-domain filtering.

[0017] As a further aspect of the present invention: step S600 includes: S610: Decomposes the mixed image into RGB three-channel light signals and generates the initial light field through the micro-display unit; S620: Dynamically adjusts light field brightness based on ambient light intensity. When the ambient light intensity is lower than the human eye's dark adaptation threshold, the low blue light spectrum output mode is activated. When the ambient light intensity is higher than the glare threshold, local area brightness suppression is activated. S630: Performs the following modulation operation on the optical field through a diffraction waveguide structure: a) Expand the exit pupil range to cover the user's natural eye movement area; b) Compensate for optical distortion caused by field of view shift; S640: An augmented reality view matching the real-world field of view is formed on the waveguide output surface, wherein the virtual information layer is superimposed on the real scene in a retinal projection manner.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves high-precision scene perception in low-light / no-light environments by fusing dual-channel data from low-light imaging and infrared thermal imaging, combined with cross-modal feature matching and dynamic pose compensation. Low-light image enhancement technology restores the detailed structure of the environment, and thermal imaging target segmentation accurately identifies thermal feature targets such as organisms. Heterogeneous feature fusion constructs a three-dimensional environment model that combines spatial geometric information and target attributes. Then, through an adaptive mixing mechanism of virtual information layer driven by thermal target attributes, the display transparency of key targets is dynamically adjusted in the AR field of view, ultimately outputting an enhanced view that blends virtual and real elements. This breaks through the lighting limitations of traditional AR glasses and simultaneously solves the defects of single night vision devices that "see the environment but have difficulty recognizing targets" or "recognize targets but have difficulty distinguishing the environment." It provides users with an immersive augmented reality experience in dark environments that combines panoramic detail perception and accurate thermal target positioning. Attached Figure Description

[0019] Figure 1 This is a flowchart of S100-S600 in this invention; Figure 2 This is a flowchart of S110-S150 in this invention; Figure 3 This is a flowchart of S210-S240 in this invention; Figure 4 This is a flowchart of S310-S340 in this invention; Figure 5 This is a flowchart of S410-S440 in this invention; Figure 6 This is a flowchart of S510-S550 in this invention; Figure 7This is a flowchart of S610-S640 in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1-7 An infrared night vision AR smart glasses based on multispectral fusion includes the following steps: S100: Acquires ambient visible light images through a low-light imaging unit and simultaneously acquires thermal radiation images through an infrared thermal imaging unit; S200: Enhance the visible light image to improve details in dark areas, and perform target segmentation on the thermal radiation image to identify thermal target features, generating an optimized visible light image and a thermal target identification map respectively. S300: Extract spatial geometric features from the optimized visible light image, extract thermal attribute features from the thermal target identification map, and achieve heterogeneous feature fusion through cross-modal feature matching to generate a fused feature map containing environmental structure and target information; S400: Construct a three-dimensional environment model based on the fused feature map, and combine it with the real-time pose data of the AR glasses to spatially register the environment model with the user's field of view through a dynamic pose compensation mechanism; S500: Generate a virtual information layer according to task requirements and perform adaptive transparency blending with the optimized visible light image, wherein the blended transparency is dynamically adjusted based on thermal target properties; S600: The mixed image is output to the user's field of view through an optical projection system to form an augmented reality view with superimposed virtual information; By deeply integrating low-light imaging and infrared thermal imaging, and using an intelligent AR overlay mechanism, the limitations of existing AR glasses in low-light / no-light environments are effectively overcome. The core pain points of single night vision devices, such as "difficulty in recognizing targets in the environment" or "difficulty in recognizing targets in the environment", are solved, ultimately providing users with a comprehensive, high-definition, and highly interactive augmented reality experience in dark environments. Completely eliminating the strong dependence of traditional AR glasses on ambient light, by actively collecting and fusing low-light images (using weak natural light) and infrared thermal images (relying on the target's own thermal radiation), the system can work normally under harsh visual conditions such as complete darkness, smoke, and fog, significantly expanding the application scenarios of AR glasses (such as night combat, underground exploration, rescue in lightless environments, and night driving assistance), and solving the problem of AR glasses failing in low-light environments in the prior art as described in the published patents CN218350626U and CN109143590B; Targeted enhancement processing (such as adaptive histogram equalization and noise suppression) is applied to low-light images to significantly improve the brightness and contrast of dark areas, restoring environmental structure, texture, and contour details (such as road edges, obstacle shapes, and terrain features) that are obscured by darkness. Target segmentation processing (such as semantic segmentation based on deep learning) is performed on infrared thermal images to accurately separate targets with significant thermal features (such as human bodies, vehicles, animals, and heat-generating devices) and generate target identification maps. This highlights the inherent advantage of infrared thermal imagers—"easy to see the environment but easy to spot targets"—while overcoming their disadvantage of "difficulty in identifying environmental details of non-thermal targets." It directly corresponds to and surpasses the limitations of the single night vision mode (low-light or infrared) described in CN115100556B. By combining cross-modal feature matching and fusion, the high-resolution spatial geometric information (shape, position, structure) provided by low-light images and the target thermal attribute information (temperature distribution, target category, motion state) provided by thermal images are organically integrated, which goes beyond simple image superposition and achieves a 1+1>2 effect at the feature level; The output fused feature map contains both detailed environmental background and clear thermal target identification, providing a more complete and semantically rich data foundation for subsequent 3D modeling and AR overlay, which solves the problem of the one-sidedness of information from a single sensor. A 3D environment model is constructed based on fused feature maps, providing a precise spatial reference framework for the overlay of virtual information. Combined with real-time pose data (position and orientation) from the AR glasses, a dynamic pose compensation mechanism corrects for changes in the field of view caused by the user's head movements in real time. This ensures that the generated virtual information layer can be stably and accurately spatially aligned with the real-world scene, avoiding virtual information "drifting" or "misalignment," and maintaining immersion and usability, especially when the user is moving. Adaptive information overlay... Generate corresponding virtual information layers based on specific task requirements (such as navigation markers, target recognition boxes, and device status information) to optimize human-computer interaction efficiency. By employing an adaptive transparency blending mechanism based on the attributes of hot targets, the transparency of virtual information in areas of critical hot targets (such as high-threat targets, personnel / equipment requiring attention) is automatically reduced to ensure that these important real targets are not obscured by virtual information; while in non-critical areas or background environments, virtual information can maintain high transparency or be completely opaque to clearly convey information. This intelligent presentation of information layers greatly optimizes the efficiency of information acquisition and situational awareness for users in complex dark environments, preventing critical information from being obscured. The intelligently blended image (which integrates an optimized view of the real environment, key thermal target markers, and mission-related virtual information) is directly output to the user's field of view through an optical projection system. The user ultimately obtains a seamlessly blended, information-rich, key target-highlighted, and spatially accurate augmented reality view. This not only enables the user to "see" in the dark, but also to "see clearly" and "understand," and to interact effectively with the virtual information, significantly improving operational efficiency, safety, and decision-making quality in low-light / no-light environments. This invention achieves high-precision scene perception in low-light / no-light environments by fusing dual-channel data from low-light imaging and infrared thermal imaging, combined with cross-modal feature matching and dynamic pose compensation. Low-light image enhancement technology restores the detailed structure of the environment, and thermal imaging target segmentation accurately identifies thermal feature targets such as organisms. Heterogeneous feature fusion constructs a three-dimensional environment model that combines spatial geometric information and target attributes. Then, through an adaptive mixing mechanism of virtual information layer driven by thermal target attributes, the display transparency of key targets is dynamically adjusted in the AR field of view, ultimately outputting an enhanced view that blends virtual and real elements. This breaks through the lighting limitations of traditional AR glasses and simultaneously solves the defects of single night vision devices that "see the environment but have difficulty recognizing targets" or "recognize targets but have difficulty distinguishing the environment." It provides users with an immersive augmented reality experience in dark environments that combines panoramic detail perception and accurate thermal target positioning.

[0022] In this embodiment of the invention, step S100 includes: S110: The smart glasses are provided with a first imaging channel and a second imaging channel. The first imaging channel is equipped with a low-light sensor, and the second imaging channel is equipped with an infrared sensor. S120: Acquire an ambient visible light image through a low-light sensor in the first imaging channel, the low-light sensor being configured to respond under illumination conditions below the human eye's visibility threshold; S130: Acquire thermal radiation images through an infrared sensor in the second imaging channel, the infrared sensor being configured to sense the temperature radiation characteristics of a living organism or mechanical device; S140: Dynamically triggers dual-channel acquisition mode based on ambient light intensity. The infrared sensor is disabled when the ambient light intensity is higher than a set threshold. When the ambient light intensity is lower than the set threshold, dual-channel acquisition is started simultaneously; S150: Performs spatiotemporal alignment on the acquired dual-modal images, and compensates for parallax shift caused by differences in optical paths through feature point matching; Through hardware architecture innovation and intelligent acquisition strategies, the system's environmental adaptability and energy efficiency have been significantly improved. The physical isolation design of the dual independent imaging channels (low-light sensor + infrared sensor) avoids spectral crosstalk, ensuring that the low-light sensor can still capture environmental details in low-light environments below the human eye's visibility threshold, while the infrared sensor specializes in the temperature radiation characteristics of biological / mechanical devices. Its core value lies in the ambient light intensity adaptive dual-modal triggering mechanism—when the ambient light is sufficient, only the low-light channel is activated to reduce power consumption, while in dark environments, the dual channels are automatically activated for synchronous acquisition, which expands the full-time domain operation capability and optimizes the system's endurance. Furthermore, the spatiotemporal alignment algorithm eliminates parallax shift caused by differences in optical paths, laying a precise data foundation for subsequent cross-modal fusion, and ultimately achieving collaborative optimization from the hardware layer to the data layer, achieving an integrated perception effect of "on-demand start-up, precise synchronization, and high energy efficiency".

[0023] In this embodiment of the invention, step S200 includes: S210: Perform multi-scale detail enhancement processing on the visible light image to improve the recognizability of dark area texture features while suppressing image noise; S220: Perform adaptive threshold segmentation on the thermal radiation image to identify target areas with significant thermal radiation characteristics based on temperature gradient distribution; S230: Establish a processing priority mapping between visible light images and thermal radiation images, where: When the ambient illuminance is below the first threshold, priority is given to ensuring the real-time generation of thermal target identification maps; When a moving target is detected, the dynamic range compression efficiency of the visible light image is improved first. S240: Perform pixel-level association registration between the processed visible light image and the thermal target identification map to generate an optimized visible light image and thermal target identification map with spatial correspondence; Through intelligent image processing strategies and dynamic resource scheduling mechanisms, the system significantly improves perception accuracy and response efficiency in complex night vision scenarios. First, multi-scale detail enhancement processing suppresses noise in visible light images while strengthening texture features in dark areas, solving the problem of blurred environmental structures in low-light environments. Second, adaptive threshold segmentation based on temperature gradient distribution accurately extracts thermal target regions, overcoming the false detection problem of traditional fixed threshold methods in complex thermal scenarios. A dynamic mapping mechanism for processing priorities is introduced—when the ambient illumination is extremely low, priority is given to ensuring the real-time performance of thermal target recognition to meet the rapid response requirements of security / military scenarios, while when a moving target is detected, the dynamic range compression efficiency of the visible light image is automatically improved to capture key motion details. Finally, pixel-level association registration ensures the spatial consistency of dual-modal images, building an accurate correspondence for subsequent cross-modal fusion. This solution achieves collaborative optimization from low-level image processing to high-level decision-making logic, dynamically balancing the dual requirements of "environmental detail restoration" and "thermal target recognition" under energy efficiency constraints, and greatly enhancing the system's adaptive perception capability in dark environments.

[0024] In this embodiment of the invention, step S300 includes: S310: Extract a set of spatial geometric features from the optimized visible light image, wherein the spatial geometric features include edge structure descriptors and texture distribution feature vectors; S320: Extract a set of thermal attribute features from the thermal target identification map, wherein the thermal attribute features include a temperature gradient vector and a semantic description of hot spot contours; S330: Construct a mapping model between the visible light feature space and the infrared feature space, and achieve feature-level association matching through cross-modal similarity measurement; S340: Generate a fused feature map based on the matching results, where: Spatial geometric features constitute the environmental structure layer, characterizing the physical contours and surface properties of the scene; Thermal attribute features constitute the target information layer, marking the location and attributes of targets with significant thermal radiation characteristics; The two layers of features are superimposed at the pixel level through a spatial transformation matrix; By hierarchically decoupling and precisely associating heterogeneous features, a cross-modal fusion architecture with strong interpretability and high spatial consistency is constructed. An environmental structure layer is formed by extracting edge structure descriptors and texture distribution features from visible light images, fully preserving the geometric contours and surface characteristics of the physical world. Simultaneously, temperature gradient vectors and hot spot contour semantics from thermal images are extracted to construct a target information layer, accurately labeling the location and attributes of thermal targets such as organisms and mechanical equipment. Furthermore, an innovative cross-modal mapping relationship model is introduced, utilizing a similarity measurement algorithm to achieve intelligent matching of heterogeneous feature spaces (geometric space and thermal attribute space), effectively overcoming the spatial misalignment problem caused by modal differences in traditional image fusion. Finally, pixel-level hierarchical overlay is achieved through a spatial transformation matrix, enabling precise registration of the environmental structure layer and the target information layer in a unified coordinate system, generating a fusion feature map that combines clear environmental topology with salient target identification. This scheme not only provides high-precision spatial anchors for AR virtual information overlay but also significantly improves the system's semantic understanding of complex scenes through feature decoupling, laying a structured data foundation for subsequent 3D reconstruction and virtual-real interaction.

[0025] In this embodiment of the invention, step S400 includes: S410: Generate a sparse feature point cloud based on the fused feature map, and calculate the pose of the AR glasses through visual odometry. The AR glasses are equipped with an extrinsic parameter matrix composed of a camera and an inertial measurement unit. S420: Integrates angular velocity and acceleration data from the inertial measurement unit to construct a six-degree-of-freedom motion prediction model; S430: Perform online calibration of multi-source sensors. When the difference between the motion prediction model of the inertial measurement unit and the pose of the AR glasses calculated by the visual odometry exceeds a preset threshold, the camera-IMU extrinsic parameter matrix is ​​optimized based on the reprojection error of the feature points in the environmental structure layer. When environmental texture is missing, stable thermal features in the thermal target information layer are used as an auxiliary localization reference. S440: Through a sliding window for human-computer interaction, it jointly optimizes the 3D environment model and real-time pose data, and outputs the registered augmented reality spatial coordinate system; By using multi-source sensing fusion and dynamic calibration technology, the spatial positioning accuracy and stability of AR glasses in dark environments are significantly improved. The sparse feature point cloud generated based on the fused feature map combined with visual odometry provides the initial pose, effectively overcoming the reliability defects of a single sensor in low-light environments. The six-degree-of-freedom motion prediction model that integrates IMU (Inertial Measurement Unit) compensates for pose drift caused by high-speed motion in real time. Its core innovation lies in the intelligent online calibration mechanism of multi-source sensors. When the pose difference exceeds the threshold, the camera-IMU extrinsic parameter matrix is ​​optimized in real time based on the reprojection error of the feature points of the environmental structure layer (step S430). It automatically compensates for the spatial offset caused by equipment deformation or assembly error, eliminates the cumulative error of inertial navigation, and ensures that the virtual information is aligned with the physical world. When low light conditions lead to a scarcity of visible light features, stable thermal features (such as the outline of high-temperature equipment and constant-temperature biological hot spots) in the thermal target information layer are innovatively used as auxiliary localization references (step S430), breaking through the bottleneck of traditional visual SLAM failure in dark / low-texture scenes. By using a sliding window nonlinear optimization algorithm (step S440), the three-dimensional environment model and real-time pose data are jointly optimized under limited computing resources—only the constraint relationship of the most recent N key frames is retained, which avoids global optimization computing power explosion and uses IMU motion prior and multimodal feature observation to build a tightly coupled optimization model, and outputs a high-confidence augmented reality spatial coordinate system in real time. The collaboration of multiple technologies forms a closed loop of "dynamic hardware error correction + online calibration of multi-source sensors + precise solution of resource constraints", enabling AR glasses to maintain centimeter-level positioning accuracy and millisecond-level response speed in dynamic dark environments, providing reliable spatiotemporal consistency for key scenarios such as military reconnaissance and night rescue. Finally, by using sliding window nonlinear optimization to jointly optimize the environment model and pose data, an AR coordinate system with extremely high spatial consistency is output, ensuring that virtual information maintains millimeter-level accurate registration with the real world in dynamic dark scenes, laying a highly robust spatial foundation for virtual-real interaction.

[0026] In this embodiment of the invention, step S500 includes: S510: Generate a virtual information layer according to the task mode selected by the user, wherein the task mode includes at least industrial inspection mode and security monitoring mode; S520: Establish rules for mapping thermal target attributes to transparency. In industrial inspection mode, transparency is positively correlated with target temperature; In security monitoring mode, transparency is negatively correlated with the intensity of target movement; S530: Perform dynamic range compression processing on the optimized visible light image to generate a basal layer image; S540: The virtual information layer is superimposed on the base layer image, and the pixel-level transparency coefficient α is calculated in real time based on the mapping rule, where: α=λ(P,M), and the transparency coefficient is jointly determined by the target attribute vector P and the task mode parameter M; S550: Eliminates the flickering effect of virtual-real fusion caused by image jitter through time-domain filtering; Through a task-driven intelligent information layering and dynamic fusion mechanism, the information transmission efficiency and user experience of AR glasses in professional scenarios are significantly improved. Customized virtual information layers are generated based on user-selected task modes (such as industrial inspection / security monitoring) to ensure accurate matching of key information with scenario requirements. An innovative dynamic mapping rule between thermal target attributes and transparency is established—in industrial mode, the transparency of high-temperature targets increases with temperature to avoid virtual information obscuring high-risk heat sources, while in security mode, the transparency of moving targets decreases with movement intensity to ensure rapid locking of threat targets. Furthermore, the visible light substrate image is optimized through dynamic range compression processing to solve the overexposure / underexposure problem in low-light scenes. Finally, the pixel-level transparency coefficient α=λ(P,M) is used to achieve adaptive mixing of virtual information and real scenes (P is the thermal attribute vector, M is the task parameter), and the flickering effect caused by image jitter is eliminated through temporal filtering. Stable, clear, and semantically focused augmented reality views are output in complex night vision environments to meet the differentiated needs of industrial fault diagnosis and security situational awareness.

[0027] In this embodiment of the invention, step S600 includes: S610: Decomposes the mixed image into RGB three-channel light signals and generates the initial light field through the micro-display unit; S620: Dynamically adjusts light field brightness based on ambient light intensity. When the ambient light intensity is lower than the human eye's dark adaptation threshold, the low blue light spectrum output mode is activated. When the ambient light intensity is higher than the glare threshold, local area brightness suppression is activated. S630: Performs the following modulation operation on the optical field through a diffraction waveguide structure: a) Expand the exit pupil range to cover the user's natural eye movement area; b) Compensate for optical distortion caused by field of view shift; S640: An augmented reality view matching the real-world field of view is formed on the waveguide output surface, wherein the virtual information layer is superimposed on the real scene in a retinal projection manner; First, the mixed image is decomposed into RGB channels and an initial light field is generated through a micro-display unit, laying a high-quality signal foundation for optical projection. When the ambient light is below the human eye's dark adaptation threshold, a low blue light spectrum output mode is automatically activated, effectively protecting the user's dark vision and avoiding glare interference in night vision imaging. In strong light environments, local brightness suppression is intelligently activated to prevent visual overload caused by high-brightness virtual information. Furthermore, through the dual modulation of the diffraction waveguide structure, on the one hand, the exit pupil range is expanded to cover the natural eye movement area (horizontal ±30°), eliminating the "pupil swimming" effect of traditional AR glasses. On the other hand, off-axis distortion is compensated in real time to ensure the geometric fidelity of the image in the edge area of ​​the large viewing angle. Finally, an enhanced view that accurately matches the space of the real world is formed on the waveguide output surface, and the virtual information is seamlessly superimposed by retinal projection, so that users can always obtain an immersive interactive experience with low fatigue, no glare, and full field of view coverage in complex scenarios such as day and night transitions and sudden changes in brightness.

[0028] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An infrared night vision AR smart glasses based on multispectral fusion, characterized in that, Includes the following steps: S100: Acquires ambient visible light images through a low-light imaging unit and simultaneously acquires thermal radiation images through an infrared thermal imaging unit; S200: Enhance the visible light image to improve details in dark areas, and perform target segmentation on the thermal radiation image to identify thermal target features, generating an optimized visible light image and a thermal target identification map respectively. S300: Extract spatial geometric features from the optimized visible light image, extract thermal attribute features from the thermal target identification map, and achieve heterogeneous feature fusion through cross-modal feature matching to generate a fused feature map containing environmental structure and target information; S400: Construct a three-dimensional environment model based on the fused feature map, and combine it with the real-time pose data of the AR glasses to spatially register the environment model with the user's field of view through a dynamic pose compensation mechanism; S500: Generate a virtual information layer according to task requirements and perform adaptive transparency blending with the optimized visible light image, wherein the blended transparency is dynamically adjusted based on thermal target properties; S600: The mixed image is output to the user's field of view through an optical projection system to form an augmented reality view with superimposed virtual information.

2. The infrared night vision AR smart glasses based on multispectral fusion according to claim 1, characterized in that, Step S100 includes: S110: The smart glasses are provided with a first imaging channel and a second imaging channel. The first imaging channel is equipped with a low-light sensor, and the second imaging channel is equipped with an infrared sensor. S120: Acquire an ambient visible light image through a low-light sensor in the first imaging channel, the low-light sensor being configured to respond under illumination conditions below the human eye's visibility threshold; S130: Acquire thermal radiation images through an infrared sensor in the second imaging channel, the infrared sensor being configured to sense the temperature radiation characteristics of a living organism or mechanical device; S140: Dynamically triggers dual-channel acquisition mode based on ambient light intensity. The infrared sensor is disabled when the ambient light intensity is higher than a set threshold. When the ambient light intensity is lower than the set threshold, dual-channel acquisition is started simultaneously; S150: Performs spatiotemporal alignment on the acquired dual-modal images, compensating for parallax shift caused by differences in optical paths through feature point matching.

3. The infrared night vision AR smart glasses based on multispectral fusion according to claim 2, characterized in that, Step S200 includes: S210: Perform multi-scale detail enhancement processing on the visible light image to improve the recognizability of dark area texture features while suppressing image noise; S220: Perform adaptive threshold segmentation on the thermal radiation image to identify target areas with significant thermal radiation characteristics based on temperature gradient distribution; S230: Establish a processing priority mapping between visible light images and thermal radiation images, where: When the ambient illuminance is below the first threshold, priority is given to ensuring the real-time generation of thermal target identification maps; When a moving target is detected, the dynamic range compression efficiency of the visible light image is improved first. S240: Perform pixel-level association registration between the processed visible light image and the thermal target identification map to generate an optimized visible light image and thermal target identification map with spatial correspondence.

4. The infrared night vision AR smart glasses based on multispectral fusion according to claim 3, characterized in that, Step S300 includes: S310: Extract a set of spatial geometric features from the optimized visible light image, wherein the spatial geometric features include edge structure descriptors and texture distribution feature vectors; S320: Extract a set of thermal attribute features from the thermal target identification map, wherein the thermal attribute features include a temperature gradient vector and a semantic description of hot spot contours; S330: Construct a mapping model between the visible light feature space and the infrared feature space, and achieve feature-level association matching through cross-modal similarity measurement; S340: Generate a fused feature map based on the matching results, where: Spatial geometric features constitute the environmental structure layer, characterizing the physical contours and surface properties of the scene; Thermal attribute features constitute the target information layer, marking the location and attributes of targets with significant thermal radiation characteristics; The two layers of features are superimposed at the pixel level through a spatial transformation matrix.

5. The infrared night vision AR smart glasses based on multispectral fusion according to claim 4, characterized in that, Step S400 includes: S410: Generate a sparse feature point cloud based on the fused feature map, and calculate the pose of the AR glasses through visual odometry. The AR glasses are equipped with an extrinsic parameter matrix composed of a camera and an inertial measurement unit. S420: Integrates angular velocity and acceleration data from the inertial measurement unit to construct a six-degree-of-freedom motion prediction model; S430: Perform online calibration of multi-source sensors. When the difference between the motion prediction model of the inertial measurement unit and the pose of the AR glasses calculated by the visual odometry exceeds a preset threshold, the camera-IMU extrinsic parameter matrix is ​​optimized based on the reprojection error of the feature points in the environmental structure layer. When environmental texture is missing, stable thermal features in the thermal target information layer are used as an auxiliary localization reference. S440: Through a sliding window for human-computer interaction, it jointly optimizes the 3D environment model and real-time pose data, and outputs the registered augmented reality spatial coordinate system.

6. The infrared night vision AR smart glasses based on multispectral fusion according to claim 5, characterized in that, Step S500 includes: S510: Generate a virtual information layer according to the task mode selected by the user, wherein the task mode includes at least industrial inspection mode and security monitoring mode; S520: Establish rules for mapping thermal target attributes to transparency. In industrial inspection mode, transparency is positively correlated with target temperature; In security monitoring mode, transparency is negatively correlated with the intensity of target movement; S530: Perform dynamic range compression processing on the optimized visible light image to generate a basal layer image; S540: The virtual information layer is superimposed on the base layer image, and the pixel-level transparency coefficient α is calculated in real time based on the mapping rule, where: α=λ(P,M), and the transparency coefficient is jointly determined by the target attribute vector P and the task mode parameter M; S550: Eliminates the flickering effect of virtual-real fusion caused by image jitter through time-domain filtering.

7. The infrared night vision AR smart glasses based on multispectral fusion according to claim 6, characterized in that, Step S600 includes: S610: Decomposes the mixed image into RGB three-channel light signals and generates the initial light field through the micro-display unit; S620: Dynamically adjusts light field brightness based on ambient light intensity. When the ambient light intensity is lower than the human eye's dark adaptation threshold, the low blue light spectrum output mode is activated. When the ambient light intensity is higher than the glare threshold, local area brightness suppression is activated. S630: Performs the following modulation operation on the optical field through a diffraction waveguide structure: a) Expand the exit pupil range to cover the user's natural eye movement area; b) Compensate for optical distortion caused by field of view shift; S640: An augmented reality view matching the real-world field of view is formed on the waveguide output surface, wherein the virtual information layer is superimposed on the real scene in a retinal projection manner.

Images