Active infrared vision fusion VR positioning method in under-illumination environment

Abstract

The invention relates to an active infrared vision fusion VR positioning method in an under-illumination environment, and the method comprises the steps: configuring an active infrared illumination unit on a head-mounted device, controlling the active infrared illumination unit to emit infrared light to the environment in a preset illumination mode, and collecting an infrared image; synchronously acquiring measurement data of the inertial measurement unit; performing preprocessing and feature extraction and tracking on the infrared image to obtain visual features; carrying out pre-integration on the measurement data to obtain IMU pre-integration data; visual features and IMU pre-integration data are fused, and state estimation and map construction are carried out through a multi-sensor tight coupling optimization algorithm; outputting a six-degree-of-freedom pose; and based on a state estimation result or visual feature extraction quality, dynamically adjusting illumination parameters, and repeating the steps. The fundamental problem that a traditional visual scheme fails in the dark is thoroughly solved, stable and continuous six-degree-of-freedom pose output is achieved, meanwhile, the optimal balance of power consumption and performance is achieved, mutual interference among multiple devices is effectively restrained, and the precision and robustness of the system are greatly improved.

Description

Technical Field

[0001] This invention relates to the technical field of general image data processing or generation, and in particular to an active infrared visual fusion VR positioning method in low-light environments. Background Technology

[0002] In virtual reality (VR) applications, accurate and stable six degrees of freedom (6DoF) positioning is the technological cornerstone for ensuring an immersive experience. However, in dark or extremely low-light environments, mainstream positioning technologies face fundamental challenges.

[0003] Traditional visual positioning solutions, such as visible light-based RGB cameras or various visual SLAM systems, rely on ambient light illumination. In the absence of light, these systems cannot acquire effective image information, causing feature extraction and tracking functions to fail completely. Even with high-sensitivity sensors, the image signal-to-noise ratio is far from sufficient to support reliable matching and pose calculation in completely dark environments.

[0004] To address dark environments, some existing technologies employ infrared bands. However, these solutions are mostly based on passive principles, such as relying on a combination of pre-placed infrared reflective markers and infrared cameras. This approach has several inherent drawbacks: First, system performance heavily depends on the intensity of infrared light reflected by the markers from the environment, resulting in extremely weak signals in dark and open scenes. Second, it requires a large-scale, high-density network of external infrared light sources or high-reflectivity markers, making deployment complex and costly. Third, passive markers are easily blocked by the user or others, causing momentary loss of positioning. Fourth, in multi-user concurrent scenarios, light sources from different devices or environments are prone to interference, making independent and accurate positioning difficult.

[0005] In addition, non-visual single-sensor solutions have also been considered, but these also have insurmountable limitations. Pure inertial navigation systems are unaffected by lighting conditions, but their inherent integral drift prevents them from operating independently for extended periods. Active depth cameras, such as structured light or Time-of-Flight (ToF) cameras, have significantly reduced effective measurement ranges in darkness and suffer from high power consumption and potential interference between multiple devices. Other technologies, such as ultrasonic sensors, generally suffer from limited accuracy, high latency, or susceptibility to environmental factors.

[0006] Existing sensor fusion attempts, such as vision-inertial fusion schemes, also encounter bottlenecks in dark environments. This is because when visual information is continuously missing or of extremely low quality, the core constraints of the fusion algorithm fail, and its performance drops sharply or even becomes completely inoperable. Simple sensor switching or degradation strategies are insufficient to provide continuous, stable, and high-precision pose output during periods of visual failure.

[0007] Therefore, there is an urgent need in this field for an active positioning system that can be fundamentally adapted to dark environments. It should be able to autonomously generate reliable sensing information under conditions of no ambient light, thereby providing continuous, high-precision, low-latency six-degree-of-freedom positioning, while simultaneously meeting the stringent requirements of system robustness, practicality, and multi-device compatibility. Summary of the Invention

[0008] This invention solves the problems existing in the prior art and provides an active infrared visual fusion VR positioning method in low-light environments, and constructs a system framework for deep fusion of active lighting, visual perception and inertial navigation.

[0009] The technical solution adopted in this invention is an active infrared visual fusion VR positioning method for low-light environments, which involves configuring an active infrared illumination unit on a head-mounted device. The method includes the following steps:

[0010] S1 controls the active infrared illumination unit to emit infrared light into the environment in a preset illumination mode and acquire infrared images; it also simultaneously acquires measurement data from the inertial measurement unit.

[0011] S2 performs preprocessing, feature extraction, and tracking on the infrared image to obtain visual features; it also performs pre-integration on the measurement data to obtain IMU pre-integrated data.

[0012] S3 integrates visual features with IMU pre-integrated data, and performs state estimation and map construction through a multi-sensor tightly coupled optimization algorithm; outputting a six-DOF pose.

[0013] S4 Based on the state estimation result or visual feature extraction quality, dynamically adjust the illumination parameters of the active infrared illumination unit, repeat S1, and enter the next cycle.

[0014] Preferably, in S1, the preset lighting mode includes:

[0015] Uniform illumination mode is used for initialization and stable tracking;

[0016] Encoded pulse lighting mode for multi-device anti-interference scenarios;

[0017] Structured pattern lighting mode, used for areas with fewer features than preset.

[0018] Preferably, when S1 is executed repeatedly, the switching of the lighting mode is as follows:

[0019] If the current feature quality score is greater than the preset high value and the battery level is greater than the low battery threshold, then switch to uniform lighting mode in the next cycle.

[0020] If the number of nearby devices detected is greater than 1 and the time since the last mode switch is greater than the cooling time, then the next cycle will switch to the coded pulse lighting mode.

[0021] If the current feature quality score is less than the preset low value and the current mode holding time is greater than the preset time, then the next cycle will switch to the structured pattern lighting mode.

[0022] Otherwise, maintain the current lighting mode in the next cycle.

[0023] Preferably, an optimization problem is established to dynamically adjust the illumination parameters of the active infrared illumination unit in the next cycle; the optimization problem is related to the feature extraction quality evaluation function, power consumption penalty term, and multi-device interference metric.

[0024] The optimization problem is solved using the gradient descent method.

[0025] Preferably, during the initialization or calibration phase of the active infrared illumination unit, spot uniformity optimization is performed by independently controlling the driving current of the LED array in the active infrared illumination unit using a NURBS model to generate a uniform infrared illumination field.

[0026] Preferably, the driving current of the active infrared illumination unit is compensated according to its temperature T, and the compensation is performed after the illumination parameters are adjusted or the light spot uniformity is optimized.

[0027] Preferably, when the active infrared illumination unit emits infrared light, it performs safety constraints, including eye safety constraints and overheat protection constraints, wherein the eye safety constraints are used to control the infrared radiation power emitted by the active infrared illumination unit.

[0028] Preferably, the helmet used in conjunction with the active infrared illumination unit is equipped with a grayscale visual perception unit, including a grayscale camera with an infrared cutoff filter removed.

[0029] Preferably, infrared reflection enhancement markers are arranged in the environment in conjunction with the active infrared illumination unit.

[0030] Preferably, a comprehensive performance evaluation function for the lighting system is established to quantify and optimize the overall performance of the active infrared lighting unit.

[0031] This invention relates to an active infrared visual fusion VR positioning method for low-light environments. An active infrared illumination unit is configured on a head-mounted device. The active infrared illumination unit is controlled to emit infrared light into the environment in a preset illumination mode to acquire infrared images. Simultaneously, measurement data from an inertial measurement unit (IMU) is acquired. The infrared images are preprocessed and feature extracted and tracked to obtain visual features. The measurement data is pre-integrated to obtain IMU pre-integrated data. The visual features and IMU pre-integrated data are fused, and a state estimation and map construction are performed using a multi-sensor tightly coupled optimization algorithm. A six-degree-of-freedom pose is output. Based on the state estimation result or the quality of the visual feature extraction, the illumination parameters of the active infrared illumination unit are dynamically adjusted, and the next cycle is repeated.

[0032] The beneficial effects of this invention are as follows:

[0033] (1) By integrating an active infrared illumination unit and a visual perception unit optimized for the infrared band, it can autonomously generate high-contrast scene images in low-light environments or even in the absence of ambient light, thus completely solving the fundamental problem of traditional visual solutions failing in the dark and achieving stable and continuous six-degree-of-freedom pose output.

[0034] (2) It can dynamically adjust the lighting mode and parameters according to the real-time status such as feature abundance, device power, and proximity interference, which not only ensures the optimal acquisition of positioning features, but also achieves the best balance between power consumption and performance, and effectively suppresses mutual interference between multiple devices.

[0035] (3) Construct a tightly coupled optimization framework based on inertial measurement unit data as the basis for high-frequency motion prediction and active infrared visual features as the absolute observation constraint. It can still provide sufficient geometric constraints in low-light environments, effectively suppress the integral drift of pure inertial navigation, and provide smooth state estimation when visual features are temporarily scarce, thus greatly improving the accuracy and robustness of the system.

[0036] (4) From optimizing the uniformity of the illumination field and compensating for the temperature drift of key components to strict eye safety and overheat protection mechanisms, not only is the positioning accuracy and feature detection rate improved, but also the long-term operational reliability, energy efficiency and user safety of the equipment are ensured, meeting the practical requirements of consumer products.

[0037] (5) Supports scalable collaborative working modes. The diversity of lighting modes lays the technical foundation for realizing multi-user, large-scale dark environment VR applications. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0040] This invention relates to an active infrared visual fusion VR positioning method for low-light environments. In practical applications, in addition to configuring an active infrared illumination unit on the head-mounted device, a series of hardware upgrades and modifications are required, including:

[0041] Infrared LED arrays with wavelengths of 850nm or 940nm are integrated into the head-mounted device, supporting multi-mode programmable lighting;

[0042] The helmet, which works in conjunction with the active infrared illumination unit, is equipped with a grayscale visual perception unit, including a grayscale camera with the infrared cutoff filter removed. It is a high-sensitivity global shutter grayscale camera, which is optimized for the infrared band and used to acquire infrared images.

[0043] An inertial measurement unit, typically a nine-axis IMU, is integrated into a head-mounted device, including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer.

[0044] The configuration processing control unit, typically an embedded processor, enables real-time data fusion and lighting control.

[0045] To better implement the method, infrared reflection enhancement markers are arranged in the environment in conjunction with the active infrared illumination unit, which can be used to enhance feature richness.

[0046] The method includes the following steps:

[0047] S1 controls the active infrared illumination unit to emit infrared light into the environment in a preset illumination mode and acquire infrared images; it also simultaneously acquires measurement data from the inertial measurement unit.

[0048] S2 performs preprocessing, feature extraction, and tracking on the infrared image to obtain visual features; it also performs pre-integration on the measurement data to obtain IMU pre-integrated data.

[0049] S3 integrates visual features with IMU pre-integrated data, and performs state estimation and map construction through a multi-sensor tightly coupled optimization algorithm; outputting a six-DOF pose.

[0050] S4 Based on the state estimation result or visual feature extraction quality, dynamically adjust the illumination parameters of the active infrared illumination unit, repeat S1, and enter the next cycle.

[0051] The steps are explained below with reference to specific implementation methods.

[0052] S1 controls the active infrared illumination unit to emit infrared light into the environment in a preset illumination mode and acquire infrared images; it also simultaneously acquires measurement data from the inertial measurement unit.

[0053] The preset lighting modes include:

[0054] Uniform illumination mode, used for initialization and stable tracking, has the following illumination intensity distribution at coordinate point (x,y):

[0055]

[0056] in, As the base strength, and For the width and height of the lighting area; in fact, It can also be expressed as However, in uniform lighting mode, t is a continuous time.

[0057] Encoded pulse lighting mode, used in multi-device interference-resistant scenarios, where the lighting intensity varies over time to meet certain requirements.

[0058]

[0059] in, It is a pseudo-random encoded sequence. For pulse period, Where N is the pulse width, N is the length of the encoding period, and n is its index;

[0060] Structured pattern lighting patterns are used in areas with fewer than preset features, where the spatial distribution of lighting intensity satisfies certain conditions.

[0061]

[0062] in, For spatial frequency, For phase parameters, The modulation depth.

[0063] S2 performs preprocessing, feature extraction, and tracking on the infrared image to obtain visual features; it also performs pre-integration on the measurement data to obtain IMU pre-integrated data.

[0064] S3 integrates visual features with IMU pre-integrated data, and performs state estimation and map construction through a multi-sensor tightly coupled optimization algorithm; outputting a six-DOF pose.

[0065] The multi-sensor tight coupling optimization algorithm here is an optimization algorithm based on visual inertial odometry, which is implemented by fusing visual reprojection error and IMU pre-integration constraints.

[0066] S4 Based on the state estimation result or the visual feature extraction quality, dynamically adjust the illumination parameters of the active infrared illumination unit, repeat S1, and enter the next cycle.

[0067] When S1 is executed repeatedly, the lighting mode is switched as follows:

[0068] If the current feature quality score is greater than the preset high value and the battery level is greater than the low battery threshold, then the system will switch to uniform lighting mode in the next cycle. ;

[0069] If the number of nearby devices detected is greater than 1 and the time since the last mode switch is greater than the cooldown time, then the next cycle will switch to the coded pulse illumination mode. ;

[0070] If the current feature quality score is lower than the preset low value and the current mode hold time is longer than the preset time, then the next cycle will switch to the structured patterned illumination mode. ;

[0071] Otherwise, maintain the current lighting mode in the next cycle. ;

[0072] It is expressed as follows:

[0073]

[0074] in, For feature quality scoring, and Preset high and low values ​​for quality. For low battery threshold, Cooling time, Minimum mode hold time (preset time).

[0075] An optimization problem is established to dynamically adjust the illumination parameters of the active infrared illumination unit in the next cycle; the optimization problem is related to the feature extraction quality evaluation function. Power consumption penalty item Multi-device interference measurement Related;

[0076] Specifically,

[0077]

[0078]

[0079]

[0080]

[0081] in, and The adjustable upper and lower limits are for the lighting intensity I or the driving current to achieve it. The rate of change of lighting intensity;

[0082] Current lighting intensity The number of feature points detected below For the maximum expected number of features, For image intensity variance, These are the weighting coefficients; Rated operating current, For scale parameters; For the first Gain of each interference source For distance, For frequency difference, Interference threshold ;

[0083] It is important to note that 'I' uniformly represents the intensity component in the independently controllable lighting parameter vector. In actual hardware implementation, this intensity component is achieved by controlling the driving current of the infrared LED array. Therefore, in the context of power calculations, such as... In this context, the physical meaning of I focuses on the driving current value, while in the context of optical sensing and interference, such as , In this context, the physical meaning of I focuses on the intensity of emitted light, and the two are related through the electro-optical conversion characteristics of LEDs;

[0084] α, β, and γ are weighting coefficients that are dynamically adjusted based on the current system state. Their adjustment is adaptive based on the current system state, and the rules are as follows:

[0085]

[0086]

[0087]

[0088] in, The threshold for the number of features. This is the current battery level. Fully charged. The number of nearby devices detected. Maximum number of devices, , , Let α, β, and γ be the initial values;

[0089] The optimization problem is solved using the gradient descent method.

[0090] Specifically, the lighting parameter vector for the next cycle Among them, the objective function of the optimization problem is... gradient satisfy,

[0091]

[0092] θ=[I, f x , f y , ϕ x , ϕ y ] ⊤ This is a vector of lighting parameters, containing vectors of intensity, spatial frequency, and phase parameters. For an adaptive learning rate, satisfying the following:

[0093]

[0094] The initial learning rate, The decay time constant, For the objective function In the current parameters The gradient vector at that point, For the objective function In initial parameters The gradient vector at that point.

[0095] Furthermore, during the initialization or calibration phase of the active infrared illumination unit, spot uniformity optimization is performed by independently controlling the driving current of the LED array in the active infrared illumination unit using a NURBS model to generate a uniform infrared illumination field.

[0096] The driving current of the active infrared illumination unit is compensated based on its temperature T. The compensation is performed after the illumination parameters are adjusted or the light spot uniformity is optimized.

[0097] Specifically,

[0098]

[0099] in, To control the point current value, As weight, and For B-spline basis functions, For parameter domain coordinates, and for The maximum index value of the control points in the direction is satisfied by the temperature compensation function.

[0100]

[0101] in, The current temperature. For reference temperature, and This is the temperature coefficient.

[0102] When the active infrared illumination unit emits infrared light, it executes safety constraints, including eye safety constraints and overheat protection constraints. The eye safety constraints are used to control the infrared radiation power emitted by the active infrared illumination unit. .

[0103] Specifically,

[0104]

[0105] Where MPE is the maximum permissible exposure. The pupil area, For the exposure time, The number of pulses;

[0106] Overheat protection constraints refer to the situation where the temperature exceeds a safe threshold. When the temperature exceeds the maximum value, the power will be automatically reduced. Close directly when satisfied.

[0107]

[0108] A comprehensive performance evaluation function for the lighting system is established to quantify and optimize the overall performance of the active infrared lighting unit. This is reflected in the following:

[0109]

[0110] in, For uniform lighting, For ideal lighting uniformity, For photoelectric conversion efficiency, This represents the theoretical maximum photoelectric conversion efficiency. For feature detection rate, For target feature detection rate, In response to the rise time, For the required response rise time, , , , For the weighting coefficients, satisfying ;

[0111] The FOM value obtained here can be used to quantify the lighting effect.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0117] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for active infrared visual fusion VR positioning in low-light environments, characterized in that: The method of configuring an active infrared illumination unit on a head-mounted device includes the following steps: S1 controls the active infrared illumination unit to emit infrared light into the environment in a preset illumination mode and acquire infrared images; it also simultaneously acquires measurement data from the inertial measurement unit. S2 performs preprocessing, feature extraction, and tracking on the infrared image to obtain visual features; it also performs pre-integration on the measurement data to obtain IMU pre-integrated data. S3 integrates visual features with IMU pre-integrated data, and performs state estimation and map construction through a multi-sensor tightly coupled optimization algorithm; outputting a six-DOF pose. S4 Based on the state estimation result or visual feature extraction quality, dynamically adjust the illumination parameters of the active infrared illumination unit, repeat S1, and enter the next cycle.

2. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: In S1, the preset lighting modes include: Uniform illumination mode is used for initialization and stable tracking; Encoded pulse lighting mode for multi-device anti-interference scenarios; Structured pattern lighting mode, used for areas with fewer features than preset.

3. The active infrared visual fusion VR positioning method under low-light conditions according to claim 2, characterized in that: When S1 is executed repeatedly, the lighting mode is switched as follows: If the current feature quality score is greater than the preset high value and the battery level is greater than the low battery threshold, then switch to uniform lighting mode in the next cycle. If the number of nearby devices detected is greater than 1 and the time since the last mode switch is greater than the cooling time, then the next cycle will switch to the coded pulse lighting mode. If the current feature quality score is less than the preset low value and the current mode holding time is greater than the preset time, then the next cycle will switch to the structured pattern lighting mode. Otherwise, maintain the current lighting mode in the next cycle.

4. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: An optimization problem is established to dynamically adjust the illumination parameters of the active infrared illumination unit in the next cycle; the optimization problem is related to the feature extraction quality evaluation function, power consumption penalty term, and multi-device interference metric. The optimization problem is solved using the gradient descent method.

5. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: During the initialization or calibration phase of the active infrared illumination unit, spot uniformity optimization is performed. The driving current of the LED array in the active infrared illumination unit is independently controlled using a NURBS model to generate a uniform infrared illumination field.

6. The active infrared visual fusion VR positioning method in low-light environments according to claim 5, characterized in that: The driving current of the active infrared illumination unit is compensated based on its temperature T. The compensation is performed after the illumination parameters are adjusted or the light spot uniformity is optimized.

7. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: When the active infrared illumination unit emits infrared light, it executes safety constraints, including eye safety constraints and overheat protection constraints. The eye safety constraints are used to control the infrared radiation power emitted by the active infrared illumination unit.

8. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: The helmet, which works in conjunction with the active infrared illumination unit, is equipped with a grayscale visual perception unit, including a grayscale camera with the infrared cutoff filter removed.

9. The active infrared visual fusion VR positioning method in low-light environments according to claim 1, characterized in that: Infrared reflection enhancement markers are arranged in the environment in conjunction with the active infrared illumination unit.

10. The active infrared visual fusion VR positioning method under low-light conditions according to claim 1, characterized in that: A comprehensive performance evaluation function for the lighting system is established to quantify and optimize the overall performance of the active infrared lighting unit.

Images