Infrared thermal imaging fusion and adaptive rendering method and system based on near-eye display

Abstract

The present application relates to a near-eye display-based infrared thermal imaging fusion and adaptive rendering method and system, belonging to the technical field of infrared thermal imaging and near-eye display. Among them, the method comprises: acquiring infrared thermal imaging data and display light path adjustment parameters, generating a dynamic temperature matrix through noise filtering and radiation calibration; traversing the dynamic temperature matrix to lock temperature extreme points and generate motion trajectories, and synchronously generating extreme value cursors and real-time temperature labels; according to the display light path adjustment parameters, calculating the edge indication marker angle and the nonlinear deviation amplitude, and dynamically compensating the projection offset of the extreme value cursor and the real-time temperature label, and superimposing to generate an OSD information area; monitoring the spatial relationship between the motion trajectory and the OSD information area, adjusting the transparency of the OSD information area or performing displacement avoidance according to the change rate of the display light path adjustment parameters, and outputting a fusion rendering picture. The accurate visual alignment of the extreme value cursor and the actual heat source position during the display lens retraction and extension process and the dynamic avoidance of the OSD information area are realized.

Description

Technical Field

[0001] This invention belongs to the field of infrared thermal imaging and near-eye display technology, specifically relating to an infrared thermal imaging fusion and adaptive rendering method and system based on near-eye display. Background Technology

[0002] Infrared thermal imaging technology can convert the infrared radiation of a target object into a visualized temperature distribution image. It has advantages such as all-weather operation, strong anti-interference ability, and good detection of concealed targets. It has wide applications in military reconnaissance, fire rescue, industrial inspection, medical diagnosis and other fields. With the development of augmented reality and near-eye display technology, integrating infrared thermal imaging function into head-mounted near-eye display devices has become an important trend. Such devices combine infrared detection modules with near-eye display modules, allowing users to observe infrared thermal images of the scene in real time while wearing them. They have significant application value in scenarios where hands-free operation is required, such as fire rescue, night operation, and equipment inspection.

[0003] In existing head-mounted infrared thermal imaging display technologies, relevant research mainly focuses on the following aspects: In image acquisition and display, existing technologies acquire infrared images using infrared thermal imaging cameras on head-mounted display devices and synthesize these images with a pre-stored background based on the device's position and angle information; In image fusion, the fusion of infrared and visible light images is a research hotspot. Existing methods extract thermal information features from infrared images and color features from visible light images, employing dynamic fusion methods to identify image targets; other methods allow manual adjustment of the fusion weights of visible light and infrared using weight controls on the user interface; In temperature information display, handheld thermal imagers generally possess automatic hot and cold spot tracking capabilities, enabling real-time marking of extreme temperature points in images and labeling temperature values; In near-eye display, existing research has proposed dynamic resolution rendering methods based on gaze tracking, improving display efficiency and image smoothness through dynamic resolution allocation.

[0004] However, existing technologies still have the following shortcomings: First, existing infrared thermal imaging near-eye display devices lack the ability to adapt to changes in the display optical path when the display lens is retractable. When the position or angle of the display lens changes relative to the user's line of sight, a projection offset will occur between the spatial location of the temperature extreme point in the image and the superimposed information of the image. This results in the user's perception of the extreme point position being inconsistent with the actual heat source position, reducing the accuracy and usability of thermal image information.

[0005] Second, existing infrared image fusion and rendering methods are mainly designed for static display scenarios or handheld devices, without considering the impact of dynamic changes in the display optical path on image rendering in head-mounted near-eye display devices. When the display lens retracts or expands, the field of view changes accordingly. However, the temperature extreme value tracking, edge indicator markers, and OSD information overlay in existing methods all use fixed display parameters. They cannot dynamically adjust the position compensation of the extreme value cursor, the pointing sensitivity of the edge indicator markers, and the display strategy of the OSD information area according to changes in the display optical path, which affects the user's visual experience and information acquisition efficiency in dynamic usage scenarios.

[0006] Third, existing OSD information overlay methods do not consider the spatial occlusion problem between the OSD information area and the dynamically tracked target (such as the temperature extreme value cursor) in head-mounted near-eye display scenarios. When the OSD information area and the extreme value cursor overlap in the image, the user cannot clearly obtain both types of information at the same time, and existing technologies lack intelligent avoidance mechanisms for this scenario. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides an infrared thermal imaging fusion and adaptive rendering method and system based on near-eye displays. The objective of this invention can be achieved through the following technical solutions: Includes the following steps: Infrared thermal imaging data and display optical path adjustment parameters are acquired, and a dynamic temperature matrix is ​​generated through noise filtering and radiometric calibration; the display optical path adjustment parameters include at least the retraction and extension position information of the display lens relative to the line of sight; The dynamic temperature matrix is ​​traversed to lock the extreme temperature points and generate motion trajectories. The motion trajectory is smoothed by an interpolation algorithm, and extreme value cursors and real-time temperature labels are generated simultaneously. Based on the display optical path adjustment parameters, the edge indicator mark angle and nonlinear deviation amplitude are calculated, and the projection offset of the extreme value vernier and the real-time temperature label is dynamically compensated. Coordinate mapping is performed and the OSD information area is generated by superimposing the coordinates. The spatial relationship between the motion trajectory and the OSD information area is monitored, and the transparency of the OSD information area is adjusted or displacement is performed according to the rate of change of the display light path adjustment parameters, and finally the blended rendering image is output.

[0008] Specifically, the process of acquiring infrared thermal imaging data and displaying optical path adjustment parameters is as follows: The raw thermal radiation signal is collected by an infrared detector array, and the pulse count value of the lens displacement sensor is read synchronously. The pulse count value is converted into lens pitch angle and telescopic travel data, and timestamped with the original thermal radiation signal to establish a multi-source data synchronous acquisition link, outputting infrared thermal imaging data and displaying optical path adjustment parameters.

[0009] Specifically, the process of generating the dynamic temperature matrix through noise filtering and radiation calibration is as follows: Median filtering is performed on the original thermal radiation signal to remove salt-and-pepper noise, and a non-uniformity correction algorithm is used to eliminate differences in detector pixel response. By combining a pre-set blackbody radiation lookup table, the corrected signal values ​​are mapped to absolute temperature values, and then reassembled into the dynamic temperature matrix according to the frame sequence to construct a standardized temperature data set.

[0010] Specifically, the process of locking the extreme temperature point and generating the motion trajectory is as follows: The pixel coordinates corresponding to the global highest and lowest temperature values ​​are retrieved frame by frame in the dynamic temperature matrix. The pixel coordinates of multiple consecutive frames are input into the Kalman filter model for state prediction and measurement update. Abnormal jump points are removed and a smooth curve is fitted to generate the motion trajectory and output a spatiotemporal sequence of continuous extreme points.

[0011] Specifically, the process of generating the extreme value vernier and the real-time temperature label is as follows: Extract the coordinates of the extreme point in the current frame of the motion trajectory, and render a crosshair graphic at the coordinate position as an extreme point cursor; Read the temperature value at the corresponding coordinate in the dynamic temperature matrix, convert the temperature value into a character dot matrix and bind it to the side of the extreme value cursor to form the real-time temperature label, and generate a set of interface interactive elements.

[0012] Specifically, the process of calculating the edge indicator mark angle and the nonlinear deviation amplitude is as follows: The angle between the vectors of the extreme point coordinates and the center point of the display screen is calculated as the angle of the edge indicator mark. Obtain the field of view scaling factor corresponding to the lens extension / retraction travel data, substitute the field of view scaling factor into the exponential decay function to calculate the nonlinear deviation amplitude, so that the deviation amplitude increases nonlinearly with the degree of lens retraction.

[0013] Specifically, the process for dynamically compensating for the projection offset between the extreme value vernier and the real-time temperature label is as follows: A three-dimensional spatial transformation matrix is ​​constructed based on the display optical path adjustment parameters, and the initial screen coordinates of the extreme value vernier and the real-time temperature tag are multiplied by the three-dimensional spatial transformation matrix. Calculate the pixel-level displacement difference caused by optical path deformation, and then superimpose the displacement difference onto the initial screen coordinates to eliminate the visual deviation caused by optical path deformation.

[0014] Specifically, the process of performing coordinate mapping and superimposing to generate the OSD information area is as follows: The compensated extreme value vernier and the real-time temperature tag coordinates are transformed to the physical pixel coordinate system of the near-eye display panel; Read device operating status data and battery remaining power data, render the data as a semi-transparent rectangular layer, and overlay it on the temperature screen in a preset layer order to generate the OSD information area.

[0015] Specifically, the process of adjusting the transparency of the OSD information area or performing shift avoidance is as follows: Calculate the Euclidean distance between the endpoint of the motion trajectory and the bounding box of the OSD information area in real time; When the Euclidean distance is less than a preset safety threshold, the rate of change of the lens extension stroke data is extracted, and the rate of change is mapped to a transparency attenuation coefficient to reduce the layer transparency value, or the OSD information area boundary box is translated along the direction perpendicular to the motion trajectory.

[0016] Specifically, the process of finally outputting the blended rendering image is as follows: The offset-compensated extreme value cursor layer, the real-time temperature label layer, and the OSD information area layer are input into the graphics rendering pipeline; A multi-channel transparency blending calculation is performed to generate a single-frame composite image. The single-frame composite image is written into the frame buffer of the display controller, and the near-eye display panel is driven to perform photoelectric conversion at a fixed refresh rate.

[0017] Specifically, the process of translating the OSD information area boundary box is as follows: Obtain the initial and target position coordinates of the bounding box, and use a linear interpolation algorithm to calculate the intermediate transition coordinate sequence; The intermediate transition coordinate sequence is written into the graphics rendering pipeline in segments according to the display refresh cycle. Before each frame is rendered, the rendering area boundary of the OSD information area is updated, and the smoothly shifted display image is output.

[0018] Specifically, the infrared thermal imaging fusion and adaptive rendering system based on near-eye display includes a processor.

[0019] The beneficial effects of this invention are as follows: (1) By setting a step to dynamically compensate for the projection offset of the extreme value vernier and the real-time temperature label according to the display optical path adjustment parameters, the extreme value vernier and the real-time temperature label are always visually aligned with the spatial position of the actual temperature extreme point during the retraction and extension of the display lens. This effectively eliminates the projection deviation caused by optical path deformation, solves the problem of inaccurate positioning of thermal images in dynamic usage scenarios of head-mounted devices, and significantly improves the spatial accuracy and availability of infrared thermal image information in near-eye display. (2) By setting steps such as monitoring the spatial relationship between the motion trajectory and the OSD information area, dynamically adjusting the transparency of the OSD information area, or performing displacement avoidance, the avoidance strategy can be automatically triggered according to the motion trend of the temperature extreme point and the optical path adjustment rate, so as to avoid the interface information from obscuring the thermal image target, ensuring the synchronous readability of temperature extreme tracking information and device status information, and optimizing the visual interaction experience in the context of head-mounted near-eye display. Attached Figure Description

[0020] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating the infrared thermal imaging fusion and adaptive rendering method based on near-eye display of the present invention. Figure 2 This is a structural block diagram of the infrared thermal imaging fusion and adaptive rendering system based on near-eye display of the present invention; Figure 3 This is a schematic diagram of temperature extreme value marking and OSD information display under white-thermal mode; Figure 4 This is a schematic diagram showing the temperature extreme value markings and OSD information display in lava mode. Detailed Implementation

[0022] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0023] Please see Figures 1-4 A method and system for infrared thermal imaging fusion and adaptive rendering based on near-eye display; Includes the following steps: Step 1: Acquire infrared thermal imaging data and display optical path adjustment parameters, and generate a dynamic temperature matrix through noise filtering and radiometric calibration; the display optical path adjustment parameters shall at least include the retraction and extension position information of the display lens relative to the line of sight; In this embodiment, infrared thermal imaging data refers to the raw digital data generated by the infrared detector array built into the head-mounted near-eye display device after real-time collection of the target scene thermal radiation signal and analog-to-digital conversion. It includes the grayscale value or response voltage value of each pixel. This data has a corresponding relationship with the temperature distribution at each location in the target scene. Display optical path adjustment parameters refer to a set of parameters that characterize the current mechanical position and optical state of the display lens. They include at least the retraction and extension position information of the display lens relative to the user's line of sight. Specifically, the retraction and extension position information includes lens extension stroke data and lens pitch angle data. The extension stroke data reflects the displacement of the display lens along the optical axis relative to its initial position, and the pitch angle data reflects the rotation angle of the display lens around the horizontal axis. Together, they determine the current distance and current angle between the eyepiece and the user's eye. The retraction and extension position information is obtained by setting a displacement sensor or angle sensor at the mechanical pivot and sliding rail of the display lens. When the user manually or the device automatically adjusts the retraction and extension of the display lens, the sensor outputs a detection signal synchronously. After sampling and analog-to-digital conversion, digital retraction and extension position information is formed. In this embodiment, the display lens preferably adopts a hinged flip structure or a sliding rail telescopic structure to achieve the function of free retraction and extension of the lens without obstructing the line of sight. When the lens is fully retracted, it is outside the user's line of sight. When fully extended, it is located in front of the user's eyes and perpendicular to the line of sight. The specific process of acquiring infrared thermal imaging data and displaying optical path adjustment parameters is as follows: the infrared detector array is started at a fixed frame rate to collect the original thermal radiation signal within the current field of view, and the output signal of the lens displacement sensor is read at a sampling frequency of not less than 100Hz. The two sets of data are then time-stamped, packaged, and transmitted to the image processing unit. The frame rate is preferably 25Hz or 30Hz, which balances image quality and system power consumption while ensuring real-time performance. The specific process for acquiring infrared thermal imaging data and display optical path adjustment parameters is as follows: Raw thermal radiation signals are acquired through an infrared detector array. The infrared detector array refers to a focal plane array detector composed of multiple infrared sensitive units arranged in a planar array. Each sensitive unit independently responds to incident infrared radiation and outputs a corresponding electrical signal. The output signals of all sensitive units are read in parallel by a readout circuit to form a frame of raw thermal radiation signal image. Simultaneously, the pulse count value of the lens displacement sensor is read. The lens displacement sensor is a position detection element installed at the display lens extension / retraction mechanism, used to convert the mechanical displacement of the lens into a measurable electrical signal. The pulse count value is the value obtained after accumulating the pulse signals output by the sensor; this value has a definite correspondence with the amount of lens displacement. The pulse count... The numerical values ​​are converted into lens pitch angle and telescopic travel data. The conversion method is based on the sensor's factory calibration parameters or on-site calibration curve. The pulse count value is substituted into a preset conversion formula to calculate the corresponding angle value and travel value. The pitch angle and telescopic travel data are timestamped with the original thermal radiation signal. Timestamping means marking the sensor data and image data collected at the same time with the same time tag to ensure that the two types of data are strictly synchronized in time during subsequent processing. A multi-source data synchronous acquisition link is established to output infrared thermal imaging data and display optical path adjustment parameters. The multi-source data synchronous acquisition link refers to the complete data acquisition and transmission channel from the infrared detector and displacement sensor to the image processing unit, including signal conditioning, analog-to-digital conversion, timing synchronization, and data packet transmission. The specific process of noise filtering and radiometric calibration of the original thermal radiation signal is as follows: First, the median filtering algorithm is used to filter the original thermal radiation signal to remove salt-and-pepper noise and random noise introduced by the detector during signal transmission and amplification. The preferred window size for median filtering is 3×3 pixels or 5×5 pixels, which effectively suppresses noise while preserving image edge and detail information to the maximum extent. Then, the radiometric calibration method is used to perform temperature inversion on the filtered signal. Radiometric calibration refers to the process of mapping the original response value (grayscale value or voltage value) of the infrared detector to the real temperature value through a preset calibration curve. This calibration curve is obtained in advance by calibrating the detector at different temperature points using a blackbody radiation source. Specifically, it is the fitting relationship curve between the response value and the temperature value. Through radiometric calibration, the original response value of each pixel in the entire frame image is mapped to the corresponding temperature value one by one, generating a temperature matrix corresponding to the current field of view. The specific process of generating a dynamic temperature matrix through noise filtering and radiation calibration is as follows: Median filtering is performed on the original thermal radiation signal to remove salt-and-pepper noise. Median filtering is a non-linear filtering method that uses the median value of all pixels within its neighborhood window as the output value for each pixel, effectively removing isolated noise points without blurring image edges. A non-uniformity correction algorithm is used to eliminate detector pixel response differences. Non-uniformity correction addresses the inconsistency in response rates caused by manufacturing process deviations among the detector units in the focal plane array. The response curve of each pixel is normalized using a two-point or multi-point correction method to ensure consistent output from all pixels under the same irradiance. The correction coefficients for non-uniformity correction are pre-defined in the equipment. The signal is calibrated using uniform blackbody radiation at the factory and updated periodically during use. A pre-set blackbody radiation lookup table maps the corrected signal values ​​to absolute temperature values. This blackbody radiation lookup table is a pre-stored table of response values ​​and temperatures obtained from blackbody radiation sources calibrated at various temperature points. During processing, the response values ​​of each pixel are converted into precise temperature values ​​through table lookup and interpolation. The signal is then reassembled into a dynamic temperature matrix according to the frame sequence. Each frame of the original thermal radiation signal, after the above filtering, correction, and mapping processes, forms a two-dimensional temperature data array corresponding to the current field of view. These temperature arrays are arranged according to the acquisition sequence to form a dynamic temperature matrix, constructing a standardized temperature data set for subsequent extreme value retrieval and tracking. The specific process of generating a dynamic temperature matrix based on the display optical path adjustment parameters is as follows: After generating the temperature matrix, the current effective field of view is determined based on the retraction and extension position information in the display optical path adjustment parameters. The current effective field of view refers to the boundary of the scene area that the user can actually observe through the eyepiece in the current retraction and extension state of the display lens. When the lens retracts, the field of view expands, and the pixel domain participating in the subsequent extreme value retrieval in the temperature matrix increases. When the lens extends, the field of view narrows, and the pixel domain participating in the extreme value retrieval shrinks accordingly. The pixel region corresponding to the current effective field of view is extracted from the complete temperature matrix to form a dynamic temperature matrix, ensuring that the retrieval range of subsequent temperature extreme points always remains consistent with the actual display field of view.

[0024] Step 2: Traverse the dynamic temperature matrix to lock the extreme temperature points and generate motion trajectories. Use an interpolation algorithm to smooth the motion trajectories and simultaneously generate extreme value cursors and real-time temperature labels. In this embodiment, the extreme temperature points refer to the pixel positions corresponding to the highest and lowest temperatures within the current effective field of view. The highest temperature point corresponds to the hottest target area in the scene, and the lowest temperature point corresponds to the coldest target area in the scene. The specific process of locking the extreme temperature points is as follows: traverse all pixels in the dynamic temperature matrix, find the pixel with the largest temperature value as the highest temperature point by comparing pixels one by one, and find the pixel with the smallest temperature value as the lowest temperature point, and record the corresponding pixel coordinates and temperature values; to ensure the accuracy and anti-interference ability of the locking process, outlier values ​​are removed from the dynamic temperature matrix before traversal, that is, isolated points whose temperature values ​​deviate significantly from adjacent pixels due to detector dead pixels or instantaneous interference are excluded. In this embodiment, the specific process of locking the temperature extreme points and generating the motion trajectory is as follows: The pixel coordinates corresponding to the highest and lowest global temperature values ​​are retrieved frame by frame in the dynamic temperature matrix. Frame-by-frame retrieval means independently performing a complete traversal search on each frame of the dynamic temperature matrix to independently determine the highest and lowest temperatures and their corresponding pixel positions in that frame, rather than relying on the results of the previous frame, thus eliminating the risk of error accumulation. The pixel coordinates of multiple consecutive frames are then input into the Kalman filter model for state prediction and measurement update. The Kalman filter is a recursive optimal estimation algorithm based on a state-space model, comprising two steps: prediction and update. The measurement step predicts the prior estimate of the current frame's state based on the estimated state of the previous frame. The update step uses the measurement value of the current frame to weight and correct the prior estimate to obtain the posterior estimate. Through iterative calculation using Kalman filtering, abnormal jump points caused by detector random noise or environmental interference are eliminated. At the same time, the smoothing characteristics of the filter are used to fit the coordinate sequence to generate a smooth motion trajectory. The output is a continuous spatiotemporal sequence of extreme points. The spatiotemporal sequence refers to the sequence of extreme point positions containing timestamp information and spatial coordinate information. Each frame corresponds to a spatiotemporal sample point. The complete sequence records the continuous motion process of the extreme points in the time dimension. The specific process of smoothing the motion trajectory using an interpolation algorithm is as follows: When the coordinates of extreme points between adjacent frames fluctuate due to environmental noise or random errors of the detector, the extreme point coordinate sequence of multiple consecutive frames is smoothed using an interpolation algorithm to remove abnormal jump points and fit a smooth curve; the preferred interpolation algorithm is the Kalman filter algorithm, which establishes a state prediction equation and a measurement update equation to optimally estimate the motion state of the extreme points, maintaining a fast tracking response to the real motion while smoothing the trajectory; the state vector of the Kalman filter contains the pixel coordinates of the extreme points and their rate of change, and the measurement vector is the coordinates of the extreme points locked in the current frame. Through iterative calculations of the prediction and update steps, the smoothed extreme point coordinates are output as the final positioning result of the current frame; in this embodiment, the process noise covariance and measurement noise covariance of the Kalman filter are pre-calibrated and set according to the motion characteristics and sensor noise level in the actual application scenario; The specific process for generating the extreme value cursor and real-time temperature label is as follows: Extract the coordinates of the extreme point of the current frame from the smoothed motion trajectory. Render a crosshair graphic at this coordinate position as the extreme value cursor. The line width of the crosshair is preferably 1 to 2 pixels, and the color uses a warning color (such as red or orange) that contrasts sharply with the background infrared image, ensuring that the user can clearly identify the extreme point position even in high-brightness or complex backgrounds. Simultaneously, read the temperature value of the corresponding coordinate in the dynamic temperature matrix, convert the temperature value into a character dot matrix, and bind it to the side of the extreme value cursor to form a real-time temperature label. The real-time temperature label and the extreme value cursor constitute a complete set of interactive indicator elements, moving synchronously with the movement of the extreme point. In this embodiment, the highest and lowest temperature points are distinguished by crosshairs of different colors. For example, the highest temperature point uses a red crosshair with a red temperature label, and the lowest temperature point uses a blue crosshair with a blue temperature label, to facilitate quick identification by the user. In this embodiment, the specific process of generating the extreme value cursor and real-time temperature label is as follows: Extract the coordinates of the extreme point in the current frame of the motion trajectory; render a crosshair graphic at the coordinate position as the extreme value cursor. The crosshair consists of two mutually perpendicular line segments, with their intersection located at the extreme point coordinates, used to accurately mark the spatial position of the extreme point on the display screen; read the temperature value at the corresponding coordinate in the dynamic temperature matrix. The temperature value is the temperature value obtained after radiometric calibration, in degrees Celsius or Fahrenheit; convert the temperature value into a character dot matrix. The character dot matrix refers to converting numbers and unit symbols into a two-dimensional array composed of pixels according to a preset font. The graphic array, where each character consists of a fixed number of pixel rows and columns, can graphically present text information on the display screen; the character dot matrix is ​​rendered to the left or right of the extreme value cursor position, making the temperature value and the crosshair form an inseparable combined indicator unit, which moves as a whole as the extreme value point moves; a set of interface interactive elements is generated, which refers to the graphic unit group composed of the extreme value cursor and its bound real-time temperature label. This group of elements is the basic interactive unit for subsequent spatial conflict detection and avoidance processing with the OSD information area.

[0025] like Figure 3 , Figure 4 As shown, in two different pseudo-color modes (incandescent and lava), the real-time values ​​of the highest temperature point (TH), the central reference point (TC), and the lowest temperature point (TL) are clearly marked in the image, along with the corresponding extreme value cursor (crosshair) positions, enabling users to quickly identify extreme temperature areas in the scene.

[0026] Step 3: Based on the display optical path adjustment parameters, calculate the edge indicator mark angle and nonlinear deviation amplitude, dynamically compensate for the projection offset of the extreme value vernier and the real-time temperature label, perform coordinate mapping and superimpose to generate the OSD information area; In this embodiment, the edge indicator mark refers to the guiding arrow or direction indicator symbol generated at the edge of the display screen when the temperature extreme point moves out of the current effective display area, which is used to indicate to the user which direction the extreme point is located outside the current field of view; the nonlinear deviation amplitude refers to the degree to which the deflection amplitude of the edge indicator mark changes nonlinearly with the lens extension and retraction state; The specific process for calculating the edge indicator angle and nonlinear deviation amplitude is as follows: The angle between the vectors of the extreme point coordinates and the center point of the display screen is calculated as the edge indicator angle. The vector angle refers to the angle between the direction vector formed with the center point of the display screen as the starting point and the extreme point coordinates as the ending point and the horizontal rightward direction of the display screen. This angle determines the pointing direction of the edge indicator at the edge of the screen. The field of view scaling factor corresponding to the lens extension / retraction data is obtained. The field of view scaling factor is a pre-calibrated parameter used to describe the scaling ratio of the field of view angle relative to the standard field of view angle under different extension / retraction strokes. When the lens retracts, the field of view expands, and the scaling factor increases; when the lens extends, the field of view narrows, and the scaling factor decreases. The field of view scaling factor is substituted into the exponential decay function to calculate the nonlinear deviation amplitude, so that the deviation amplitude increases nonlinearly with the increase of the lens retraction degree. The specific parameters of the exponential decay function are pre-calibrated and set based on the lens optical characteristics and user visual perception experimental data. The specific process of dynamically compensating for the projection offset of the extreme value vernier and the real-time temperature label is as follows: When the position or angle of the display lens changes relative to the user's line of sight, the optical path between the eyepiece and the eye changes, resulting in a projection offset between the superimposed information on the display screen and the real scene; based on the lens extension / retraction data and pitch angle data in the display optical path adjustment parameters, a three-dimensional spatial transformation matrix is ​​constructed in combination with the pre-calibrated optical system parameters. This transformation matrix describes the mapping relationship from physical space coordinates to pixel coordinates of the display screen; the initial screen coordinates of the extreme value vernier and the real-time temperature label are multiplied by the three-dimensional spatial transformation matrix to calculate the pixel-level displacement difference caused by optical path deformation. This displacement difference is then superimposed on the initial screen coordinates to eliminate the visual deviation caused by optical path deformation, ensuring that the extreme value vernier and the real-time temperature label always maintain visual alignment with the spatial position of the actual temperature extreme point in the image during the lens extension and retraction process; The specific process of dynamically compensating for the projection offset of the extreme value vernier and the real-time temperature label is as follows: A three-dimensional spatial transformation matrix is ​​constructed based on the display optical path adjustment parameters. This matrix is ​​a homogeneous coordinate transformation matrix containing rotation and translation components, used to describe the rigid transformation relationship between the display screen coordinate system and the initial reference coordinate system caused by the extension and retraction of the display lens. This matrix is ​​pre-established based on the kinematic model of the lens mechanical structure and the imaging geometry of the optical system. The initial screen coordinates of the extreme value vernier and the real-time temperature label are multiplied by the three-dimensional spatial transformation matrix to obtain the actual display coordinates under the current lens extension and retraction state. The displacement difference between the actual display coordinates and the initial screen coordinates in the horizontal and vertical directions is calculated. This displacement difference is then superimposed onto the initial screen coordinates, causing the position of the extreme value vernier and the real-time temperature label on the screen to shift in the opposite direction of the optical path deformation, thereby compensating for the visual deviation caused by lens movement. The specific process of performing coordinate mapping and overlaying to generate the OSD information area is as follows: The extreme value cursor coordinates and real-time temperature label coordinates, after projection offset compensation, are transformed from the image processing coordinate space to the physical pixel coordinate system of the near-eye display panel; at the same time, the current operating status data of the device is read, including the current display mode information, the remaining battery power information, and the overall temperature range information of the dynamic temperature matrix. The above information is rendered into a semi-transparent rectangular layer according to a preset layout, and this layer is the OSD information area; the OSD information area is overlaid on the infrared temperature screen in a preset hierarchical order, with the extreme value cursor and real-time temperature label on the top layer to ensure that they are always visible, the OSD information area on the middle layer, and the infrared temperature screen on the bottom layer, forming a complete display screen; The specific process of performing coordinate mapping and overlaying to generate the OSD information area is as follows: The compensated extreme value cursor and real-time temperature label coordinates are transformed to the physical pixel coordinate system of the near-eye display panel. The physical pixel coordinate system refers to a pixel coordinate system with the upper left corner of the display panel as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. Scaling and translation transformations are needed to map the logical coordinates to the actual physical pixel positions. Device operating status data and remaining battery power data are read. The device operating status data includes the current display mode information and the overall temperature range information of the dynamic temperature matrix. The remaining battery power data refers to the percentage value of the device's current battery power. The data is rendered as a semi-transparent rectangular layer, which is a graphic area with a rectangular frame as the background, filled with a semi-transparent color, and superimposed with text information. The transparency is preferably 70% to 80%. The OSD information area is generated by overlaying the temperature screen in a preset layer order. The preset layer order refers to the infrared temperature screen layer, OSD information area layer, and extreme value interactive element layer from the bottom layer to the top layer. Each layer is combined in this order to ensure that the extreme value cursor and real-time temperature label are always on the top layer and are not obscured by the OSD information area.

[0027] Step 4: Monitor the spatial relationship between the motion trajectory and the OSD information area, adjust the transparency of the OSD information area or perform displacement avoidance according to the rate of change of the display light path adjustment parameters, and finally output the blended rendering screen; In this embodiment, the rate of change refers to the amount of change of the display optical path adjustment parameters per unit time, specifically corresponding to the speed of the display lens retraction and extension. The specific process of adjusting the transparency of the OSD information area or performing shift avoidance is as follows: The Euclidean distance between the endpoint of the motion trajectory and the bounding box of the OSD information area is calculated in real time. The endpoint of the motion trajectory refers to the coordinate position of the current frame's extreme cursor in the display screen. The bounding box of the OSD information area refers to the closed border formed by the four sides of the rectangular area occupied by the OSD information area. The Euclidean distance refers to the shortest straight-line distance from the endpoint coordinates to the four sides of the bounding box. When the Euclidean distance is less than a preset safety threshold, the avoidance mechanism is triggered. The rate of change of the lens extension / retraction data is extracted and mapped to a transparency attenuation coefficient to reduce the layer's transparency value. The greater the rate of change, the higher the transparency of the OSD information area. Alternatively, the bounding box of the OSD information area can be translated along a direction perpendicular to the motion trajectory to avoid the movement path of the extreme cursor. The specific process of translating the OSD information area bounding box is as follows: obtain the initial position coordinates and target position coordinates of the bounding box. The target position coordinates refer to the position after avoidance calculated according to the preset translation direction and translation distance; use a linear interpolation algorithm to generate several intermediate position points between the initial position and the target position at equal steps to make the movement of the bounding box present a smooth and continuous visual effect; write the intermediate transition coordinate sequence into the graphics rendering pipeline in segments according to the display refresh cycle, update the rendering area boundary of the OSD information area before rendering each frame, so that the OSD information area moves gradually frame by frame to complete the smooth displacement avoidance; The specific process of finally outputting the blended rendering image is as follows: the extreme value cursor layer, real-time temperature label layer and OSD information area layer after projection offset compensation, together with the infrared temperature image layer, are input into the graphics rendering pipeline. Multi-channel transparency mixing calculation is performed to generate a single frame composite image. The single frame composite image is written into the frame buffer area of ​​the display controller. The near-eye display panel is driven to perform photoelectric conversion according to the set refresh rate. Finally, the blended rendering image is output for the user to observe through the eyepiece. The specific process for finally outputting the blended rendered image is as follows: The offset-compensated extreme value cursor layer, real-time temperature label layer, and OSD information area layer are input into the graphics rendering pipeline. The graphics rendering pipeline refers to the processing pipeline in the graphics processing unit used to synthesize multiple layers into a single-frame image. Multi-channel transparency blending calculations are performed to generate a single-frame composite image. Multi-channel transparency blending refers to performing blending calculations for each pixel position based on the pixel color value and transparency value of each layer at that position, according to a preset weighted superposition rule, to obtain the final composite color value. The single-frame composite image is written into the frame buffer of the display controller. The frame buffer refers to the storage area in the display controller used to store the image data to be displayed. The near-eye display panel is driven to perform photoelectric conversion at a fixed refresh rate, preferably 25Hz to 60Hz. The display panel converts the digital image data in the frame buffer into optical signals and outputs them to the user's eyepiece.

[0028] Example: A head-mounted infrared thermal imaging near-eye display device for a fire rescue scenario; Firefighters wearing head-mounted near-eye display devices enter the fire scene. After the device is activated, it automatically executes the initialization process. The infrared detector array begins to collect infrared thermal radiation signals from the scene. The detector's operating wavelength covers 8μm to 14μm, and the detectable temperature range is -20°C to 1500°C. The single measurement response time is no more than 30ms. Firefighters manually unfold the display lens from the retracted state to the front of their line of sight according to the scene conditions. The device simultaneously acquires the display optical path adjustment parameters, including the current lens extension and tilt angle data. The system performs median filtering and radiometric calibration on the collected raw thermal radiation signals to generate a temperature matrix for the current field of view. Based on the lens unfolding state, it extracts the dynamic temperature matrix corresponding to the effective field of view from the complete temperature matrix. Subsequently, the system traverses the dynamic temperature matrix, locks the highest temperature point (such as the ignition point or high-temperature obstacle) and the lowest temperature point (such as a low-temperature area or window) in the current field of view, generates extreme value cursors and real-time temperature labels, and overlays them on the display screen. The end-to-end latency of the entire processing process is no more than 100ms. As firefighters move through a fire scene, their head rotations or movements cause slight displacement of the display lens. The system acquires real-time data on changes in the display's optical path adjustment parameters and dynamically compensates for the projection offset between the extreme value cursor and the real-time temperature label. This ensures that the extreme temperature point on the display screen is always precisely aligned with the heat source location in the actual scene. During the lens's retraction and extension, the system continuously reads displacement sensor data at a sampling frequency of 100Hz. The projection offset compensation accuracy between the extreme value cursor and the temperature label reaches the sub-pixel level, controlling the positioning error to within 0.2px. This corresponds to a spatial positioning deviation of no more than 2mm in the actual scene, effectively guaranteeing the accuracy of heat source indication in dynamic usage scenarios. When the extreme temperature point moves out of the display edge due to the user turning their head, the system generates an indicator mark at the edge of the screen to guide firefighters to turn towards the heat source. The pointing angle error of the indicator mark does not exceed ±3°. When the extreme value cursor moves near the OSD information area, the system automatically makes the OSD information area semi-transparent or moves it away to avoid information obstruction. The trigger safety threshold for the avoidance mechanism is set to 50px. When the Euclidean distance between the extreme value cursor and the OSD information area boundary box is less than this threshold, the system completes the avoidance action response within a single frame processing cycle (approximately 33ms) to ensure that the user's line of sight is not disturbed. The system continuously outputs fused rendering images at a refresh rate of 25 frames per second. Firefighters can obtain real-time temperature distribution information on site through the eyepiece, and complete fire source location and safety path determination without the need for handheld devices. The device's built-in battery can support continuous operation for more than 4 hours. A single monitoring can cover the distribution of heat sources within a radius of 100m, and the temperature measurement accuracy is better than ±2°C or ±2% of the reading. The above process is continuously executed in a loop until the device is turned off or the firefighters manually retract the display lens, realizing handheld, high-precision, adaptive near-eye display of infrared thermal imaging information in fire rescue scenarios.

[0029] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An infrared thermal imaging fusion and adaptive rendering method based on near-eye display, characterized in that, Includes the following steps: Acquire infrared thermal imaging data and display optical path adjustment parameters, and generate a dynamic temperature matrix through noise filtering and radiometric calibration; The display optical path adjustment parameters include at least the retraction and extension position information of the display lens relative to the line of sight, and the display lens can rotate freely around the axis of rotation; The dynamic temperature matrix is ​​traversed to lock the extreme temperature points and generate motion trajectories. The motion trajectory is smoothed by an interpolation algorithm, and extreme value cursors and real-time temperature labels are generated simultaneously. Based on the display optical path adjustment parameters, the edge indicator mark angle and nonlinear deviation amplitude are calculated, and the projection offset of the extreme value vernier and the real-time temperature label is dynamically compensated. Coordinate mapping is performed and the OSD information area is generated by superimposing the coordinates. The spatial relationship between the motion trajectory and the OSD information area is monitored, and the transparency of the OSD information area is adjusted or displacement is performed according to the rate of change of the display light path adjustment parameters, and finally the blended rendering image is output.

2. The method according to claim 1, characterized in that, The specific process for acquiring infrared thermal imaging data and displaying optical path adjustment parameters is as follows: The raw thermal radiation signal is collected by an infrared detector array, and the pulse count value of the lens displacement sensor is read synchronously. The pulse count value is converted into lens pitch angle and telescopic travel data, and timestamped with the original thermal radiation signal to establish a multi-source data synchronous acquisition link, outputting infrared thermal imaging data and displaying optical path adjustment parameters.

3. The method according to claim 1, characterized in that, The specific process of generating the dynamic temperature matrix through noise filtering and radiation calibration is as follows: Median filtering is performed on the original thermal radiation signal to remove salt-and-pepper noise, and a non-uniformity correction algorithm is used to eliminate differences in detector pixel response. By combining a pre-set blackbody radiation lookup table, the corrected signal values ​​are mapped to absolute temperature values, and then reassembled into the dynamic temperature matrix according to the frame sequence to construct a standardized temperature data set.

4. The method according to claim 1, characterized in that, The specific process of locking the extreme temperature point and generating the motion trajectory is as follows: The pixel coordinates corresponding to the global highest and lowest temperature values ​​are retrieved frame by frame in the dynamic temperature matrix. The pixel coordinates of multiple consecutive frames are input into the Kalman filter model for state prediction and measurement update. Abnormal jump points are removed and a smooth curve is fitted to generate the motion trajectory and output a spatiotemporal sequence of continuous extreme points.

5. The method according to claim 1, characterized in that, The specific process for generating the extreme value vernier and real-time temperature label is as follows: Extract the coordinates of the extreme point in the current frame of the motion trajectory, and render a crosshair graphic at the coordinate position as an extreme point cursor; Read the temperature value at the corresponding coordinate in the dynamic temperature matrix, convert the temperature value into a character dot matrix and bind it to the side of the extreme value cursor to form the real-time temperature label, and generate a set of interface interactive elements.

6. The method according to claim 1, characterized in that, The specific process for calculating the edge indicator marker angle and the nonlinear deviation magnitude is as follows: The angle between the vectors of the extreme point coordinates and the center point of the display screen is calculated as the angle of the edge indicator mark. Obtain the field of view scaling factor corresponding to the lens extension / retraction travel data, substitute the field of view scaling factor into the exponential decay function to calculate the nonlinear deviation amplitude, so that the deviation amplitude increases nonlinearly with the degree of lens retraction.

7. The method according to claim 1, characterized in that, The specific process for dynamically compensating for the projection offset between the extreme value vernier and the real-time temperature label is as follows: A three-dimensional spatial transformation matrix is ​​constructed based on the display optical path adjustment parameters, and the initial screen coordinates of the extreme value vernier and the real-time temperature tag are multiplied by the three-dimensional spatial transformation matrix. Calculate the pixel-level displacement difference caused by optical path deformation, and then superimpose the displacement difference onto the initial screen coordinates to eliminate the visual deviation caused by optical path deformation.

8. The method according to claim 1, characterized in that, The specific process of performing coordinate mapping and superimposing to generate the OSD information area is as follows: The compensated extreme value vernier and the real-time temperature tag coordinates are transformed to the physical pixel coordinate system of the near-eye display panel; Read device operating status data and battery remaining power data, render the data as a semi-transparent rectangular layer, and overlay it on the temperature screen in a preset layer order to generate the OSD information area.

9. The method according to claim 1, characterized in that, The specific process of adjusting the transparency of the OSD information area or performing shift avoidance is as follows: Calculate the Euclidean distance between the endpoint of the motion trajectory and the bounding box of the OSD information area in real time; When the Euclidean distance is less than a preset safety threshold, the rate of change of the lens extension stroke data is extracted, and the rate of change is mapped to a transparency attenuation coefficient to reduce the layer transparency value, or the OSD information area boundary box is translated along the direction perpendicular to the motion trajectory.

10. The method according to claim 1, characterized in that, The specific process for the final output of the blended rendered image is as follows: The offset-compensated extreme value cursor layer, the real-time temperature label layer, and the OSD information area layer are input into the graphics rendering pipeline; A multi-channel transparency blending calculation is performed to generate a single-frame composite image. The single-frame composite image is written into the frame buffer of the display controller, and the near-eye display panel is driven to perform photoelectric conversion at a fixed refresh rate.

11. The method according to claim 9, characterized in that, The specific process of translating the OSD information area boundary box is as follows: Obtain the initial and target position coordinates of the bounding box, and use a linear interpolation algorithm to calculate the intermediate transition coordinate sequence; The intermediate transition coordinate sequence is written into the graphics rendering pipeline in segments according to the display refresh cycle. Before each frame is rendered, the rendering area boundary of the OSD information area is updated, and the smoothly shifted display image is output.

12. An infrared thermal imaging fusion and adaptive rendering system based on near-eye display, comprising a processor for performing the method as described in any one of claims 1 to 11.

Images