Vehicle-mounted thermal imaging intelligent sensing method for linkage air imaging system

Abstract

The present application relates to the technical field of vehicle auxiliary driving, in particular to a vehicle thermal imaging intelligent perception method of a linkage aerial imaging system. The method solves the technical problem that the prior art is difficult to realize stable and accurate adaptive parallax correction based on a thermal imaging video stream only. The method comprises: determining a first calibration parameter and a second calibration parameter based on user interaction operation; performing coordinate conversion on a tracking target detected in the thermal imaging video stream based on the first calibration parameter, and constructing a time sequence of normalized position features; performing similarity matching on the time sequence and a preset motion trajectory template to determine a matching degree weight; calculating a display position compensation amount based on the time sequence, the matching degree weight, the first calibration parameter and the second calibration parameter; and adjusting the projection position of the tracking target in the display picture based on the display position compensation amount. The present application is used in a vehicle thermal imaging intelligent perception scene.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted assisted driving technology, specifically to a vehicle thermal imaging intelligent perception method linked to an aerial imaging system. Background Technology

[0002] The integration of automotive thermal imaging technology with head-up displays (HUDs) or aerial imaging systems has become a crucial technological direction for improving vehicle safety at night and in low-visibility environments. By installing thermal imaging cameras at the front of the vehicle to detect heat-generating targets such as pedestrians and animals, and projecting warning information into the driver's field of vision in augmented reality form, the limitations of traditional optical sensors in adverse lighting conditions can be effectively compensated for. However, thermal imaging cameras are typically installed at the front of the vehicle (e.g., in the grille), while the generated warning images must be projected through a display system located in the cockpit. This physical separation leads to a difference in viewing angle. This difference causes a spatial discrepancy between the directly mapped warning image and the actual target, a parallax problem that affects the driver's intuitive and accurate judgment of dangerous locations. To eliminate this parallax, the position of the target in the thermal image needs to be corrected in real time. Existing correction schemes mostly rely on precise vehicle and camera geometry or additional ranging sensors, which are poorly applicable and costly in the aftermarket where installation parameters vary and vehicle models are diverse. Furthermore, relying solely on thermal imaging video streams makes stable and accurate adaptive parallax correction difficult to achieve. Summary of the Invention

[0003] To address the technical problem that existing solutions based solely on thermal imaging video streams struggle to achieve stable and accurate adaptive parallax correction, the present invention aims to provide an intelligent perception method for vehicle thermal imaging linked to an aerial imaging system. The specific technical solution adopted is as follows:

[0004] In a first aspect, the present invention provides a vehicle thermal imaging intelligent perception method for a linked aerial imaging system. The method includes: determining a first calibration parameter and a second calibration parameter based on user interaction operations; the user interaction operations include: adjusting the display position of a first reference graphic in the display screen until the first reference graphic visually coincides with the vanishing line of the road in the thermal imaging screen; adjusting the display parameters of a second reference graphic in the display screen until the second reference graphic is spatially aligned with a road surface reference point in the thermal imaging screen from the driver's perspective; the first calibration parameter is used to characterize the height of the vanishing point in the thermal imaging screen; the second calibration parameter is used to characterize the mapping relationship between the pixel displacement of the thermal imaging screen and the pixel displacement of the display screen; based on the first calibration parameter, performing coordinate transformation on the tracked target detected in the thermal imaging video stream to construct a time series of normalized position features; performing similarity matching between the time series and a preset motion trajectory template to determine the matching degree weight; calculating a display position compensation amount based on the time series, the matching degree weight, the first calibration parameter, and the second calibration parameter; and adjusting the projection position of the tracked target in the display screen based on the display position compensation amount.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the ordinate of the first reference graphic in the coordinate system of the thermal imaging image when it coincides with the vanishing line, and using the ordinate as the first calibration parameter; determining the adjustment gain value corresponding to the second reference graphic when it is aligned with the road surface reference point, and using the adjustment gain value as the second calibration parameter.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: detecting each frame of the thermal imaging video stream to determine the original image coordinates of the tracking target; performing temporal filtering on the original image coordinates to determine the smoothed target coordinates; calculating the normalized position features of the tracking target in each frame based on the target coordinates and the first calibration parameter; and generating a time series based on the normalized position features of multiple consecutive frames within a preset time period.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining normalized position features based on the relative positional relationship between the ordinate of the smoothed target coordinates and the first calibration parameter; the normalized position features are used to characterize the relative proportion of the tracking target in the vertical direction between the first calibration parameter and the bottom of the image.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: obtaining a preset motion trajectory template; the preset motion trajectory template is a preset data sequence that characterizes the normalized position feature change law when an object approaches longitudinally at a constant speed; determining the morphological difference degree between the time series and the preset motion trajectory template; determining the matching degree weight based on the morphological difference degree and a preset mapping relationship; the matching degree weight decreases as the morphological difference degree increases.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the curvature path with the lowest cost that aligns the time series with the preset motion trajectory template based on a dynamic time warping algorithm; and determining the cumulative distance between corresponding data points on the curvature path as the morphological difference degree.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: obtaining the last normalized position feature in the time series; determining the basic linear compensation component based on the second calibration parameter and the last normalized position feature; determining the additional compensation component based on the last normalized position feature using the matching degree weight as the modulation coefficient; and determining the display position compensation amount based on the basic linear compensation component and the additional compensation component.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining a first intermediate quantity that is positively correlated with the squared value of the last normalized position feature; using the matching degree weight as a scaling factor for the first intermediate quantity, weighting the first intermediate quantity, and determining an additional compensation component.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: mapping the image coordinates of the tracked target in the thermal imaging image to the display coordinate system based on the resolution ratio between the thermal imaging image and the display image to determine the initial display coordinates; correcting the initial display coordinates based on the display pixel displacement corresponding to the display position compensation amount; and rendering the projection position of the tracked target in the display image based on the corrected display coordinates.

[0013] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: linearly scaling the last normalized position feature according to the second calibration parameter to determine the basic linear compensation component.

[0014] The present invention has the following beneficial effects:

[0015] This invention achieves adaptive parallax correction between a thermal imaging system and an aerial imaging display system by constructing a complete user-interactive calibration, dynamic feature sequence construction, motion state matching, and dual-modal compensation method, without relying on precise vehicle extrinsic parameters or additional ranging sensors. It establishes a unified geometric benchmark through interactive calibration and intelligently distinguishes dangerous approach states by analyzing the morphology of the target's motion trajectory. Finally, by combining linear and nonlinear compensation mechanisms, the projected warning image can stably match the real target's spatial position across the entire distance range. This significantly improves the aftermarket system's adaptability, environmental robustness, and warning intuitiveness, thus solving the technical problem of existing solutions that struggle to achieve stable and accurate adaptive parallax correction based solely on thermal imaging video streams. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the vehicle thermal imaging intelligent perception method of a linkage aerial imaging system provided in one embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of a vehicle thermal imaging intelligent sensing device for a linkage aerial imaging system provided in an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the vehicle thermal imaging intelligent sensing method for a linked aerial imaging system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The specific solution of the vehicle thermal imaging intelligent perception method of the linkage aerial imaging system provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Please see Figure 1The diagram shows a flowchart of a vehicle thermal imaging intelligent perception method for a linkage aerial imaging system provided by an embodiment of the present invention. The method includes the following steps S101-S105, which will be described in detail below.

[0023] S101. Determine the first calibration parameter and the second calibration parameter based on user interaction operations.

[0024] The first calibration parameter is used to characterize the height of the vanishing point in the thermal imaging image; the second calibration parameter is used to characterize the mapping relationship between the pixel displacement of the thermal imaging image and the pixel displacement of the displayed image.

[0025] In one possible implementation, based on the perspective geometric relationship between the thermal imaging image in front of the vehicle and the image displayed inside the vehicle, two different types of user visual alignment operations are used to determine a first calibration parameter for representing the zero point in the vertical direction and a second calibration parameter for representing the spatial mapping ratio.

[0026] For example, an interactive visual calibration mechanism can be used: For the first calibration parameter, calibration is completed by adjusting a horizontal baseline in the user-displayed screen to visually coincide with the vanishing line of the road in the thermal imaging image; for the second calibration parameter, calibration is completed by adjusting the display parameters (such as vertical position) of a virtual marker in the user-displayed screen to spatially align the marker with the road surface position at a predetermined close distance (for example, 5-10 meters) in front of the vehicle in the thermal imaging image from the driver's perspective. The calibration result of the first calibration parameter is directly related to the vertical coordinate value of the horizontal baseline in the thermal imaging image coordinate system; the calibration result of the second calibration parameter is directly related to the adjustment gain value corresponding to the completion of spatial alignment.

[0027] S102. Based on the first calibration parameters, perform coordinate transformation on the tracked target detected in the thermal imaging video stream to construct a time series of normalized position features.

[0028] In one possible implementation, for each tracked target detected in real time in the thermal imaging video stream, its position information is standardized according to a first calibration parameter (visual horizon position), and its historical change trajectory is maintained.

[0029] For example, a dynamic feature calculation mechanism can be adopted: First, target detection is performed on each frame of thermal imaging image to locate and extract the original image coordinates of the target in the image; second, the original image coordinates are subjected to temporal smoothing filtering to suppress high-frequency jitter caused by vehicle driving bumps, and smoothed target coordinates are obtained; then, based on the relative positional relationship between the smoothed target coordinates and the first calibration parameter, and combined with the size information of the thermal imaging image, a normalized position feature characterizing the vertical sinking degree of the target in the current frame is calculated, and the feature value is mapped to a fixed numerical range (e.g., [0, 1]); finally, a fixed-capacity storage queue is maintained for each target, and the normalized position features calculated from multiple consecutive frames are stored in the queue in chronological order, thereby constructing a time series reflecting the recent motion trajectory of the target.

[0030] S103. Perform similarity matching between the time series and the preset motion trajectory template, and determine the matching weight.

[0031] In one possible implementation, the time series of normalized location features corresponding to each tracked target is compared with a predefined reference pattern that represents typical longitudinal approximation behavior, and a quantified matching weight is calculated based on the comparison results.

[0032] For example, a trajectory morphology matching decision mechanism can be adopted: First, a standard motion trajectory template is preset. This template is a data sequence whose morphology simulates the typical pattern of monotonically increasing and gradually increasing rate of change of the normalized position features of an object approaching a vehicle longitudinally from a distance at a constant speed. Second, the morphological difference degree between the current target's time series and the preset template is calculated. This difference degree is obtained through a sequence matching algorithm that allows non-linear scaling alignment of the time axis to eliminate interference caused by changes in the target's sinking rate due to the vehicle's own acceleration and deceleration. Finally, based on the calculated morphological difference degree, it is transformed into a matching degree weight value between 0 and 1 through a preset monotonically decreasing mapping relationship, where the weight value decreases as the morphological difference degree increases.

[0033] S104. Calculate the display position compensation amount based on the time series, matching degree weight, first calibration parameter and second calibration parameter.

[0034] One possible implementation involves comprehensively utilizing the target's real-time position characteristics, the evaluation weight of its motion state, and the geometric relationships calibrated by the system, and synthesizing the display position compensation amount through a dual-modal compensation strategy.

[0035] For example, a dual-modal compensation synthesis mechanism can be adopted: First, a normalized position feature value representing the latest state of the target is obtained from the time series; second, the latest normalized position feature value is linearly scaled based on a second calibration parameter to obtain a basic linear compensation component, which provides basic compensation proportional to the target's basic position and ensures virtual-real alignment in normal scenarios; simultaneously, using the matching degree weight as the modulation coefficient, an additional compensation component is determined based on the latest normalized position feature value. This component is activated and contributes compensation only when the target is detected to have a longitudinal approach posture, and its contribution value increases non-linearly with the target's approach; finally, the basic linear compensation component and the additional compensation component are combined to obtain a comprehensive compensation term, which is then combined with the scale information of the thermal imaging image to finally determine the display position compensation amount.

[0036] S105. Adjust the projection position of the tracking target on the display screen based on the display position compensation amount.

[0037] In one possible implementation, the calculated display position compensation amount is used to reverse the displacement adjustment of the theoretical projection coordinates of the tracked target in the display screen, and the aerial imaging display module is driven to render the final image.

[0038] For example, a coordinate mapping and inverse displacement mechanism can be used: First, a pixel scaling ratio is determined based on the resolution of the thermal imaging image and the display image; then, the original image coordinates of the tracked target in the thermal imaging image are linearly mapped to the display coordinate system according to this ratio to obtain a theoretical initial display coordinate without parallax correction; next, the equivalent offset at the display pixel scale is calculated based on the display position compensation amount and the pixel scaling ratio; then, the equivalent offset is subtracted from the ordinate of the initial display coordinate to obtain the corrected final rendering coordinate; finally, based on this final rendering coordinate, the warning sign of the tracked target is rendered in the display buffer and projected by the aerial imaging optical system to form a suspended real image that can be observed by the driver.

[0039] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment, by constructing a complete user interaction calibration-dynamic feature sequence construction-motion situation matching-dual-modal compensation method flow, achieves adaptive parallax correction between the thermal imaging system and the aerial imaging display system without relying on precise vehicle extrinsic parameters or additional ranging sensors. It establishes a unified geometric benchmark using interactive calibration and intelligently distinguishes dangerous approach states by analyzing the shape of the target's motion trajectory. Finally, by combining linear and nonlinear compensation mechanisms, the projected warning image can stably fit the real target's spatial position across the entire distance range, significantly improving the aftermarket system's adaptability, environmental robustness, and warning intuitiveness. This solves the technical problem of existing solutions that struggle to achieve stable and accurate adaptive parallax correction based solely on thermal imaging video streams.

[0040] In one possible implementation, the process of determining the first calibration parameter and the second calibration parameter based on user interaction can be specifically implemented through the following S201-S202, which will be explained in detail below.

[0041] S201. Determine the ordinate of the first reference figure in the thermal imaging coordinate system when it coincides with the vanishing line, and use the ordinate as the first calibration parameter.

[0042] In one possible implementation, after the user completes the alignment operation, the corresponding position of the first reference graphic (i.e., the adjustable horizontal reference line) in the thermal imaging image coordinate system is captured and recorded, and then fixed as the first calibration parameter.

[0043] For example, a coordinate capture and parameter fixing mechanism can be adopted: when the user adjusts the horizontal baseline through the interactive interface and finally confirms that it visually coincides with the vanishing line of the road in the thermal imaging image, the confirmation command is responded to. At this time, the current coordinates of the horizontal baseline in the display screen are obtained, and based on the known initial mapping relationship between the display screen and the thermal imaging image (e.g., the initial default full-screen mapping), the thermal imaging image area covered by the baseline is calculated, and then the ordinate value of its center line or specified edge in the thermal imaging image pixel coordinate system is determined; then this ordinate value is assigned to an internal system variable and stored, and this variable is defined as the first calibration parameter.

[0044] S202. Determine the adjustment gain value corresponding to the alignment of the second reference graphic with the road surface reference point, and use the adjustment gain value as the second calibration parameter.

[0045] In one possible implementation, after the user completes the alignment operation, the adjustment amount required to drive the second reference graphic (i.e., virtual identifier) ​​to achieve spatial alignment is captured and recorded, and then fixed as a second calibration parameter.

[0046] For example, a gain capture and parameter fixing mechanism can be adopted: when the user adjusts the display parameters of the virtual sign (such as its vertical position gain) through the interactive interface, and finally confirms that it achieves spatial alignment and fit with the road reference point at a preset close distance in the thermal imaging image from the driver's perspective, the confirmation command is responded to. At this time, the value corresponding to the currently adjusted control is read; then this value is assigned to an internal variable and stored, and this variable is defined as the second calibration parameter. This parameter is essentially a global scaling factor that maps the pixel displacement in the thermal imaging image to the compensation pixel displacement in the display image.

[0047] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment quantifies the user's visual alignment operation into specific calibration parameters (vertical coordinate and adjustment gain value), providing a stable and accurate static input benchmark for the entire parallax correction algorithm. By transforming the subjective visual alignment result into repeatable objective parameters, it eliminates systematic deviations caused by differences in camera installation pitch angle, position, and display optical path, ensuring that the coordinate origin and scale of all subsequent dynamic calculations are consistent, thus laying the foundation for high-precision spatial alignment of the entire solution.

[0048] In one possible implementation, the process of performing coordinate transformation on the tracking target detected in the thermal imaging video stream based on the first calibration parameter and constructing a time series of normalized position features can be specifically implemented through the following S301-S304, which will be described in detail below.

[0049] S301. Detect each frame of the thermal imaging video stream to determine the original image coordinates of the target being tracked.

[0050] In one possible implementation, the input thermal imaging video frames are analyzed in real time to locate and output the coordinates of key location points of all targets to be tracked.

[0051] For example, a frame-level target detection and feature point extraction mechanism can be adopted: receive each frame of image data from a thermal imaging camera, process the current frame using a preset target detection algorithm (e.g., a single detector based on deep learning or a segmentation algorithm based on thermal feature thresholds), identify all potential obstacle targets in the image, and generate a detection bounding box that outlines each identified target; for each such tracking target, further extract a specific feature point coordinate from its associated detection bounding box. Preferably, extract the pixel coordinates of the center of the bottom edge of the box as the original image coordinates, which represent the vertical projection position of the target's contact point with the road surface.

[0052] S302. Perform temporal filtering on the original image coordinates to determine the smoothed target coordinates.

[0053] In one possible implementation, the original image coordinates of each tracked target for multiple consecutive frames are smoothed in the temporal domain to filter out high-frequency noise components in the coordinate sequence.

[0054] For example, a sliding window averaging filtering mechanism can be used: a fixed-length historical coordinate buffer is maintained for each continuously tracked target; for the raw image coordinates of the newly acquired current frame, they are stored at the end of the buffer, while the oldest historical coordinates at the beginning of the buffer are removed to update the data; then, the arithmetic mean of all historical coordinates in the buffer is calculated, and this average is output as the smoothed target coordinates for the current frame. By adjusting the length of the buffer, the filtering effect and system response latency can be balanced.

[0055] For example, the smoothed bottom row coordinates of the i-th tracked target in the k-th frame. Satisfy the following formula 1:

[0056]

[0057] in, Let be the original bottom row coordinates (in pixels) of the i-th tracked target in frame kj. The moving average window size (positive integer, dimensionless) represents the number of consecutive frames used to calculate the average. The original coordinates of the current frame and the previous M frames are summed and then divided by M to obtain the smoothed coordinates of the current frame. This operation is a linear smoothing process designed to filter out high-frequency noise. The output result is as follows: It reflects the low-frequency trend of target position changes and eliminates the target vertical position estimate caused by high-frequency jitter of the camera viewpoint due to brief vehicle bumps.

[0058] In one possible implementation, when historical data is insufficient, the original coordinates of the current frame are used. Alternatively, the average of all currently available historical data can be used to fill the calculation window until enough data for M frames is accumulated, at which point the calculation can be switched to the standard moving average calculation mode. For example, when k=0 (the first frame), the smoothed coordinates can be directly equal to the original coordinates.

[0059] S303. Based on the target coordinates and the first calibration parameters, calculate the normalized position features of the tracking target in each frame.

[0060] In one possible implementation, the smoothed target coordinates are standardized based on the first calibration parameter (visual horizon position) to obtain a feature value that represents the relative proportion of the target's vertical position.

[0061] For example, a benchmark relativization and scale normalization mechanism can be employed: First, the difference between the smoothed target's ordinate and a first calibration parameter is calculated, representing the target's vertical drop relative to the visual horizon. Then, this difference is divided by a scale factor determined based on the total height of the thermal image and the first calibration parameter, thereby mapping the target's vertical position to a predetermined numerical range, resulting in a normalized position feature. This feature value is designed to monotonically increase as the target approaches the vehicle from a distant horizon.

[0062] S304. Generate a time series based on the normalized position features of multiple consecutive frames within a preset time period.

[0063] In one possible implementation, a fixed-capacity data storage queue is maintained for each independent tracked target. This queue operates on a first-in, first-out (FIFO) principle, caching the normalized positional feature values ​​calculated for that target in chronological order. For newly appearing targets detected for the first time, due to a lack of historical observation data, a pre-defined initialization strategy is used to fill the entire queue with its first calculated normalized positional feature value, thereby constructing an initially stable state sequence and preventing analysis logic from being interrupted due to data gaps. For existing targets that have been continuously tracked by the system, during each frame processing, the latest calculated normalized positional feature value for the current frame is inserted at the tail of the queue, while the oldest historical feature value stored at the head of the queue is simultaneously removed.

[0064] For example, a queue can be implemented by maintaining a fixed-length first-in-first-out (FIFO) buffer. The fixed length of the queue directly determines the size of the time window for observing the target's motion. The update mechanism ensures that the time series always reflects the latest motion state of the target, while outdated historical data is automatically discarded.

[0065] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment transforms unstable original detection signals into clean, comparable temporal features by performing a coherent process from original coordinate extraction, temporal smoothing to normalized feature generation and time series construction. Coordinate smoothing effectively suppresses high-frequency noise introduced by vehicle bumps, normalization processing eliminates the influence of image resolution and calibration position, and the time series provides the necessary data structure for subsequent analysis of target motion trends, thus improving the overall system's tracking stability and feature consistency for dynamic targets.

[0066] In one possible implementation, the process of calculating the normalized position features of the tracking target in each frame based on the target coordinates and the first calibration parameters can be specifically implemented through the following S401, which will be described in detail below.

[0067] S401. Based on the relative positional relationship between the ordinate of the smoothed target coordinates and the first calibration parameter, determine the normalized positional features.

[0068] In one possible implementation, normalized positional features are defined and obtained by calculating a normalized sink rate. Based on the smoothed target coordinate ordinate and the visual horizon position represented by a first calibration parameter, the vertical pixel distance between them is calculated. This distance value is then proportionally converted to a normalized reference range determined by the total image height and the first calibration parameter, thereby outputting the normalized sink rate. The first calibration parameter serves as the geometric zero-point reference for all vertical position calculations. The calculation logic ensures that the normalized sink rate is zero when the target is at or above the horizon; as the target moves closer to the bottom of the image, the value monotonically increases and approaches one. This process includes truncation of negative distances and numerical stability protection for cases where the normalized reference range may approach zero.

[0069] For example, the normalized sink rate of the i-th target in the k-th frame. The following formula 2 is satisfied:

[0070]

[0071] in, The coordinates of the bottom of the target after smoothing; The vertical coordinate of the visual horizon is obtained through user calibration. This represents the total pixel height of the thermal image. These are parameter tuning coefficients, and their values ​​should be extremely small positive numbers (e.g., 0.01) to avoid a denominator of 0. Calculate the net drop pixel distance of the target relative to the horizon. When the target is above the horizon, force it to be set to 0, which is consistent with physical intuition (no drop). This represents the maximum possible range of pixels that can sink from the horizon to the bottom of the image; Dividing the net sinking distance of the target by the entire observable sinking range yields a ratio between 0 and 1, which intuitively represents the relative proximity of the target in the vertical direction from the horizon to directly below the vehicle.

[0072] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment calculates normalized position features based on the relative positional relationship between the target and the visual horizon, creating a general index that characterizes the vertical sinking degree of the target, decoupled from absolute coordinates. This feature value is constrained to the range of 0 to 1, intuitively reflecting the continuous process of the target approaching the vehicle's underside from the distant horizon. This makes the target positions comparable under different vehicle models and installation conditions, providing standardized input for subsequent unified motion situation judgment algorithms.

[0073] In one possible implementation, the process of matching the time series with the preset motion trajectory template to determine the matching weight can be specifically implemented through the following S501-S503, which will be explained in detail below.

[0074] S501. Obtain the preset motion trajectory template.

[0075] Among them, the preset motion trajectory template is a preset data sequence that represents the normalized position feature change law when an object approaches longitudinally at a constant speed.

[0076] In one possible implementation, the preset motion trajectory template is a numerical array pre-calculated and stored during system initialization or configuration. Its generation principle is based on the reciprocal relationship in perspective projection geometry, aiming to simulate the typical shape of the normalized positional characteristics (i.e., normalized sinking rate) changing over time when a hypothetical target approaches along the vehicle's longitudinal axis at a constant speed. This template sequence typically presents as a curve that starts near zero, monotonically increases with time, and has a gradually increasing slope, thus mathematically capturing the visual physics principle that the closer the target, the faster its image position sinks.

[0077] For example, the template can be constructed based on a specific function that uses normalization time as a variable to generate the original data points. This function can produce the required monotonically increasing nonlinear curve. After generation, the original sequence is normalized so that its numerical range fits the interval between 0 and 1, to be consistent with the normalized position feature value range calculated in real time, which facilitates subsequent similarity comparison calculations.

[0078] For example, the value of the t-th point in the preset data sequence The following formula 3 is satisfied:

[0079]

[0080] in, The sequence index is used to simulate time steps; the total length of the template sequence (consistent with the feature queue length) N is the total length of the template sequence (consistent with the feature queue length). The approximation rate factor (a dimensionless constant, ranging from 0.5 to 0.9) controls the curvature of the curve; the formula is derived based on the reciprocal relationship between the object distance and the image size in perspective projection. The normalized time was simulated; as t increases from 1 to N, ( )from( ) decrease to ( ), making Monotonically increasing from values ​​close to 0; for the sequence Perform maximum and minimum value normalization on the sequence. Mapping to the [0, 1] interval yields the motion trajectory template. ; It is a numerical sequence of length N that simulates the typical shape of the target normalized sinking rate over time under ideal conditions (constant velocity longitudinal approximation) (a rising curve that is slow at first and then rapid).

[0081] S502. Determine the morphological difference between the time series and the preset motion trajectory template.

[0082] One possible implementation involves comparing a time series with a preset motion trajectory template using a sequence similarity measurement algorithm that allows for non-rigid alignment of the time axis. This algorithm does not require a strict one-to-one correspondence between the two sequences at specific time points, but rather dynamically seeks the optimal correspondence between their data points, thereby eliminating the time-dimensional distortion caused by changes in the target sinking rate due to the vehicle's own acceleration or deceleration. Through this alignment operation, the algorithm ultimately calculates a single scalar value that comprehensively reflects the overall morphological deviation between the two sequences, namely, the morphological difference degree. A larger value indicates a greater difference in shape between the two sequences; a smaller value indicates greater similarity in shape.

[0083] For example, a dynamic time warping algorithm can be used as the specific implementation of this step. First, a distance matrix of all corresponding point pairs between the two sequences is constructed. Then, a dynamic programming technique is used to search for an optimal curved path from the starting point to the ending point of the matrix, which minimizes the sum of the distances between all point pairs traversed on the path. This minimum cumulative distance value is determined as the morphological difference.

[0084] S503. Determine the matching degree weight based on the morphological difference degree and the preset mapping relationship.

[0085] In one possible implementation, a pre-defined, monotonically decreasing function (exemplarily a Gaussian kernel function) is invoked as the mapping relationship. This mapping relationship takes the calculated morphological difference as input and outputs a value between 0 and 1 as the matching weight. The mapping relationship is configured such that: when the morphological difference is large (extremely dissimilar morphology), the output weight is close to 0; when the morphological difference is small (highly similar morphology), the output weight is close to 1; in intermediate states, the weight decreases smoothly and continuously as the difference increases. This design ensures that the weight value continuously and unambiguously reflects the confidence level that the current target trajectory conforms to the danger approximation template.

[0086] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment achieves intelligent conversion from geometric position change to motion threat assessment by matching the real-time target trajectory with a preset typical approximation template and mapping the matching degree weight according to the morphological difference degree.

[0087] In one possible implementation, the process of determining the morphological difference between the time series and the preset motion trajectory template can be specifically implemented through the following S601-S602, which will be described in detail below.

[0088] S601. Based on the dynamic time warping algorithm, determine the bending path with the minimum cost that aligns the time series with the preset motion trajectory template.

[0089] One possible implementation involves first constructing a distance matrix, where each element represents the absolute difference between a data point in the time series and a data point in a pre-defined motion trajectory template; this difference represents the local cost when the two points are misaligned. Then, using dynamic programming, a continuous path from the starting point (beginning of the sequence) to the ending point (end of the sequence) is searched within this matrix. The search process must satisfy several constraints, such as the path must be continuous (allowing horizontal, vertical, or diagonal movement) and must be continuous throughout the entire sequence. The core of the algorithm is to iteratively calculate the minimum cumulative cost to reach each position in the matrix and record its origin. Finally, tracing back from the ending point, the optimal matching path that minimizes the global cumulative cost is obtained—the cost-minimizing curved path. This path allows the sequence to undergo non-uniform compression or stretching along the time axis to find the most morphologically similar segments for comparison.

[0090] S602. The cumulative distance between corresponding data points on the curved path is determined as the morphological difference degree.

[0091] In one possible implementation, after determining the curved path connecting the time series and the preset motion trajectory template using a dynamic time warping algorithm, each alignment point pair on the path is traversed. For each pair of corresponding data points on the path, the absolute difference between their values ​​is calculated as the local distance of that point pair. Subsequently, the local distances of all point pairs are summed, and the resulting sum is determined as the morphological difference degree.

[0092] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment cleverly solves the matching problem caused by the nonlinear change of the target sinking rate due to the vehicle's own acceleration and deceleration by using a dynamic time warping algorithm to calculate the morphological difference between trajectories.

[0093] In one possible implementation, the process of calculating the display position compensation based on the time series, matching degree weight, first calibration parameter and second calibration parameter can be specifically implemented through the following S701-S704, which will be described in detail below.

[0094] S701. Obtain the last normalized position feature in the time series.

[0095] In one possible implementation, a first-in-first-out queue maintained for each tracked target is accessed; this queue is a time series. The latest record at the tail of this queue data structure is read; its value is the last normalized position feature.

[0096] S702. Based on the second calibration parameter and the last normalized position feature, determine the basic linear compensation component.

[0097] In one possible implementation, the projection scaling relationship characterized by the second calibration parameter is used as a linear scaling factor to scale the last normalized position feature (i.e., the normalized sinking rate of the current frame). Specifically, the basic linear compensation component is determined as the result derived from the linear operation relationship between the second calibration parameter and the last normalized position feature. This operation aims to establish the following logic: the closer the target is to the vehicle (the larger the normalized position feature value), the greater the basic reverse displacement (basic linear compensation component) required to counteract its visual sinking should be. The second calibration parameter here determines the global slope of this linear scaling relationship.

[0098] For example, the last normalized position feature is taken as input and multiplied by the second calibration parameter, which serves as a scaling factor. The output product is then determined as the basic linear compensation component. This linearity ensures that the compensation behavior is predictable and smooth, providing continuous, abrupt baseline correction throughout the entire process of a target approaching from a distance.

[0099] S703. Using the matching degree weight as the modulation coefficient, determine the additional compensation component based on the last normalized position feature.

[0100] In one possible implementation, a nonlinear growth term is first generated based on the last normalized position feature (i.e., the normalized sink rate of the current frame). This term grows superlinearly with increasing position features. Then, a matching degree weight, representing the degree of danger approach, is used as a modulation coefficient to scale this nonlinear growth term. The final product is then determined as the additional compensation component. This design achieves fine-grained modulation: the additional component is activated only when the matching degree weight is greater than zero (identified as an approach); and the larger the weight (the higher the danger), the greater the contribution of this component to the final compensation.

[0101] For example, the nonlinear growth term can be obtained by calculating the square of the last normalized positional feature. The nonlinear growth term design of this invention aims to make its growth rate greater than the linear growth of the normalized positional feature itself. In other embodiments, a higher power (such as a cube) or other nonlinear function of the normalized positional feature can also be used to construct the nonlinear growth term to adjust the sensitivity and intensity of the dynamic enhancement effect to adapt to different alert style requirements. Based on this, a multiplication operation is performed, multiplying the matching degree weight by the squared value; the output product is the additional compensation component.

[0102] S704. Determine the display position compensation amount based on the basic linear compensation component and the additional compensation component.

[0103] In one possible implementation, a fusion operation is performed on the basic linear compensation component and the additional compensation component. The basic linear compensation component is a baseline value calculated based on the geometric calibration relationship and the real-time position of the target, ensuring basic perspective alignment; the additional compensation component is an enhancement value generated nonlinearly based on the target position and modulated by the matching degree weight, used to provide additional warning. The fusion operation combines these two to form the display position compensation amount. This process ensures that the two components work synergistically: the basic component always provides stable support, while the additional component dynamically superimposes when danger is detected, so that the final compensation amount can simultaneously meet the dual requirements of normal stability and emergency warning.

[0104] For example, the sum of the basic linear compensation component and the additional compensation component is determined as the display position compensation amount.

[0105] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment, by designing a dual-modal compensation mechanism including a basic linear compensation component and a weighted modulation additional compensation component, takes into account both stable alignment in normal scenarios and enhanced warning in dangerous scenarios. The basic component ensures that baseline correction conforming to perspective laws can be provided under any circumstances, preventing the image from floating or sinking; while the additional component is activated and superimposed only when the system confirms that danger is approaching, thereby providing additional and more conspicuous visual elevation in emergency situations, achieving a balance between safety and warning.

[0106] In one possible implementation, the process of determining the additional compensation component based on the last normalized position feature, using the matching degree weight as the modulation coefficient, can be specifically implemented through the following S801-S802, which will be explained in detail below.

[0107] S801. Determine the first intermediate value that is positively correlated with the square value of the last normalized location feature.

[0108] One possible implementation involves a non-linear operation on the last normalized position feature (i.e., the normalized sink rate of the current frame). The squared correlation indicates that the value of this first intermediate quantity increases with the increase of the last normalized position feature value, and the growth rate exceeds a linear relationship. This non-linear mapping is designed to capture and amplify the state changes of a target when it is very close to a vehicle. Physically, the normalized position feature itself characterizes the relative distance of the target (the larger the value, the closer the distance), while its squared value further amplifies the numerical difference at extremely close range, making it more sensitive to the state of a target entering a high-risk proximity area.

[0109] For example, this step can be implemented using a computational unit whose input is the last normalized location feature, and whose output is a value proportional to the square of the input value; this output is the first intermediate quantity. This computational relationship essentially transforms a linear indicator representing the proximity of a target into a non-linear indicator that better matches human perception of urgency.

[0110] S802. Use the matching degree weight as a scaling factor for the first intermediate quantity, weight the first intermediate quantity, and determine the additional compensation component.

[0111] In one possible implementation, a scaling operation is performed. The matching degree weight serves as the scaling factor (or modulation coefficient) for this operation, and the first intermediate quantity is the object being scaled. The output of the operation is the additional compensation component. The technical logic is as follows: when the matching degree weight approaches 0 (indicating that the target is not in a dangerous approaching state), regardless of the value of the first intermediate quantity, the result of the scaling operation (additional compensation component) is suppressed to near 0, and the dynamic enhancement compensation function is essentially turned off; when the matching degree weight is greater than 0 (indicating that the target is identified as having a longitudinal approaching state), the scaling operation is activated, and the output value is proportional to the magnitude of the weight. The larger the weight, the higher the danger, and the more significant the enhancement compensation applied to the same close-range state (first intermediate quantity).

[0112] For example, multiplying the matching degree weight by the first intermediate quantity, the resulting product is determined as the additional compensation component. This multiplicative relationship clearly and intuitively demonstrates the dual on / off and gain adjustment effect of the matching degree weight on the nonlinear enhancement term, serving as a mathematical bridge connecting situational awareness and compensation execution.

[0113] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment achieves a fine correlation between the degree of danger and the intensity of compensation by specifically designing the additional compensation component as the product of the matching degree weight and a nonlinear growth term based on normalized position features. The nonlinear growth term ensures that the compensation amount will increase sharply when the target is very close, producing a strong visual warning effect; while the matching degree weight, as a modulation factor, ensures that this enhancement effect is precisely triggered only for high-threat targets, avoiding false alarms and disturbing the public, and significantly improving the intelligence and effectiveness of the warning.

[0114] In one possible implementation, the process of adjusting the projection position of the tracking target on the display screen based on the display position compensation amount can be specifically implemented through the following S901-S903, which will be described in detail below.

[0115] S901. Based on the resolution ratio between the thermal imaging image and the display image, the image coordinates of the tracked target in the thermal imaging image are mapped to the display coordinate system to determine the initial display coordinates.

[0116] In one possible implementation, the vertical resolution of the thermal imaging camera and the vertical resolution of the aerial imaging display module are first obtained, and their ratio is calculated as a resolution scaling factor. Then, the x and y coordinates of the target's image coordinates in the thermal image (usually the center point coordinates of its detection box) are multiplied by the resolution scaling factor. This multiplication operation achieves linear scaling of the coordinates, and the scaling result is the target's initial display coordinates in the display coordinate system. This process assumes that the origins of the two coordinate systems (usually the upper left corner of the image) are aligned, and that the only difference is the resolution scaling.

[0117] S902. Correct the initial display coordinates based on the display pixel displacement corresponding to the display position compensation amount.

[0118] In one possible implementation, the display pixel displacement is first calculated based on the display position compensation and resolution scaling factor. Specifically, multiplying the display position compensation by the resolution scaling factor yields the pixel-level displacement value to be applied in the display coordinate system. Then, a subtraction operation is performed on the initial display coordinate's ordinate, subtracting the display pixel displacement from the initial ordinate. This subtraction achieves a reverse coordinate displacement: because the target in the thermal imaging image is sinking (ordinate value increases), to visually elevate its projected image, its position in the display coordinate system must be adjusted upwards (ordinate value decreases). The coordinates obtained through this calculation are the corrected display coordinates.

[0119] S903. Based on the corrected display coordinates, render the projection position of the tracked target in the display screen.

[0120] In one possible implementation, the corrected display coordinates are passed to the graphics rendering engine of the aerial imaging display module. Using these coordinates as a positioning reference, the engine draws a warning graphic representing the tracked target at the corresponding position in the display buffer, such as a highlighted box, icon, or indicator. The display module then projects the complete image containing this warning graphic through its specific optical system (such as a set of lenses and mirrors), forming a suspended virtual or real image near the windshield in front of the driver or at a specific location. This rendering and projection process ultimately makes the corrected digital coordinates appear as a visually perceptible warning sign in physical space.

[0121] For example, the rendering process is driven by the onboard computing unit to complete the display controller. The corrected display coordinates determine the pixel position of the warning graphic on the display panel. The light emitted from the display panel is modulated through an optical path, ultimately forming a floating image within the driver's field of vision that appears to be located on the road in front of the vehicle and overlaps with the real target.

[0122] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment performs a precise spatial mapping from thermal imaging coordinates to display coordinates, and performs inverse subtraction correction on the initial projection position by combining the calculated pixel offset, thus fully realizing the closed loop from digital compensation amount to physical optical path adjustment.

[0123] In one possible implementation, the process of determining the basic linear compensation component based on the second calibration parameter and the last normalized position feature can be specifically implemented through the following S1001, which will be described in detail below.

[0124] S1001. Based on the second calibration parameters, the last normalized position feature is linearly scaled to determine the basic linear compensation component.

[0125] In one possible implementation, the second calibration parameter is used as the linear scaling factor. The last normalized position feature, i.e., the scalar value representing the current relative proximity of the target obtained by calculating the normalized subsidence rate, is the object to be scaled. Performing linear scaling involves calculating these two parameters according to a preset proportional relationship, and the output is the basic linear compensation component. The second calibration parameter here defines the amount of basic compensation change that should be caused by a unit change in the normalized position feature; the larger its value, the greater the basic lift compensation applied by the system to targets with the same proximity.

[0126] For example, this linear scaling can be achieved through a multiplication operation. Multiplying the last normalized position feature (with a value between 0 and 1) by the second calibration parameter (a dimensionless scaling factor) yields a product that is determined as the basic linear compensation component. This directly reflects the most basic compensation required to counteract the visual drop of the target caused by the basic perspective projection, given the established system calibration.

[0127] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment, by defining the basic linear compensation component as a linear scaling of the normalized position features by the second calibration parameter, reveals that the essence of basic compensation is to perform proportional reverse compensation for target sinking based on the proportional relationship calibrated by the system. This design ensures the predictability and smoothness of the compensation behavior, provides a stable and reliable benchmark for the entire dual-modal compensation system, and enables the system to maintain excellent parallax correction performance under non-hazardous conventional working conditions.

[0128] Please see Figure 2 The diagram illustrates a structural schematic of a vehicle-mounted thermal imaging intelligent sensing device 200 for a linked aerial imaging system according to an embodiment of the present invention. The device includes: a processing unit 201; the processing unit 201 is configured to determine a first calibration parameter and a second calibration parameter based on user interaction operations; the user interaction operations include: adjusting the display position of a first reference graphic in the display screen until the first reference graphic visually coincides with the vanishing line of the road in the thermal imaging image; adjusting the display parameters of a second reference graphic in the display screen until the second reference graphic, from the driver's perspective, spatially aligns with the road surface reference point in the thermal imaging image. The first calibration parameter is used to characterize the height of the vanishing point in the thermal imaging image; the second calibration parameter is used to characterize the mapping relationship between the pixel displacement of the thermal imaging image and the pixel displacement of the display image; based on the first calibration parameter, the coordinate transformation of the tracked target detected in the thermal imaging video stream is performed to construct a time series of normalized position features; the time series is matched with a preset motion trajectory template to determine the matching degree weight; based on the time series, matching degree weight, first calibration parameter, and second calibration parameter, the display position compensation amount is calculated; the projection position of the tracked target in the display image is adjusted based on the display position compensation amount.

[0129] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A vehicle-mounted thermal imaging intelligent sensing method linked to an aerial imaging system, characterized in that, The method includes: The first calibration parameter and the second calibration parameter are determined based on user interaction operations. The user interaction operations include: adjusting the display position of the first reference graphic in the display screen until the first reference graphic visually coincides with the vanishing line of the road in the thermal imaging image; adjusting the display parameters of the second reference graphic in the display screen until the second reference graphic is spatially aligned with the road reference point in the thermal imaging image from the driver's perspective; the first calibration parameter is used to characterize the height of the vanishing point in the thermal imaging image; the second calibration parameter is used to characterize the mapping relationship between the pixel displacement of the thermal imaging image and the pixel displacement of the display screen; determining the first calibration parameter includes: determining the ordinate in the coordinate system of the thermal imaging image when the first reference graphic coincides with the vanishing line, and using the ordinate as the first calibration parameter; determining the second parameter includes: determining the adjustment gain value corresponding to the alignment of the second reference graphic with the road reference point, and using the adjustment gain value as the second calibration parameter. Based on the first calibration parameters, coordinate transformation is performed on the tracked target detected in the thermal imaging video stream to construct a time series of normalized position features; The time series is matched with a preset motion trajectory template to determine the matching weight; Obtain the last normalized position feature in the time series; Based on the second calibration parameters and the last normalized position feature, the basic linear compensation component is determined; Determine a first intermediate quantity that is positively correlated with the squared value of the last normalized location feature; The matching degree weight is used as a scaling factor for the first intermediate quantity, and the first intermediate quantity is weighted to determine the additional compensation component. The display position compensation amount is determined based on the basic linear compensation component and the additional compensation component; the projection position of the tracking target in the display screen is adjusted based on the display position compensation amount.

2. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 1, characterized in that, The step of performing coordinate transformation on the tracked target detected in the thermal imaging video stream based on the first calibration parameters to construct a time series of normalized position features includes: Each frame of the thermal imaging video stream is detected to determine the original image coordinates of the tracked target; The original image coordinates are subjected to temporal filtering to determine the smoothed target coordinates; Based on the target coordinates and the first calibration parameters, the normalized position features of the tracked target in each frame are calculated; The time series is generated based on the normalized positional features of multiple consecutive frames within a preset time period.

3. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 2, characterized in that, The step of calculating the normalized position features of the tracked target in each frame based on the target coordinates and the first calibration parameters includes: Based on the relative positional relationship between the ordinate of the smoothed target coordinates and the first calibration parameter, the normalized positional feature is determined; the normalized positional feature is used to characterize the relative proportion of the tracked target in the vertical direction between the first calibration parameter and the bottom of the image.

4. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 1, characterized in that, The step of performing similarity matching between the time series and the preset motion trajectory template to determine the matching weight includes: Obtain the preset motion trajectory template; the preset motion trajectory template is a preset data sequence that characterizes the normalized position feature change law when an object approaches longitudinally at a constant speed; Determine the morphological difference between the time series and the preset motion trajectory template; The matching degree weight is determined based on the morphological difference degree and a preset mapping relationship; the matching degree weight decreases as the morphological difference degree increases.

5. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 4, characterized in that, Determining the morphological difference between the time series and the preset motion trajectory template includes: Based on the dynamic time warping algorithm, the curved path with the minimum cost of aligning the time series with the preset motion trajectory template is determined; The cumulative distance between corresponding data points on the curved path is determined as the morphological difference degree.

6. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 1, characterized in that, Adjusting the projection position of the tracking target on the display screen based on the display position compensation amount includes: Based on the resolution ratio between the thermal imaging image and the display image, the image coordinates of the tracked target in the thermal imaging image are mapped to the display coordinate system to determine the initial display coordinates; The initial display coordinates are corrected based on the display pixel displacement corresponding to the display position compensation amount. Based on the corrected display coordinates, the projected position of the tracked target is rendered in the display screen.

7. The vehicle thermal imaging intelligent sensing method for a linked aerial imaging system according to claim 1, characterized in that, The step of determining the basic linear compensation component based on the second calibration parameter and the last normalized position feature includes: The last normalized position feature is linearly scaled according to the second calibration parameter to determine the basic linear compensation component.

Images