Multi-spectral video dynamic registration method and system of refrigeration type infrared imager

Abstract

The invention relates to a multispectral video dynamic registration method and system of a refrigeration type infrared imager. The method comprises the following steps: acquiring a multispectral video, extracting spectral feature data by adopting a spectral band separation algorithm, constructing a feature retrieval framework in combination with a time sequence information integration and spatial information extraction technology, and retrieving to obtain an initial spectral feature positioning result; performing feature enhancement and dimension reduction processing on the initial result by using a multi-dimensional information fusion technology, then recombining the initial result into a structured feature data set, acquiring real-time environment change data, registering the data set by using a preset dynamic registration model based on the real-time environment change data to generate a preliminary registration result, and calculating a registration error of the result; and adaptively updating the dynamic registration model by means of a preset closed-loop feedback control mechanism until a stable registration result is obtained. According to the method, through multi-dimensional feature processing and a closed-loop feedback mechanism, dynamic and accurate registration of a multispectral video is realized, and the registration stability and precision in a complex environment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of photoelectric imaging, and in particular relates to a multi-spectral video dynamic registration method and system for a refrigerated infrared imager. Background Art

[0002] With the development of optoelectronic imaging technology, multispectral video dynamic registration technology has been widely used in cooled infrared imagers. This technology improves imaging accuracy and environmental adaptability by integrating multispectral information. Traditional multispectral video registration methods usually use a fixed-parameter registration model, combined with simple feature extraction technology to locate and integrate spectral features, and lack a dynamic response mechanism to real-time environmental changes and equipment motion parameters. However, in the existing technology, due to the lack of a closed-loop feedback control mechanism for adaptive updates and the lack of multi-dimensional information fusion and dynamic error compensation in the feature processing process, the registration error is large and the stability is insufficient in complex environments. In particular, it is difficult to effectively handle motion distortion in equipment motion scenes, and it cannot meet the dynamic and accurate registration requirements of high-precision infrared imaging. Summary of the Invention

[0003] Based on this, it is necessary to provide a multispectral video dynamic registration method and system for a cooled infrared imager that can solve the above problems.

[0004] In a first aspect, the present application provides a multispectral video dynamic registration method for a cooled infrared imager, comprising:

[0005] Acquire multispectral video and extract spectral feature data from the multispectral video using a multispectral decomposition algorithm based on wavelet transform;

[0006] Based on the spectral feature data, a feature retrieval framework is constructed by combining temporal information integration and spatial information extraction technology, and the initial spectral feature positioning results are retrieved using the feature retrieval framework;

[0007] The multi-dimensional information fusion technology is used to perform feature enhancement and dimensionality reduction processing on the initial spectral feature positioning results, and the processing results are reorganized into a structured feature data set to generate a spectral feature data set;

[0008] Acquire real-time environmental change data, and based on the real-time environmental change data, use the preset dynamic registration model to register the spectral feature data set to generate preliminary registration results;

[0009] The registration error of the preliminary registration result is calculated, and based on the registration error, the dynamic registration model is adaptively updated using a preset closed-loop feedback control mechanism until a stable registration result is obtained.

[0010] In one embodiment, the method further includes reversely optimizing the feature retrieval framework based on the registration error of the preliminary registration result by the following steps:

[0011] The correlation coefficient analysis method is used to analyze the relationship between the registration error and the feature matching results in the feature retrieval framework;

[0012] According to the correlation relationship, adjust the weight coefficients of temporal information integration technology and spatial information extraction technology;

[0013] The feature retrieval framework is updated using the adjusted weight parameters to obtain an optimized feature retrieval framework.

[0014] In one embodiment, the method further includes reversely optimizing the initial spectral feature location results based on the optimized feature retrieval framework by the following steps:

[0015] According to the difference in weight parameters between the feature retrieval framework and the optimized feature retrieval framework, a spectral feature positioning update mechanism is constructed;

[0016] Using the spectral feature positioning update mechanism to update the initial spectral feature positioning result to obtain an updated spectral feature positioning result;

[0017] An adaptive weighted fusion strategy is used to integrate the updated spectral feature positioning results and the initial spectral feature positioning results to generate optimized spectral feature positioning results.

[0018] In one embodiment, the method further includes reconstructing the spectral feature dataset based on the optimized spectral feature positioning result by the following steps:

[0019] The optimized spectral feature positioning results are used to resample the regional features of the multispectral video to generate a reconstructed spectral feature dataset;

[0020] The spatial and temporal consistency constraint algorithm is used to align and correct the reconstructed spectral features with the spectral feature dataset;

[0021] The corrected reconstructed spectral feature dataset is integrated into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset.

[0022] In one embodiment, the corrected reconstructed spectral feature dataset is integrated into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset, which is achieved by the following formula:

[0023]

[0024] Among them, F opt For the optimized spectral feature dataset, To reconstruct the spectral feature dataset, is the spectral feature dataset, m is the number of reconstructed spectral feature frames, n is the number of spectral feature dataset frames, d r and d are the dimensions of the reconstructed spectral feature dataset and the spectral feature dataset, respectively; For the channel alignment weight matrix, by minimizing The solution is that Concat(·) is a cascade operation along the feature channel dimension.

[0025] In one embodiment, the method further comprises:

[0026] Obtain real-time motion parameters; motion parameters include the imager's three-dimensional angular velocity, acceleration, and geographic location coordinates;

[0027] A spatiotemporal motion compensation model is constructed based on motion parameters, and motion distortion correction is performed on the stable registration result through the spatiotemporal motion compensation model to generate a motion compensated registration result;

[0028] A convolutional neural network is used to perform feature mapping learning on the historical data of registration errors and the corresponding motion parameters to generate a motion-error prediction model;

[0029] According to the current motion parameters, the motion-error prediction model is used to pre-compensate and optimize the motion-compensated registration results to generate the final registration results.

[0030] In one embodiment, the step of training the motion-error prediction model includes:

[0031] The training set is constructed using the historical data of registration errors and the corresponding motion parameters;

[0032] A two-branch convolutional neural network is constructed based on the training set. The first branch of the two-branch convolutional neural network is used to process the motion parameter tensor, and the second branch is used to process the temporal features of the registration error and generate the initial prediction error through the feature fusion layer.

[0033] Dynamically adjust the sample weight coefficient of the training set according to the acceleration amplitude in the motion parameters to generate weighted training samples;

[0034] Random angular velocity noise is injected into the weighted training samples, and the corresponding parameters of the two-branch convolutional neural network are iteratively updated through the back-propagation algorithm until the initial prediction error drops to a preset threshold.

[0035] In a second aspect, the present application also provides a multispectral video dynamic registration system for a cooled infrared imager, comprising:

[0036] Spectral feature extraction module, used to obtain multispectral video and extract spectral feature data from the multispectral video using a multispectral decomposition algorithm based on wavelet transform;

[0037] The retrieval framework modeling module is configured to construct a feature retrieval framework by integrating spatial information extraction technology based on spectral feature data combined with timing information, and to retrieve an initial spectral feature positioning result by using the feature retrieval framework.

[0038] The information fusion processing module is configured to perform feature enhancement and dimension reduction processing on the initial spectral feature positioning result by using a multi-dimensional information fusion technology, and to reorganize the processing result into a structured feature data set to generate a spectral feature data set.

[0039] The registration model execution module is configured to obtain real-time environmental change data, and to perform registration on the spectral feature data set by using a preset dynamic registration model based on the real-time environmental change data to generate a preliminary registration result.

[0040] The closed-loop feedback optimization module is configured to calculate a registration error of the preliminary registration result, and to perform adaptive updating on the dynamic registration model by using a preset closed-loop feedback control mechanism based on the registration error until a stable registration result is obtained.

[0041] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned multi-spectral video dynamic registration method of the refrigeration-type infrared imager when executing the computer program.

[0042] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the above-mentioned multi-spectral video dynamic registration method of the refrigeration-type infrared imager.

[0043] The above-mentioned multi-spectral video dynamic registration method of the refrigeration-type infrared imager and system, computer device and storage medium can extract spectral feature data by using a spectral band separation algorithm, construct a feature retrieval framework by combining timing and spatial information, realize multi-dimensional feature integration and accurate positioning, perform feature enhancement and dimension reduction processing by using a multi-dimensional information fusion technology to improve data representation capability, generate a preliminary registration result by using a dynamic registration model based on real-time environmental change data, and adaptively update model parameters according to a registration error by means of a closed-loop feedback control mechanism, which can solve the problem of lack of dynamic response of a traditional fixed parameter registration model, can adjust the registration strategy in real time in a complex environment, reduce the registration error, and improve the stability of the registration result. At the same time, by dynamically processing real-time environmental data, the compensation capability for motion distortion in a device motion scene is enhanced, and the real-time registration requirement of higher precision infrared imaging is met. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of a multispectral video dynamic registration method for a cooled infrared imager of the present invention;

[0046] Figure 2 This is a structural diagram of a multi-spectral video dynamic registration system of a cooled infrared imager of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] In one embodiment, Figure 1 As shown, a method for dynamic multispectral video registration using a cooled infrared imager is provided. This embodiment illustrates this method using a computing terminal device equipped with a cooled infrared imager. The device integrates hardware functional units such as a spectral feature extraction module and a closed-loop feedback optimization module. The processor executes the multispectral decomposition and dynamic registration algorithm, and the memory stores the feature dataset and registration model parameters. It is understood that this method can also be applied to a server with computing capabilities, or to a collaborative system consisting of a terminal and server. The terminal captures multispectral video in real time and transmits it to the server. The server completes iterative optimization of the registration model through multidimensional information fusion and a closed-loop feedback mechanism, and then transmits the stable registration results back to the terminal. This method is suitable for high-precision real-time registration requirements in scenarios such as military reconnaissance and industrial inspection, where equipment is in motion or the environment undergoes drastic changes.

[0049] In this embodiment, the method includes the following steps:

[0050] S01, obtain a multispectral video, and use a multispectral decomposition algorithm based on wavelet transform to extract spectral feature data from the multispectral video.

[0051] Among them, multispectral video (continuous image sequence data containing multiple infrared bands collected by a cooled infrared imager, each spectral band represents different wavelength information to capture the changes in the spectral characteristics of the target) is obtained; a multispectral decomposition algorithm based on wavelet transform can be used to process the video, and the video frame is decomposed into sub-bands of multiple scales and directions through wavelet transform (a time-frequency analysis method) to achieve spectral band separation and feature extraction: discrete wavelet transform (such as Daubechies wavelet basis) is applied to the video frame to obtain low-frequency approximate components and high-frequency detail components, and combined with multi-scale analysis to filter noise and enhance spectral features, generate spectral feature data (a structured data set containing spectral band intensity, frequency distribution and spatial position information) as the basic input for subsequent feature retrieval and registration, and support subsequent dynamic registration optimization by converting the original video data into a high-dimensional feature representation.

[0052] S02, based on the spectral feature data, combines the temporal information integration and spatial information extraction technology to build a feature retrieval framework, and uses the feature retrieval framework to retrieve the initial spectral feature positioning results.

[0053] Among them, the temporal information integration technology can perform temporal correlation analysis on the spectral features of continuous video frames through a sliding window mechanism (including calculating the feature similarity between frames and modeling the motion trajectory); the spatial information extraction technology can use convolution kernel operations to perform local feature response enhancement and boundary enhancement operations on the spatial distribution of the spectral features of a single frame; and a feature retrieval framework is constructed by weighted fusion of the temporal correlation matrix and the spatial feature map (a searchable data structure composed of spatial-temporal dual-channel feature maps, in which the spatial channel stores the gradient features processed by Gaussian filtering, and the temporal channel stores the inter-frame matching path generated by the dynamic time warping (DTW) algorithm). The optical flow method can be used to calculate the spectral feature displacement vectors of adjacent frames to construct a temporal correlation model. At the same time, the Sobel operator is used to extract the spatial edge features of each frame, and a fused feature map is generated through adaptive weight allocation. The coordinates of spectral feature points are retrieved on the fused feature map using the K nearest neighbor algorithm, and the initial spectral feature positioning results (including the spatial coordinates of the feature points and their confidence) are output. The above steps realize the transformation from multi-dimensional feature analysis to quantifiable positioning results.

[0054] S03, using multi-dimensional information fusion technology to perform feature enhancement and dimensionality reduction processing on the initial spectral feature positioning results, and reorganizing the processing results into a structured feature data set to generate a spectral feature data set.

[0055] Among them, multidimensional information fusion technology (which can be a method combining principal component analysis (PCA) and feature pyramid network (FPN)) realizes information complementarity through cross-dimensional feature interaction; feature enhancement operations on positioning results include: adaptive weighting processing of feature points based on confidence weights, dimensionality reduction processing of weighted high-dimensional features using PCA algorithm, and reducing the feature dimension from the original d dimension to k dimension by calculating the eigenvectors of the covariance matrix and retaining the first k principal components; through spatial grid reorganization, the reduced-dimensional features are reorganized according to the spatial topological structure of the video frame to generate a spectral feature data set with a three-dimensional tensor structure. Through this step, feature compression of more than 97% of the original information can be retained, providing an efficient data basis for subsequent dynamic alignment.

[0056] S04, acquiring real-time environmental change data, and based on the real-time environmental change data, registering the spectral feature data set using a preset dynamic registration model to generate a preliminary registration result.

[0057] Among them, the real-time environmental change data (dynamic parameters of temperature gradient, atmospheric turbulence intensity and light intensity collected by a multi-sensor fusion system) is defined in the data format of a four-dimensional tensor [B×S×T×C] (B is the sensor batch, S is the spatial resolution, T is the time series, and C is the environmental parameter channel); the preset dynamic registration model (using a differentiable Spatial Transformer Network architecture and a learnable affine transformation parameter matrix to achieve geometric deformation modeling) cross-modally fuses the environmental change data with the spectral feature dataset through an attention gating mechanism to generate an environmental perception feature map and input it into the dynamic registration model for multi-scale feature alignment. A bilinear sampler is used in the hidden layer of the model to spatially transform the feature map. Inter-frame registration is achieved by minimizing the spectral correlation distance between feature points in adjacent frames. The output is a preliminary registration result containing the coordinates of the registered feature points and their transformation parameters. Through the collaborative mechanism of environmental perception and parameterized spatial transformation, robust registration under complex interference is achieved.

[0058] S05, calculating the registration error of the preliminary registration result, and based on the registration error, adaptively updating the dynamic registration model using a preset closed-loop feedback control mechanism until a stable registration result is obtained.

[0059] The registration error can be quantified using the root mean square error method by calculating the positional deviation between the initial registration result and the preset ideal reference position. A preset closed-loop feedback control mechanism is a closed-loop system based on the proportional-integral-derivative control principle. The internal parameters of the dynamic registration model are dynamically adjusted through real-time error feedback signals. The calculated registration error is input into the closed-loop system, and the error value is automatically compared with a preset threshold (e.g., 0.1 pixel unit) to determine the need to update the dynamic registration model. If the error exceeds the threshold, the adjustment direction of the dynamic registration model parameters is calculated using a gradient optimization algorithm. The update step size is controlled by the learning rate, and the corresponding weights of the dynamic registration model are iteratively corrected. The optimization cycle continues and the error is re-evaluated after each update until the error value drops below the threshold and the error fluctuation over multiple consecutive iterations is extremely small (e.g., less than 0.01 pixel unit). A stable registration result (error convergence and parameter stability) is output, and its feature point confidence is increased to above 0.95. The closed-loop feedback mechanism transforms the registration error into the core driving force of model self-optimization, achieving adaptive evolution from the initial registration state to high precision and high stability.

[0060] The multispectral video dynamic registration method for a cooled infrared imager achieves spectral feature separation by acquiring multispectral video and extracting spectral feature data using a wavelet transform-based multispectral decomposition algorithm. A feature retrieval framework is constructed based on the spectral feature data, combined with temporal information integration and spatial information extraction techniques, to retrieve initial spectral feature positioning results, enhancing feature consistency and positioning accuracy in both temporal and spatial dimensions. Multidimensional information fusion technology is used to perform feature enhancement and dimensionality reduction on the initial spectral feature positioning results, and the processed results are reorganized into a structured feature dataset to optimize data representation efficiency and compression rate. Real-time environmental change data is acquired and registered with the spectral feature dataset using a preset dynamic registration model to generate a preliminary registration result, achieving real-time response to dynamic environmental changes and adaptive geometric transformation. The registration error of the preliminary registration result is calculated and, based on the error, the dynamic registration model is adaptively updated using a closed-loop feedback control mechanism until a stable registration result is obtained. This method addresses the large registration error and insufficient stability caused by the lack of a dynamic response mechanism in fixed parameter models, improves registration accuracy and robustness in complex environments, and meets the real-time requirements of high-precision infrared imaging.

[0061] In one embodiment, the method further includes reversely optimizing the feature retrieval framework based on the registration error of the preliminary registration result by the following steps:

[0062] S11, using correlation coefficient analysis method to analyze the correlation between registration error and feature matching results in feature retrieval framework;

[0063] S12, adjusting the weight coefficients of the temporal information integration technology and the spatial information extraction technology according to the correlation relationship;

[0064] S13, using the adjusted weight parameters to update the feature retrieval framework to obtain an optimized feature retrieval framework.

[0065] Specifically, the correlation coefficient analysis method can be used to analyze the correlation between the registration error and the feature matching results in the feature retrieval framework (the spectral feature point coordinates and their confidence levels output by the feature retrieval framework) (which can be expressed by the Pearson correlation coefficient) to quantify the linear dependence strength between the two; by constructing an error-matching matrix and applying least squares fitting to calculate the correlation coefficient value, when the absolute value of the correlation coefficient exceeds 0.5, it is determined that there is a significant correlation, and the main source of the error (matching deviation of temporal or spatial features) is identified. According to the correlation relationship, the weight coefficients of the temporal information integration technology and the spatial information extraction technology are adjusted (the contribution ratio of temporal information integration (such as the inter-frame displacement vector generated by the optical flow method) and spatial information extraction (such as the edge feature map generated by the Sobel operator) in the feature retrieval framework); based on the correlation coefficient analysis results, if the error is strongly correlated with the temporal sequence, the temporal weight coefficient is increased (such as from 0.6 to 0.8), while the spatial weight coefficient is reduced, and the weight value is optimized by the gradient descent algorithm to ensure the balance of feature fusion. The feature retrieval framework is updated using the adjusted weight parameters to obtain an optimized feature retrieval framework. This framework reconstructs the feature mapping channels and applies the new weights to the spatial-temporal dual-channel feature maps, such as updating the DTW path weights of the temporal channel and the Gaussian filter parameters of the spatial channel, to generate a more robust data structure. Driven by error feedback, the framework's adaptability is improved by dynamically adjusting the feature weights, reducing registration instability and enhancing positioning accuracy in complex environments.

[0066] In one embodiment, the method further includes reversely optimizing the initial spectral feature location results based on the optimized feature retrieval framework by the following steps:

[0067] S21, constructing a spectral feature positioning update mechanism based on the difference in weight parameters between the feature retrieval framework and the optimized feature retrieval framework;

[0068] S22, using a spectral feature positioning update mechanism to update the initial spectral feature positioning result to obtain an updated spectral feature positioning result;

[0069] S23, adopting an adaptive weighted fusion strategy to integrate the updated spectral feature positioning result and the initial spectral feature positioning result to generate an optimized spectral feature positioning result.

[0070] Exemplarily, the weight parameter difference (the change in weight value of the temporal information integration technology (such as the inter-frame matching path generated by the dynamic time warping algorithm) and the spatial information extraction technology (such as the spatial feature map generated by the Sobel edge detection operator) in the feature retrieval framework before and after optimization, is quantified by calculating the Euclidean norm matrix to quantify the contribution change trend of the temporal and spatial features); the spectral feature positioning update mechanism (specifically implemented as follows: based on the difference matrix, a correction vector of the feature point coordinates is derived, where the correction direction is determined by the weight change gradient, such as when the spatial weight increases, the feature point position shifts toward the edge feature significant area, thereby dynamically adjusting the positioning deviation) uses the spectral feature positioning update mechanism to update the initial spectral feature positioning result to obtain an updated spectral feature positioning result: the initial positioning result is updated The feature point coordinate input update mechanism applies the calculated correction offset (such as adjusting it within the range of 0.05 to 0.2 based on the learning rate factor) to generate an updated spectral feature positioning result; combined with the feature confidence (exponential decay calculation of the offset amplitude (for example, the confidence decreases when the offset is high)), it is processed to ensure that the updated positioning result contains more accurate spatial position information and reliability assessment: an adaptive weighted fusion strategy is adopted (fusion weights are assigned according to the confidence ratio of the updated result to the initial result, and the final coordinates are calculated by weighted average (such as the updated coordinate weight is dynamically adjusted based on the confidence ratio), and the maximum confidence value is selected as the output) to integrate the updated spectral feature positioning result and the initial spectral feature positioning result to generate an optimized spectral feature positioning result. The fusion process eliminates local positioning noise and improves overall consistency. The weight difference is used to drive the iterative optimization of the coordinates. The confidence-weighted fusion mechanism solves the problem of feature drift in complex environments.

[0071] In one embodiment, the method further includes reconstructing the spectral feature dataset based on the optimized spectral feature positioning result by the following steps:

[0072] S31, using the optimized spectral feature positioning results to perform regional feature resampling on the multispectral video to generate a reconstructed spectral feature dataset;

[0073] S32, using a spatiotemporal consistency constraint algorithm to align and correct the reconstructed spectral features with the spectral feature dataset;

[0074] S33, integrating the corrected reconstructed spectral feature dataset into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset.

[0075] Specifically, the optimized spectral feature positioning result (a set of high-precision feature point coordinates generated after confidence-weighted fusion) can be used to perform regional feature resampling on the multispectral video to generate a reconstructed spectral feature dataset: with the feature point in the optimized positioning result as the center, a sampling window of adaptive size is constructed (the window size is positively correlated with the feature confidence), and the spectral band intensity values ​​of the corresponding area in the multispectral video are extracted by bilinear interpolation to form a reconstructed dataset containing spatial enhancement features; a spatiotemporal consistency constraint algorithm (a joint optimization method based on the continuity of feature motion trajectories and the similarity of spectral band distribution) is used to align and correct the reconstructed spectral features with the original spectral feature dataset: the residual of the optical flow displacement vector of the reconstructed features and the original features in the time dimension is calculated (the residual norm is required to be less than 0.05), and the objective function is constructed in combination with the spectral correlation coefficient (the threshold is set to 0.85), and the correction transformation matrix is ​​generated by gradient descent iteration; the corrected reconstructed spectral feature dataset is integrated into the spectral feature dataset by feature cascade fusion to generate an optimized spectral feature dataset. Dimension unification is achieved through the channel alignment weight matrix (obtained by minimizing the Frobenius norm of the reconstructed features and the original features), and cascade operations are performed according to the feature channel dimension: the original dataset and the new reconstructed dataset are projected by the weight matrix, and then spliced ​​into a high-dimensional feature tensor along the channel dimension. The resampling is driven by the positioning results, and data consistency is ensured by spatiotemporal constraints.

[0076] In one embodiment, S41, integrating the corrected reconstructed spectral feature dataset into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset is achieved by the following formula:

[0077]

[0078] Among them, F opt For the optimized spectral feature dataset, To reconstruct the spectral feature dataset, is the spectral feature dataset, m is the number of reconstructed spectral feature frames, n is the number of spectral feature dataset frames, d r and d are the dimensions of the reconstructed spectral feature dataset and the spectral feature dataset, respectively; For the channel alignment weight matrix, by minimizing The solution is that Concat(·) is a cascade operation along the feature channel dimension.

[0079] For example, the optimized spectral feature dataset F opt The spectral feature dataset is now processed by the concatenation operation Concat(·) along the feature channel dimension. and reconstructed spectral feature datasets The fusion integration of the multi-spectral video is generated by resampling the regional features based on the optimized spectral feature positioning results and corrected by the spatiotemporal consistency constraint algorithm, and the channel alignment weight matrix is ​​introduced. (Singular Value Decomposition (SVD) algorithm can be used to minimize By aligning the spectral feature dataset and the reconstructed dataset after projection by the weight matrix The feature channel dimensions are cascaded to form a structured dataset containing richer spectral feature information, improving the feature characterization capability and subsequent registration accuracy.

[0080] In one embodiment, the method further comprises:

[0081] S51, obtaining real-time motion parameters; the motion parameters include the three-dimensional angular velocity, acceleration and geographic location coordinates of the imager;

[0082] S52, constructing a spatiotemporal motion compensation model based on the motion parameters, and performing motion distortion correction on the stable registration result using the spatiotemporal motion compensation model to generate a motion compensated registration result;

[0083] S53, using a convolutional neural network to perform feature mapping learning on historical data of registration errors and corresponding motion parameters to generate a motion-error prediction model;

[0084] S54, based on the current motion parameters, the motion-error prediction model is used to perform pre-compensation optimization on the motion compensation registration result to generate the final registration result.

[0085] Specifically, the real-time motion parameters of the imager's three-dimensional angular velocity, acceleration and geographic location coordinates can be obtained through the inertial measurement unit and the global positioning system to characterize the dynamic posture and position changes of the device in space; a spatiotemporal motion compensation model is constructed based on the motion parameters, and the motion distortion of the feature points in the stable registration results is corrected by mapping the three-dimensional motion parameters into a spatial transformation matrix: the mapping relationship between the motion parameters and the displacement of the feature points is calculated to generate a motion compensation registration result, eliminating the image deformation caused by the movement of the device; a convolutional neural network can be used to learn the feature mapping of the registration error history data and the corresponding motion parameters, and a double-branch convolution is constructed. A neural network (the first branch processes the motion parameter tensor, the second branch processes the temporal characteristics of the registration error, and generates a prediction error through a feature fusion layer), combined with a training strategy based on dynamic adjustment of sample weights and random angular velocity noise injection based on acceleration amplitude, iteratively optimizes network parameters until the prediction error drops to a preset threshold, and generates a motion-error prediction model that can predict the correlation between motion parameters and registration error; the current motion parameters are input into the prediction model to obtain pre-compensation parameters and optimize the motion compensation registration result, and the final registration result is generated by pre-compensating the predicted motion error, thereby achieving a dynamic improvement in the registration accuracy of the device in motion scenarios.

[0086] In one embodiment, the step of training the motion-error prediction model includes:

[0087] S61, constructing a training set using historical data of registration errors and corresponding motion parameters;

[0088] S62, constructing a two-branch convolutional neural network based on the training set; the first branch of the two-branch convolutional neural network is used to process the motion parameter tensor, and the second branch is used to process the temporal characteristics of the registration error, and generate an initial prediction error through a feature fusion layer;

[0089] S63, dynamically adjusting the sample weight coefficient of the training set according to the acceleration amplitude in the motion parameter to generate weighted training samples;

[0090] S64, injecting random angular velocity noise into the weighted training samples, and iteratively updating the corresponding parameters of the two-branch convolutional neural network through the back propagation algorithm until the initial prediction error drops to a preset threshold.

[0091] For example, a training set consisting of historical registration error data and its corresponding motion parameters can be constructed. Based on this training set, a two-branch convolutional neural network architecture is constructed: the first branch processes the motion parameter tensor to achieve feature encoding of the motion state; the second branch processes the temporal characteristics of the registration error to capture the error evolution pattern. The outputs of the two branches are coupled through a feature fusion layer for heterogeneous data to generate an initial prediction error. To improve the model's adaptability to intense motion scenes, the weight coefficients of the training samples can be dynamically adjusted based on the acceleration amplitude in the motion parameters: the weights of samples with acceleration amplitudes exceeding a threshold (e.g., 2g) are increased exponentially, strengthening the model's learning ability for highly dynamic scenes. Random angular velocity noise (with an amplitude of 1.5-2 times the standard deviation of the measured angular velocity, generated using a Gaussian distribution) is injected into the weighted training samples to simulate device jitter and enhance the model's robustness. The network parameters are iteratively updated using a backpropagation algorithm, with the mean squared error (MSE) used as the loss function to drive model optimization until the prediction error converges to a preset threshold (e.g., 0.05 pixel units, which can be determined based on usage requirements). Through the above technical solution, the model can learn the nonlinear mapping relationship between the device motion state and the registration error, solving the common industry problem of reduced registration accuracy in device motion scenarios.

[0092] The above-mentioned multispectral video dynamic registration method for a cooled infrared imager obtains multispectral video and extracts spectral feature data using a multispectral decomposition algorithm based on wavelet transform. It then constructs a feature retrieval framework by combining temporal information integration and spatial information extraction techniques to achieve multi-dimensional feature integration and precise positioning. It also uses multi-dimensional information fusion technology to perform feature enhancement and dimensionality reduction on the initial spectral feature positioning results, improving data representation capabilities. A dynamic registration model is used to generate preliminary registration results based on real-time environmental change data. Model parameters are adaptively updated based on registration errors using a closed-loop feedback control mechanism. A spatiotemporal motion compensation model and a motion-error prediction model are constructed in conjunction with real-time motion parameters to achieve motion distortion correction and pre-compensation optimization. This method addresses the problems of traditional fixed-parameter registration models, such as the lack of a dynamic response mechanism, insufficient multi-dimensional information fusion, and lack of dynamic error compensation. It can adjust the registration strategy in real time in complex environments, reduce registration errors, improve the stability of the registration results, and enhance the ability to compensate for motion distortion in scenarios such as device movement, thus meeting the real-time registration requirements of high-precision infrared imaging.

[0093] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0094] Based on the same inventive concept, the embodiments of the present application also provide a multispectral video dynamic registration system of a refrigeration type infrared imager for implementing the above-mentioned multispectral video dynamic registration method of a refrigeration type infrared imager. The problem-solving implementation scheme provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more multispectral video dynamic registration system embodiments of a refrigeration type infrared imager provided below can be referred to the limitations of the multispectral video dynamic registration method of a refrigeration type infrared imager described above, which will not be repeated here.

[0095] In an exemplary embodiment, as shown in FIG. 1, a multispectral video dynamic registration system of a refrigeration type infrared imager is provided, comprising: Figure 2 A spectral feature extraction module 101 is configured to acquire a multispectral video and extract spectral feature data from the multispectral video using a multispectral decomposition algorithm based on wavelet transform;

[0096] A retrieval framework modeling module 102 is configured to construct a feature retrieval framework based on the spectral feature data, combined with a time sequence information integration and spatial information extraction technology, and retrieve an initial spectral feature positioning result using the feature retrieval framework;

[0097] An information fusion processing module 103 is configured to perform feature enhancement and dimension reduction processing on the initial spectral feature positioning result using a multi-dimensional information fusion technology, and reorganize the processing result into a structured feature data set to generate a spectral feature data set;

[0098] A registration model execution module 104 is configured to acquire real-time environmental change data, and perform registration on the spectral feature data set using a preset dynamic registration model based on the real-time environmental change data to generate a preliminary registration result;

[0099]

[0100] ​The closed-loop feedback optimization module 105 is used to calculate the registration error of the preliminary registration result, and based on the registration error, adaptively update the dynamic registration model using a preset closed-loop feedback control mechanism until a stable registration result is obtained.

[0101] In one embodiment, the retrieval framework building module 102 is further configured to reversely optimize the feature retrieval framework based on the registration error of the preliminary registration result through the following steps:

[0102] The correlation coefficient analysis method is used to analyze the relationship between the registration error and the feature matching results in the feature retrieval framework;

[0103] According to the correlation relationship, adjust the weight coefficients of temporal information integration technology and spatial information extraction technology;

[0104] The feature retrieval framework is updated using the adjusted weight parameters to obtain an optimized feature retrieval framework.

[0105] In one embodiment, the retrieval framework building module 102 is further configured to reversely optimize the initial spectral feature location results based on the optimized feature retrieval framework by performing the following steps:

[0106] According to the difference in weight parameters between the feature retrieval framework and the optimized feature retrieval framework, a spectral feature positioning update mechanism is constructed;

[0107] Using the spectral feature positioning update mechanism to update the initial spectral feature positioning result to obtain an updated spectral feature positioning result;

[0108] An adaptive weighted fusion strategy is used to integrate the updated spectral feature positioning results and the initial spectral feature positioning results to generate optimized spectral feature positioning results.

[0109] In one embodiment, the information fusion processing module 103 is further configured to reconstruct the spectral feature dataset based on the optimized spectral feature positioning result through the following steps:

[0110] The optimized spectral feature positioning results are used to resample the regional features of the multispectral video to generate a reconstructed spectral feature dataset;

[0111] The spatial and temporal consistency constraint algorithm is used to align and correct the reconstructed spectral features with the spectral feature dataset;

[0112] The corrected reconstructed spectral feature dataset is integrated into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset.

[0113] In one embodiment, the search framework building module 102 is further configured to integrate the corrected reconstructed spectral feature dataset into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset using the following formula:

[0114]

[0115] Among them, F opt For the optimized spectral feature dataset, To reconstruct the spectral feature dataset, is the spectral feature dataset, m is the number of reconstructed spectral feature frames, n is the number of spectral feature dataset frames, d r and d are the dimensions of the reconstructed spectral feature dataset and the spectral feature dataset, respectively; For the channel alignment weight matrix, by minimizing The solution is that Concat(·) is a cascade operation along the feature channel dimension.

[0116] In one embodiment, the registration model execution module 104 is further configured to:

[0117] Obtain real-time motion parameters; motion parameters include the imager's three-dimensional angular velocity, acceleration, and geographic location coordinates;

[0118] A spatiotemporal motion compensation model is constructed based on motion parameters, and motion distortion correction is performed on the stable registration result through the spatiotemporal motion compensation model to generate a motion compensated registration result;

[0119] A convolutional neural network is used to perform feature mapping learning on the historical data of registration errors and the corresponding motion parameters to generate a motion-error prediction model;

[0120] According to the current motion parameters, the motion-error prediction model is used to pre-compensate and optimize the motion-compensated registration results to generate the final registration results.

[0121] In one embodiment, the registration model execution module 104 is further configured to train the motion-error prediction model by:

[0122] The training set is constructed using the historical data of registration errors and the corresponding motion parameters;

[0123] A two-branch convolutional neural network is constructed based on the training set. The first branch of the two-branch convolutional neural network is used to process the motion parameter tensor, and the second branch is used to process the temporal features of the registration error and generate the initial prediction error through the feature fusion layer.

[0124] Dynamically adjust the sample weight coefficient of the training set according to the acceleration amplitude in the motion parameters to generate weighted training samples;

[0125] Random angular velocity noise is injected into the weighted training samples, and the corresponding parameters of the two-branch convolutional neural network are iteratively updated through the back-propagation algorithm until the initial prediction error drops to a preset threshold.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multispectral video dynamic registration method of a cooled infrared imager as described above are implemented.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0129] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A multispectral video dynamic registration method for a cooled infrared imager, characterized in that: The method comprises: Acquire a multispectral video, and extract spectral feature data from the multispectral video using a multispectral decomposition algorithm based on wavelet transform; Based on the spectral feature data, a feature retrieval framework is constructed by combining time series information integration and spatial information extraction technology, and the feature retrieval framework is used to retrieve and obtain the initial spectral feature positioning result; Using multidimensional information fusion technology to perform feature enhancement and dimensionality reduction processing on the initial spectral feature positioning results, and reorganizing the processing results into a structured feature data set to generate a spectral feature data set; Acquire real-time environmental change data, and based on the real-time environmental change data, register the spectral feature dataset using a preset dynamic registration model to generate a preliminary registration result; The registration error of the preliminary registration result is calculated, and based on the registration error, the dynamic registration model is adaptively updated using a preset closed-loop feedback control mechanism until a stable registration result is obtained.

2. The method according to claim 1, characterized in that The method further includes reversely optimizing the feature retrieval framework based on the registration error of the preliminary registration result by the following steps: Analyzing the correlation between the registration error and the feature matching result in the feature retrieval framework using a correlation coefficient analysis method; According to the association relationship, adjusting the weight coefficients of the temporal information integration technology and the spatial information extraction technology; The feature retrieval framework is updated using the adjusted weight parameters to obtain an optimized feature retrieval framework.

3. The method according to claim 2, characterized in that The method further includes reversely optimizing the initial spectral feature positioning result based on the optimized feature retrieval framework through the following steps: Constructing a spectral feature positioning update mechanism according to the difference in weight parameters between the feature retrieval framework and the optimized feature retrieval framework; Using the spectral feature positioning update mechanism to update the initial spectral feature positioning result to obtain an updated spectral feature positioning result; An adaptive weighted fusion strategy is used to integrate the updated spectral feature positioning result and the initial spectral feature positioning result to generate an optimized spectral feature positioning result.

4. The method according to claim 3, characterized in that The method further includes reconstructing the spectral feature dataset based on the optimized spectral feature positioning result by the following steps: Performing regional feature resampling on the multispectral video using the optimized spectral feature positioning result to generate a reconstructed spectral feature dataset; Using a spatiotemporal consistency constraint algorithm to align and correct the reconstructed spectral features with the spectral feature dataset; The corrected reconstructed spectral feature dataset is integrated into the spectral feature dataset using feature cascade fusion to generate an optimized spectral feature dataset.

5. The method according to claim 4, characterized in that The feature cascade fusion is used to integrate the corrected reconstructed spectral feature dataset into the spectral feature dataset to generate an optimized spectral feature dataset, which is achieved by the following formula: Among them, F opt For the optimized spectral feature dataset, To reconstruct the spectral feature dataset, is the spectral feature dataset, m is the number of reconstructed spectral feature frames, n is the number of spectral feature dataset frames, d r and d are the dimensions of the reconstructed spectral feature dataset and the spectral feature dataset, respectively; For the channel alignment weight matrix, by minimizing The solution is that Concat(·) is a cascade operation along the feature channel dimension.

6. The method according to claim 1, characterized in that The method further comprises: Acquiring real-time motion parameters; the motion parameters include the three-dimensional angular velocity, acceleration and geographic location coordinates of the imager; constructing a spatiotemporal motion compensation model based on the motion parameters, and performing motion distortion correction on the stable registration result using the spatiotemporal motion compensation model to generate a motion compensated registration result; Using a convolutional neural network to perform feature mapping learning on the historical data of the registration error and the corresponding motion parameters to generate a motion-error prediction model; According to the current motion parameters, the motion-error prediction model is used to perform pre-compensation optimization on the motion-compensated registration result to generate a final registration result.

7. The method according to claim 6, characterized in that The training steps of the motion-error prediction model include: Using the historical data of the registration error and the corresponding motion parameters, a training set is constructed; Constructing a two-branch convolutional neural network based on the training set; the first branch of the two-branch convolutional neural network is used to process the motion parameter tensor, and the second branch is used to process the registration error time series features, and generate an initial prediction error through a feature fusion layer; Dynamically adjusting the sample weight coefficient of the training set according to the acceleration amplitude in the motion parameter to generate weighted training samples; Random angular velocity noise is injected into the weighted training samples, and the parameters corresponding to the two-branch convolutional neural network are iteratively updated through a back-propagation algorithm until the initial prediction error drops to a preset threshold.

8. A multispectral video dynamic registration system for a cooled infrared imager, characterized in that: The system comprises: A spectral feature extraction module is used to obtain a multispectral video and extract spectral feature data from the multispectral video using a multispectral decomposition algorithm based on wavelet transform; A retrieval framework construction module is used to construct a feature retrieval framework based on the spectral feature data, combining time series information integration and spatial information extraction technology, and use the feature retrieval framework to retrieve and obtain initial spectral feature positioning results; An information fusion processing module is used to perform feature enhancement and dimensionality reduction processing on the initial spectral feature positioning results using multi-dimensional information fusion technology, and reorganize the processing results into a structured feature data set to generate a spectral feature data set; A registration model execution module is used to obtain real-time environmental change data, and based on the real-time environmental change data, register the spectral feature data set using a preset dynamic registration model to generate a preliminary registration result; The closed-loop feedback optimization module is used to calculate the registration error of the preliminary registration result, and based on the registration error, adaptively update the dynamic registration model using a preset closed-loop feedback control mechanism until a stable registration result is obtained.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images