Camera Thermal Image Drift Compensation Method Based on Multi-Feature Temporal Fusion

Abstract

The present invention discloses a method for compensating thermal-induced image drift of a camera based on multi-feature time series fusion, which relates to the technical field of camera image compensation. The present invention includes: measuring time, camera temperature, and ambient temperature data and performing preprocessing to form a time series data set of drift caused by the start-up thermal effect of the camera, and dividing the time series data set into a training set, a test set, and a validation set; constructing a neural network model, using the neural network model to fit the curve of time-drift amount, and obtaining the law of temperature drift in the stage from camera start-up to thermal equilibrium. The present invention aims at the image drift caused by the start-up thermal effect of the camera, uses the camera temperature, ambient temperature, and measurement time as characteristic values, predicts the image drift amount and eliminates it, significantly improves the measurement accuracy of the chain camera, and compared with the prior art, can reduce the influence of camera start-up heating on image drift during short-time measurement.

Description

Technical Field

[0001] The present invention relates to the technical field of camera image compensation, and specifically to a method for compensating camera thermal image drift based on multi-feature time series fusion. Background Art

[0002] With the increasing demand in the field of intelligent operation and maintenance of infrastructure in China, real-time dynamic spatial deformation monitoring of bridges can effectively reflect the health status of bridges. During the process of bridge spatial deformation monitoring based on machine vision, the camera is an important part of the monitoring system. Since the camera temperature changes continuously from startup to stable operation and usually takes a certain amount of time to reach thermal equilibrium, the change in temperature will cause regular drift in the displacement measurement results, which has a great impact on the accuracy and reliability of the measurement results. Therefore, in the bridge spatial deformation monitoring based on machine vision, a new technical solution is needed to eliminate the thermal effect of camera startup and ultimately improve the accuracy of the measurement results.

[0003] To solve the above problems, the existing patent (Publication No.: CN114326857B) proposes a device and method for actively compensating digital image processing errors under low temperature conditions, which controls the temperature by heating the resistance sheet and shortens the long waiting time from camera self-heating startup to measurement equilibrium state. However, this type of device has high requirements for the position layout of the resistance sheet. Otherwise, it is easy to cause a difference in the internal temperature distribution of the camera from the camera thermal equilibrium state under non-interference conditions. Moreover, although the camera heating process is shortened, it still takes a certain amount of time, which cannot be ignored in short-term measurements. For this reason, the present invention proposes a method for compensating camera thermal image drift based on multi-feature time series fusion. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for compensating camera thermal image drift based on multi-feature time series fusion. Aiming at the image drift caused by the thermal effect of camera startup, taking the camera real-time temperature, ambient temperature, and measurement time as characteristic values, predicting the image drift amount and eliminating it, so as to achieve the purpose of improving the accuracy of bridge spatial deformation monitoring.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A method for compensating camera thermal image drift based on multi-feature time series fusion, including the following steps:

[0006] Receive time, camera temperature, and ambient temperature data and perform preprocessing to form a time series data set of drift caused by the thermal effect of camera startup, and divide the time series data set into a training set, a test set, and a validation set;

[0007] Construct a neural network model, use the neural network model to fit the curve of time-drift amount, and obtain the law of temperature drift in the stage from camera startup to thermal equilibrium;

[0008] The neural network model is trained using a training set, with the mean squared error as the training objective function. The adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size, and the trained neural network model is obtained after iterative calculation.

[0009] The trained neural network model is used to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assuming that the static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is the standard static data, the similarity between the non-standard static data and the standard static data is calculated.

[0010] A mapping relationship between the neural network model parameters and the similarity is established, and the parameters of the neural network model are adjusted according to the validation set for a new round of iterative calculation until the final thermal effect fitting prediction model is obtained.

[0011] Further, a time-series data set caused by the thermal effect of camera startup drift is constructed as follows:

[0012] Step 2-1. Temperature sensors are arranged on the camera surface and the surrounding environment, and the camera temperature and ambient temperature data are collected using the temperature sensors.

[0013] Step 2-2. Using a chain camera at different spacings, multiple groups of static tests are carried out. After each measurement, the camera is shut down for a period of time until it cools down sufficiently, and the change law of the thermal effect displacement drift result of the chain camera with time is collected.

[0014] Further, the measurement time, camera temperature, and ambient temperature data are preprocessed using Z-score standardization.

[0015] Further, the neural network model structure includes an input layer, a feature fusion layer, a hidden layer, and an output layer. The number of neurons in the input layer is 3, which is used to receive input feature parameters, where the input feature parameters include measurement time, camera temperature, and ambient temperature.

[0016] Further, the feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer;

[0017] Among them, the first fully connected layer contains 256 neurons and uses the ReLU activation function;

[0018] The dropout rate of the Dropout layer is set to 0.3;

[0019] The second fully connected layer contains 128 neurons and uses the ReLU activation function;

[0020] The third fully connected layer contains 64 neurons and uses the ReLU activation function.

[0021] Further, the neural network model is trained using a training set, with the mean squared error as the training objective function, and the adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size. After iterative calculation, the trained neural network model is obtained, as follows:

[0022] Step 6-1. Use the mean squared error MSE to measure the difference between the prediction and the true drift amount:

[0023]

[0024] where is the true drift amount, is the predicted drift amount, and n is the number of samples;

[0025] Step 6-2. Use the Adam optimization algorithm to dynamically adjust the parameter update step size, and maintain the first-order moment estimate , the second-order moment estimate , where is an approximation of the gradient mean value, is an approximation of the gradient variance, is the decay rate for controlling the first-order moment estimate, is the decay rate for controlling the second-order moment estimate, is the gradient of the mean squared error with respect to the parameter θ; is the first-order moment estimate value at time , that is, the approximation of the gradient mean value up to time is the second-order moment estimate value at time

[0026] Step 6-3. The parameter update formula is:

[0027]

[0028] In the formula is the learning rate, set to 0.001, , , is to prevent the denominator from being zero, ; is the parameter value obtained after the update at the current time, is the parameter value obtained after the update at time is for the second-order moment estimate value after bias correction; is for the first-order moment estimate value after bias correction;

[0029] Step 6-4. Finally, set the batch size to 32, and stop the training when the validation set loss does not decrease for 10 consecutive rounds through early stopping, so as to obtain the trained neural network model.

[0030] Further, use the trained neural network model to eliminate the thermal effect on the real-time measurement data to obtain non-standard static data. Assume that the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable within the standard time period is the standard static data, and calculate the similarity between the non-standard static data and the standard static data as follows:

[0031] Step 7-1. The temperature of the camera continuously rises after startup and remains unchanged after reaching thermal equilibrium. Define the measured displacement obtained within any 10-minute time period after the camera reaches thermal equilibrium and the temperature remains unchanged as the standard static data, and define the measured displacement obtained within any 10-minute time period when the camera has not fully reached thermal equilibrium and the temperature is still changing as the original measured displacement. The original measured displacement is fitted by the neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data;

[0032] Step 7-2. The similarity between the standard static data and the non-standard static data is characterized by the KL divergence, which is specifically as follows:

[0033]

[0034] where x is the value of the displacement, represents the probability that the random variable x appears in the distribution P of the standard static data; represents the probability that the random variable x appears in the distribution Q of the non-standard static data.

[0035] Further, adjust the parameters of the neural network model. The specific parameters include weights, learning rate, dropout rate, number of neurons, and model training time.

[0036] Further, establish the mapping relationship between the neural network model parameters and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained, which is specifically as follows:

[0037] In the model construction and training process, the time series data set is divided into a training set, a test set, and a validation set in a ratio of 70%, 15%, and 15% through cross-validation;

[0038] During training, enable the automatic hyperparameter tuning algorithm, combining grid search and random search;

[0039] For the weights, set the standard deviation range of the He normal distribution to [0.5, 2] to explore different initialization distributions; for the learning rate, set 10 trial values from 1e-5 to 0.1 on a logarithmic scale; for the dropout rate, set 11 trial values from 0 to 1 at intervals of 0.1;

[0040] For the number of neurons, in the first fully connected layer, it increases from 128 to 512 in increments of 128; in the second fully connected layer, it increases from 64 to 256 in increments of 64; in the third fully connected layer, it increases from 32 to 128 in increments of 32;

[0041] Each time the model is trained iteratively, synchronously record the KL divergence, the accuracies of the training set and the validation set. If the KL divergence increases, and if the update amplitude fluctuates by more than 20% due to unstable weight updates, switch the weight initialization method to the Xavier normal distribution; if the validation set loss increases in three consecutive rounds due to an overly large learning rate, reduce the learning rate to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, increase the dropout rate by 0.1 or reduce the number of neurons in the corresponding layer by 20%;

[0042] Continuously search through the automatic hyperparameter tuning algorithm until the KL divergence converges to the minimum value to obtain the final heat effect fitting prediction model.

[0043] According to the second aspect of the present invention, the present invention provides a camera startup heat effect image drift compensation system for implementing the above-mentioned camera thermal-induced image drift compensation method based on multi-feature time series fusion, including:

[0044] A dataset construction module, configured to receive time, camera temperature, and environmental temperature data and perform preprocessing to form a time series dataset of drift caused by the camera startup heat effect, and divide the time series dataset into a training set, a test set, and a validation set;

[0045] A model construction module, configured to construct a neural network model, use the neural network model to fit the curve of time - drift amount, and obtain the law of temperature drift during the camera startup to the thermal equilibrium stage. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer;

[0046] A training module, configured to train the neural network model using the training set, use the mean squared error as the training objective function, and adopt the adaptive moment estimation algorithm to dynamically adjust the parameter update step size, and obtain the trained neural network model after iterative calculation;

[0047] A similarity calculation module, configured to use the trained neural network model to eliminate the heat effect from the real-time measurement data to obtain non-standard static data. Assuming that the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable within a standard time period is the standard static data, calculate the similarity between the non-standard static data and the standard static data;

[0048] Parameter optimization and output module, which is used to establish the mapping relationship between the parameters of the neural network model and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained;

[0049] Image drift compensation module, which is used to use the final thermal effect fitting prediction model to predict and eliminate the image drift amount, so as to compensate for the image drift caused by the thermal effect of camera startup.

[0050] The present invention has at least the following beneficial effects:

[0051] 1. Aiming at the image drift caused by the thermal effect of camera startup, the present invention uses the real-time temperature of the camera, the ambient temperature, and the measurement time as characteristic values, predicts the image drift amount and eliminates it, significantly improves the measurement accuracy of the chain camera, and thus achieves the purpose of improving the monitoring accuracy of the bridge spatial deformation. Compared with the prior art, it can reduce the influence of the warm-up time of camera startup on image drift during short-time measurement.

[0052] 2. In the present invention, the model receives dynamic parameters such as time and temperature in real time, has good real-time performance, the model network structure is simple, the response speed is fast, and the elimination efficiency of the drift effect caused by the thermal effect is high.

[0053] 3. The present invention uses three main factors, namely the measurement time, the temperature of the camera itself, and the ambient temperature, as the prediction feature inputs of the camera drift amount, considering comprehensively and fully taking into account the influence of multi-factor coupling on the thermal effect of camera startup.

[0054] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic flow chart of the compensation method described in the present invention;

[0056] Figure 2 It is a schematic structural diagram of the neural network model in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0058] Embodiment 1:

[0059] Please refer to Figure 1, the present invention provides a technical solution: a method for compensating thermal image drift of a camera based on multi-feature time series fusion, comprising the following steps:

[0060] S1. Receive measurement time, camera temperature, and environmental temperature data and perform preprocessing to form a time series data set of drift caused by the start-up thermal effect of the camera, and divide the time series data set into a training set, a test set, and a validation set, with the division ratios being 70%, 15%, and 15% respectively;

[0061] For the technical solution of this embodiment, a bridge spatial deformation monitoring device based on machine vision is used. The device is composed of a camera module, a GPS module, a 4G module, a laser sensor module, an IMU inertial navigation unit, etc. Among them, the camera module uses a 1 / 2.8-inch sensor of the camera, with 8 million pixels and a resolution of 3840x2160. It uses a MIPI interface and conducts multiple groups of static tests at different spacings. After each measurement, the camera is turned off for a period of time until it cools down sufficiently to obtain the change of the thermal effect displacement drift result of this type of camera over time;

[0062] Secondly, temperature sensors are arranged on the surface and in the surrounding environment of the camera. After collecting a large batch of data, a rich database of thermal effect drift during camera startup is formed;

[0063] Furthermore, the measurement time, camera temperature, and environmental temperature data are preprocessed using Z-score normalization;

[0064] S2. Construct a neural network model, use the neural network model to fit the curve of time-drift amount, and obtain the law of temperature drift from camera startup to thermal equilibrium stage. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer;

[0065] Specifically, as Figure 2 shown:

[0066] Input layer: Receive three-dimensional feature parameters, including measurement time, camera temperature, and environmental temperature. The number of neurons in the input layer is 3, and it directly receives the normalized multi-source sensing data;

[0067] The feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer:

[0068] The first fully connected layer contains 256 neurons and uses the ReLU activation function to perform non-linear fusion and high-order feature extraction on the input features; the Dropout layer has a dropout rate of 0.3 to suppress overfitting during training; the second fully connected layer contains 128 neurons and is ReLU-activated to further abstract the non-linear coupling relationship between features; the third fully connected layer contains 64 neurons and is ReLU-activated to achieve feature dimensionality reduction and key information concentration;

[0069] The output layer contains one linearly activated neuron, which directly outputs the predicted image drift amount to achieve end-to-end regression mapping;

[0070] S3. Train the neural network model using the training set, with the mean squared error as the training objective function, and use the adaptive moment estimation algorithm to dynamically adjust the parameter update step size. After iterative calculation, the trained neural network model is obtained, as follows:

[0071] Use the mean squared error to measure the difference between the prediction and the true drift amount, where is the true drift amount, is the predicted drift amount, and n is the number of samples;

[0072] Use the Adam optimization algorithm to dynamically adjust the parameter update step size and maintain the first moment estimate , the second moment estimate , where is the decay rate for controlling the first moment estimate, is the decay rate for controlling the second moment estimate, is the gradient of the mean squared error with respect to the parameter θ;

[0073] The parameter update formula is , where is the learning rate (default 0.001), , , to prevent the denominator from being zero ; is the first moment estimate value at the previous moment ( ), which is an approximation of the mean gradient up to the previous moment; is the second moment estimate value at the previous moment ( ); is the parameter value obtained after updating at the current moment ( ); is the parameter value obtained after updating at the previous moment ( ); is the second moment estimate value after bias correction for ; is the first moment estimate value after bias correction for ; Finally, set the batch size to 32 and stop training when the validation set loss does not decrease for 10 consecutive rounds using early stopping to obtain the trained model;

[0074] S4. Use the trained neural network model to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assume that during the standard period, the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable is the standard static data. Calculate the similarity between the non-standard static data and the standard static data (similarity evaluation), as follows:

[0075] The temperature of the camera continuously rises after startup and remains constant after reaching thermal equilibrium; the measured displacement obtained within a certain 10-minute period after the camera reaches thermal equilibrium and the temperature remains constant is the standard static data; the original measured displacement obtained within a certain 10-minute period when the camera has not fully reached thermal equilibrium and the temperature is still changing is fitted by the above neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data; the similarity between the standard static data and the non-standard static data is characterized by the KL divergence, , where x is the value of the displacement, represents the probability that the random variable x appears in the distribution P of the standard static data; represents the probability that the random variable x appears in the distribution Q of the non-standard static data;

[0076] S5. Establish the mapping relationship between the parameters of the neural network model and the similarity, adjust the parameters of the neural network model according to the validation set (hyperparameter optimization), and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained, as follows:

[0077] In the model construction and training process, the time-series dataset is divided into a training set, a test set, and a validation set at a ratio of 70%, 15%, and 15% through cross-validation; during training, the automatic hyperparameter tuning algorithm is enabled, combining grid search and random search. For the weights, the standard deviation range of the He normal distribution is set to [0.5, 2] to explore different initialization distributions; the learning rate is set with 10 trial values from 1e-5 to 0.1 on a logarithmic scale; the dropout rate is set with 11 trial values from 0 to 1 at an interval of 0.1; in terms of the number of neurons, the first fully connected layer increases from 128 to 512 in increments of 128; the second fully connected layer increases from 64 to 256 in increments of 64; the third fully connected layer increases from 32 to 128 in increments of 32; each time the model is trained iteratively, the KL divergence, the accuracies of the training set and the validation set are recorded synchronously; if the KL divergence increases and the update amplitude fluctuates by more than 20% due to unstable weight updates, the weight initialization method is switched to the Xavier normal distribution; if the validation set loss increases for three consecutive rounds due to an overly large learning rate, the learning rate is adjusted to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, the dropout rate is increased by 0.1 or the number of neurons in the corresponding layer is reduced by 20%. Through continuous searching by the algorithm until the KL divergence converges to near the minimum value and the model accuracy and training efficiency reach the best balance, the model performance is effectively improved;

[0078] In summary, in this embodiment, for the image drift caused by the camera startup thermal effect, taking the camera real-time temperature, ambient temperature, and measurement time as eigenvalue, the image drift amount is predicted and eliminated, significantly improving the measurement accuracy of the chain camera, so as to achieve the purpose of improving the monitoring accuracy of the bridge spatial deformation.

[0079] Embodiment 2:

[0080] This embodiment provides a camera startup thermal effect image drift compensation system for implementing the camera thermal-induced image drift compensation method based on multi-feature time-series fusion described in Embodiment 1, including:

[0081] A dataset construction module, configured to receive time, camera temperature, and ambient temperature data and perform preprocessing to form a time-series dataset of drift caused by the camera startup thermal effect, and divide the time-series dataset into a training set, a test set, and a validation set;

[0082] A model construction module, configured to construct a neural network model, use the neural network model to fit the curve of time-drift amount, and obtain the law of temperature drift in the stage from camera startup to thermal equilibrium. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer;

[0083] A training module for training a neural network model using a training set, with the mean squared error as the training objective function, and dynamically adjusting the parameter update step size using the adaptive moment estimation algorithm. After iterative calculation, a trained neural network model is obtained.

[0084] A similarity calculation module for removing thermal effects from real-time measurement data using the trained neural network model to obtain non-standard static data. Assuming that the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable during the standard period is the standard static data, calculate the similarity between the non-standard static data and the standard static data.

[0085] A parameter optimization and output module for establishing a mapping relationship between the neural network model parameters and the similarity, adjusting the parameters of the neural network model according to the validation set, and performing a new round of iterative calculation until the final thermal effect fitting prediction model is obtained.

[0086] An image drift compensation module for using the final thermal effect fitting prediction model to predict and remove the image drift amount, thereby compensating for the image drift caused by the activation of the thermal effect of the camera.

[0087] Specifically, the above data set construction module, model construction module, training module, similarity calculation module, parameter optimization output module, and image drift compensation module can be embedded in a computer processing system. The computer calls the above modules to complete the task of predicting and removing the image drift amount according to the above-provided method for compensating camera thermal-induced image drift based on multi-feature time series fusion; the above data set construction module, model construction module, training module, similarity calculation module, parameter optimization output module, and image drift compensation module can perform operations according to the specific steps given by the above method for compensating camera thermal-induced image drift based on multi-feature time series fusion.

[0088] It should be noted that the division of each module of the above system is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data set construction module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and the function of the above signal processing module can be called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element mentioned here can be an integrated circuit with the ability to process signals. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuit of the hardware in the processor element or the instruction in the form of software.

[0089] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as: one or more Application Specific Integrated Circuits (ASICs), or, one or more Digital Singnal Processors (DSPs), or, one or more Field Programmable Gate Arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0090] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0091] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. When an element is referred to as "assembled on", "installed on", "fixed to" or "disposed on" another element, it can be directly on the other element or there may also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only embodiments.

[0092] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0093] In the description of this specification, the description with reference to terms such as "an embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

Claims

1. A method for compensating camera thermal image drift based on multi-feature time series fusion, characterized in that It includes the following steps: Receive measurement time, camera temperature, and ambient temperature data and perform preprocessing to form a time-series dataset of drift caused by the thermal effect during camera startup, and divide the time-series dataset into a training set, a test set, and a validation set; Construct a neural network model, use the neural network model to fit the time-drift curve, and obtain the law of temperature drift during the camera startup to thermal equilibrium stage; Use the training set to train the neural network model, use the mean squared error as the training objective function, and use the adaptive moment estimation algorithm to dynamically adjust the parameter update step size. After iterative calculation, obtain the trained neural network model; Use the trained neural network model to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assume that the static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is the standard static data, and calculate the similarity between the non-standard static data and the standard static data; Establish a mapping relationship between the neural network model parameters and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained; Use the final thermal effect fitting prediction model to predict and eliminate the image drift amount, so as to compensate for the image drift caused by the thermal effect during camera startup.

2. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 1, characterized in that Form a time-series dataset of drift caused by the thermal effect during camera startup, specifically as follows: Step 2-1. Arrange temperature sensors on the camera surface and the surrounding environment, and use the temperature sensors to collect camera temperature and ambient temperature data; Step 2-2. Use a chain camera to perform multiple groups of static tests at different spacings. After each measurement, turn off the camera for a period of time until the camera cools down sufficiently, and collect the variation law of the thermal effect displacement drift result of the chain camera with the measurement time.

3. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 2, characterized in that: The measurement time, camera temperature, and ambient temperature data are preprocessed using Z-score standardization.

4. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 3, characterized in that: The neural network model structure includes an input layer, a feature fusion layer, a hidden layer, and an output layer. The number of neurons in the input layer is 3, which is used to receive input feature parameters, where the input feature parameters include measurement time, camera temperature, and ambient temperature.

5. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 4, characterized in that: The feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer; Among them, the first fully connected layer contains 256 neurons and uses the ReLU activation function; The dropout rate of the Dropout layer is set to 0.3; The second fully connected layer contains 128 neurons and uses the ReLU activation function; The third fully connected layer contains 64 neurons and uses the ReLU activation function.

6. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 5, characterized in that Use the training set to train the neural network model, use the mean squared error as the training objective function, and use the adaptive moment estimation algorithm to dynamically adjust the parameter update step size. After iterative calculation, obtain the trained neural network model, specifically as follows: Step 6-1. Use the mean squared error MSE to measure the difference between the prediction and the real drift amount: where is the true drift amount, is the predicted drift amount, and n is the number of samples; Step 6-2. Dynamically adjust the parameter update step size using the Adam optimization algorithm and maintain the first moment estimate , the second moment estimate , where is an approximation of the gradient mean, is an approximation of the gradient variance, is the decay rate controlling the first moment estimate, is the decay rate controlling the second moment estimate, is the gradient of the mean squared error with respect to the parameter θ; is the first moment estimate value at time , that is, the approximation of the gradient mean up to time is the second moment estimate value at time Step 6-3. The parameter update formula is: where is the learning rate, set to 0.001, , , to prevent the denominator from being zero, ; is the parameter value obtained after the update at the current moment; is the parameter value obtained after the update at the moment; is the second - moment estimate value after bias correction for ; is the first - moment estimate value after bias correction for ; Finally, set the batch size to 32, and stop training when the validation set loss does not decrease for 10 consecutive rounds through early stopping, so as to obtain the trained neural network model.

7. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 6, wherein The trained neural network model is used to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assume that the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable during the standard period is the standard static data. Calculate the similarity between the non-standard static data and the standard static data as follows: Step 7-1. The temperature of the camera continuously rises after startup and remains constant after reaching thermal equilibrium. Define the measured displacement obtained within any 10-minute period after the camera reaches thermal equilibrium and the temperature remains constant as the standard static data. Define the measured displacement obtained within any 10-minute period when the camera has not fully reached thermal equilibrium and the temperature is still changing as the original measured displacement. The original measured displacement is fitted by the neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data; Step 7-2. The similarity between the standard static data and the non-standard static data is characterized by the KL divergence, as follows: where x is the value of displacement, represents the probability that the random variable x appears in the distribution P of standard static data; represents the probability that the random variable x appears in the distribution Q of non-standard static data.

8. The method for compensating camera thermal image drift based on multi-feature temporal fusion according to claim 7, wherein Adjust the parameters of the neural network model. The specific parameters include weights, learning rate, dropout rate, number of neurons, and model training time.

9. The method for compensating camera thermal image drift based on multi-feature time series fusion according to claim 8, wherein: Establish the mapping relationship between the neural network model parameters and the KL divergence. Adjust the parameters of the neural network model according to the validation set and perform a new round of iterative calculations until the final thermal effect fitting prediction model is obtained, as follows: In the model construction and training process, the time series dataset is divided into a training set, a test set, and a validation set at a ratio of 70%, 15%, and 15% through cross-validation; During training, enable the automatic hyperparameter tuning algorithm, combining grid search and random search; For the weights, set the standard deviation range of the He normal distribution to [0.5, 2] to explore different initialization distributions; For the learning rate, set 10 trial values from 1e-5 to 0.1 on a logarithmic scale; for the dropout rate, set 11 trial values from 0 to 1 at an interval of 0.1; For the number of neurons, the first fully connected layer ranges from 128 to 512, increasing in increments of 128; the second fully connected layer ranges from 64 to 256, increasing in increments of 64; the third fully connected layer ranges from 32 to 128, increasing in increments of 32; Each time the model is iteratively trained, synchronously record the KL divergence, the accuracy of the training set and the validation set. If the KL divergence increases, and if the update amplitude of the weights fluctuates by more than 20% due to unstable weight updates, switch the weight initialization method to the Xavier normal distribution; if the validation set loss increases for three consecutive rounds due to an overly large learning rate, reduce the learning rate to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, increase the dropout rate by 0.1 or reduce the number of neurons in the corresponding layer by 20%; Continuously search through the automatic hyperparameter tuning algorithm until the KL divergence converges to the minimum value to obtain the final thermal effect fitting prediction model.

10. A camera startup thermal effect image drift compensation system for implementing the camera thermal-induced image drift compensation method based on multi-feature time series fusion described in any one of claims 1 to 9, characterized in that, Including: A dataset construction module for receiving time, camera temperature, and environmental temperature data, preprocessing them, constructing a time series dataset of drift caused by the startup thermal effect of the camera, and dividing the time series dataset into a training set, a test set, and a validation set; The model building module is used to build a neural network model, and use the neural network model to fit the time-drift curve to obtain the temperature drift law from the camera startup to the thermal equilibrium stage; The training module is used to train the neural network model using the training set, taking the mean square error as the training objective function, and using the adaptive moment estimation algorithm to dynamically adjust the parameter update step size, and obtain the trained neural network model after iterative calculation; A similarity calculation module is used to use the trained neural network model to remove the thermal effect of the real-time measurement data to obtain non-standard static data. The static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is set as the standard static data, and the similarity between the non-standard static data and the standard static data is calculated; The parameter optimization and output module is used to establish the mapping relationship between the parameters of the neural network model and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained; The image drift compensation module is used to use the final thermal effect fitting prediction model to predict the image drift amount and eliminate it, so as to compensate for the image drift caused by the thermal effect when the camera is started.

Images