Camera thermally induced image drift compensation method based on multi-feature time sequence fusion

Abstract

The invention discloses a camera thermally induced image drift compensation method based on multi-feature time sequence fusion, and relates to the technical field of camera image compensation. The method comprises the steps that time, camera temperature and environment temperature data are measured and preprocessed, a time sequence data set of drifting caused by the camera starting heat effect is formed, and the time sequence data set is divided into a training set, a test set and a verification set; and constructing a neural network model, and fitting a time-drift distance curve by using the neural network model to obtain a law of temperature drift from camera starting to a heat balance stage. According to the method, the image drift amount is predicted and eliminated by taking the camera temperature, the environment temperature and the measurement time as characteristic values aiming at the image drift caused by the camera starting heat effect, the measurement accuracy of the chain-type camera is remarkably improved, and compared with the prior art, the influence of the camera starting temperature rise on the image drift can be reduced in the short-time measurement process.

Description

Technical Field

[0001] The present invention relates to the technical field of camera image compensation, and specifically to a method for compensating camera thermal image drift based on multi-feature time series fusion. Background Art

[0002] With the increasing demand in the field of intelligent operation and maintenance of infrastructure in China, real-time dynamic spatial deformation monitoring of bridges can effectively reflect the health status of bridges. During the process of bridge spatial deformation monitoring based on machine vision, the camera is an important part of the monitoring system. Since the camera temperature changes continuously from startup to stable operation and usually takes a certain time to reach thermal equilibrium, the temperature change will cause regular drift in the displacement measurement results, which has a great impact on the accuracy and reliability of the measurement results. Therefore, in the bridge spatial deformation monitoring based on machine vision, a new technical solution is needed to eliminate the thermal effect of camera startup and ultimately improve the accuracy of the measurement results.

[0003] To solve the above problems, a prior patent (Publication No.: CN114326857B) proposed a device and method for actively compensating digital image processing errors under low temperature conditions. By controlling the temperature through a heating resistor sheet, the long waiting time from camera self-heating startup to the measurement equilibrium state is shortened. However, this type of device has high requirements for the position layout of the resistor sheet. Otherwise, it is easy to cause a difference between the internal temperature distribution of the camera and the camera thermal equilibrium state under non-interference conditions. Moreover, although the camera heating process is shortened, it still takes a certain time, which cannot be ignored in short-term measurements. Therefore, the present invention proposes a method for compensating camera thermal image drift based on multi-feature time series fusion. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for compensating camera thermal image drift based on multi-feature time series fusion. Aiming at the image drift caused by the thermal effect of camera startup, using the real-time camera temperature, ambient temperature, and measurement time as eigenvalue, predicting the image drift amount and eliminating it, so as to achieve the purpose of improving the accuracy of bridge spatial deformation monitoring.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A method for compensating camera thermal image drift based on multi-feature time series fusion, including the following steps: Receiving time, camera temperature, and ambient temperature data and performing preprocessing to form a time series dataset of drift caused by the thermal effect of camera startup, and dividing the time series dataset into a training set, a test set, and a validation set; Constructing a neural network model, using the neural network model to fit the curve of time-drift amount, and obtaining the law of temperature drift during the stage from camera startup to thermal equilibrium; The neural network model is trained using a training set, with the mean squared error as the training objective function. The adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size, and the trained neural network model is obtained after iterative calculation; The trained neural network model is used to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assuming that the static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is the standard static data, the similarity between the non-standard static data and the standard static data is calculated; A mapping relationship between the neural network model parameters and the similarity is established, and the parameters of the neural network model are adjusted according to the validation set for a new round of iterative calculation until the final thermal effect fitting prediction model is obtained.

[0006] Further, a time series dataset of the drift caused by the startup thermal effect of the camera is constructed as follows: Step 2-1. Temperature sensors are arranged on the surface and in the surrounding environment of the camera, and the camera temperature and ambient temperature data are collected using the temperature sensors; Step 2-2. Using a chain camera at different spacings, multiple groups of static tests are carried out. After each measurement, the camera is shut down for a period of time until it cools down sufficiently, and the variation law of the thermal effect displacement drift result of the chain camera with time is collected.

[0007] Further, the measurement time, camera temperature, and ambient temperature data are preprocessed using Z-score standardization.

[0008] Further, the neural network model structure includes an input layer, a feature fusion layer, a hidden layer, and an output layer. The number of neurons in the input layer is 3, which is used to receive input feature parameters, where the input feature parameters include the measurement time, camera temperature, and ambient temperature.

[0009] Further, the feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer; Among them, the first fully connected layer contains 256 neurons and uses the ReLU activation function; The dropout rate of the Dropout layer is set to 0.3; The second fully connected layer contains 128 neurons and uses the ReLU activation function; The third fully connected layer contains 64 neurons and uses the ReLU activation function.

[0010] Further, the neural network model is trained using a training set, with the mean squared error as the training objective function. The adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size, and the trained neural network model is obtained after iterative calculation, specifically as follows: Step 6-1. Use the mean squared error (MSE) to measure the difference between the prediction and the true drift amount: where is the true drift amount, is the predicted drift amount, and n is the number of samples; Step 6-2. Use the Adam optimization algorithm to dynamically adjust the parameter update step size and maintain the first moment estimate , the second moment estimate , where is an approximation of the gradient mean, is an approximation of the gradient variance, is the decay rate controlling the first moment estimate, is the decay rate controlling the second moment estimate, is the gradient of the mean squared error with respect to the parameter θ; is the first moment estimate value at time , that is, the approximation of the gradient mean up to time is the second moment estimate value at time Step 6-3. The parameter update formula is: In the formula is the learning rate, set to 0.001, , , is to prevent the denominator from being zero, ; is the parameter value obtained after update at the current time, is the parameter value obtained after update at time is the second moment estimate value after bias correction for ; is the first moment estimate value after bias correction for ; Step 6-4. Finally, set the batch size to 32 and stop training when the validation set loss does not decrease for 10 consecutive rounds through early stopping, thereby obtaining the trained neural network model.

[0011] Furthermore, use the trained neural network model to eliminate the thermal effect from the real-time measurement data to obtain non-standard static data. Assume that the static test data when the camera reaches thermal equilibrium and the environmental temperature is stable during the standard period is the standard static data, and calculate the similarity between the non-standard static data and the standard static data as follows: Step 7-1. The temperature of the camera continuously rises after startup and remains constant after reaching thermal equilibrium. Define the measured displacement obtained within any 10-minute time period after the camera reaches thermal equilibrium and the temperature remains constant as the standard static data. Define the measured displacement obtained within any 10-minute time period when the camera has not fully reached thermal equilibrium and the temperature is still changing as the original measured displacement. The original measured displacement is fitted by the neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data; Step 7-2. The similarity between the standard static data and the non-standard static data is characterized by the KL divergence, which is specifically as follows: In the formula, x is the value of the displacement, represents the probability that the random variable x appears in the distribution P of the standard static data; represents the probability that the random variable x appears in the distribution Q of the non-standard static data.

[0012] Furthermore, adjust the parameters of the neural network model. The specific parameters include weights, learning rate, dropout rate, number of neurons, and model training time.

[0013] Furthermore, establish the mapping relationship between the neural network model parameters and the similarity. Adjust the parameters of the neural network model according to the validation set and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained, which is specifically as follows: In the model construction and training process, the time series data set is divided into a training set, a test set, and a validation set in the ratio of 70%, 15%, and 15% through cross-validation; During training, enable the automatic hyperparameter tuning algorithm, combining grid search and random search; For the weights, set the standard deviation range of the He normal distribution to [0.5, 2] to explore different initialization distributions; for the learning rate, set 10 trial values from 1e-5 to 0.1 on the logarithmic scale; for the dropout rate, set 11 trial values from 0 to 1 at intervals of 0.1; For the number of neurons, the first fully connected layer ranges from 128 to 512, increasing in increments of 128; the second fully connected layer ranges from 64 to 256, increasing in increments of 64; the third fully connected layer ranges from 32 to 128, increasing in increments of 32; Each time the model is iterated, the KL divergence, training set and validation set accuracy are recorded synchronously. If the KL divergence increases, if the update amplitude fluctuates by more than 20% due to unstable weight update, the weight initialization method is switched to Xavier normal distribution; if the learning rate is too large and the validation set loss increases for three consecutive rounds, the learning rate is reduced to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, the discard rate is increased by 0.1 or the number of neurons in the corresponding layer is reduced by 20%; The automatic parameter adjustment algorithm continuously searches until the KL divergence converges to the minimum value, and the final thermal effect fitting prediction model is obtained.

[0014] According to a second aspect of the present invention, the present invention provides a camera startup thermal effect image drift compensation system, which is used to implement the above-mentioned camera thermal image drift compensation method based on multi-feature time series fusion, comprising: The data set construction module is used to receive and preprocess the time, camera temperature, and ambient temperature data to form a time series data set of drift caused by thermal effects of camera startup, and divide the time series data set into a training set, a test set, and a validation set; A model building module is used to build a neural network model, and use the neural network model to fit the time-drift curve to obtain the temperature drift law from the camera startup to the thermal equilibrium stage. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer; The training module is used to train the neural network model using the training set, taking the mean square error as the training objective function, and using the adaptive moment estimation algorithm to dynamically adjust the parameter update step size, and obtain the trained neural network model after iterative calculation; A similarity calculation module is used to use the trained neural network model to remove the thermal effect of the real-time measurement data to obtain non-standard static data. The static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is set as the standard static data, and the similarity between the non-standard static data and the standard static data is calculated; The parameter optimization and output module is used to establish the mapping relationship between the parameters of the neural network model and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained; The image drift compensation module is used to use the final thermal effect fitting prediction model to predict the image drift amount and eliminate it, so as to compensate for the image drift caused by the thermal effect when the camera is started.

[0015] The present invention has at least the following beneficial effects: 1. The present invention aims at the image drift caused by the thermal effect during camera startup. Taking the real-time camera temperature, ambient temperature, and measurement time as characteristic values, it predicts the image drift amount and eliminates it, significantly improving the measurement accuracy of the chain camera, thereby achieving the purpose of improving the accuracy of bridge spatial deformation monitoring. Compared with the prior art, it can reduce the influence of the temperature rise time during camera startup on image drift during short-term measurement.

[0016] 2. In the present invention, the model receives dynamic parameters such as time and temperature in real time, has good real-time performance, a simple model network structure, a fast response speed, and a high efficiency in eliminating the drift effect caused by the thermal effect.

[0017] 3. The present invention takes three main factors, namely measurement time, the camera's own temperature, and ambient temperature, as the prediction feature inputs for the camera drift amount, considering comprehensively and fully taking into account the influence of multi-factor coupling on the thermal effect during camera startup.

[0018] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flow chart of the compensation method described in the present invention; Figure 2 is a schematic structural diagram of the neural network model in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0021] Embodiment 1: Please refer to Figure 1 , the present invention provides a technical solution: a method for compensating camera thermal-induced image drift based on multi-feature time series fusion, including the following steps: S1. Receive the measurement time, camera temperature, and ambient temperature data and perform preprocessing to form a time series data set of drift caused by the thermal effect during camera startup, and divide the time series data set into a training set, a test set, and a validation set, with the division ratios being 70%, 15%, and 15% respectively; For the technical solution of this embodiment, a bridge spatial deformation monitoring device based on machine vision is used. The device is composed of a camera module, a GPS module, a 4G module, a laser sensor module, an IMU inertial navigation unit, etc. Among them, the camera module uses a 1 / 2.8-inch sensor of the camera, with 8 million pixels and a resolution of 3840x2160. It uses the MIPI interface. Under different spacings, multiple groups of static tests are carried out. After each measurement, the machine is shut down for a period of time until the camera is fully cooled, and the change of the thermal effect displacement drift result of this type of camera over time is obtained; Secondly, temperature sensors are arranged on the surface of the camera and in the surrounding environment. After a large batch of data is collected, a rich thermal effect drift database of camera startup is formed; Furthermore, the measurement time, camera temperature, and environmental temperature data are preprocessed using Z-score normalization; S2. Build a neural network model, use the neural network model to fit the curve of time-drift amount, and obtain the law of temperature drift from camera startup to thermal equilibrium stage. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer; Specifically, as Figure 2 shown: Input layer: Receive three-dimensional feature parameters, including measurement time, camera temperature, and environmental temperature. The number of neurons in the input layer is 3, and it directly receives the normalized multi-source sensing data; The feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer: The first fully connected layer contains 256 neurons and uses the ReLU activation function to perform non-linear fusion and high-order feature extraction on the input features; the dropout rate of the Dropout layer is 0.3, which is used to suppress the overfitting phenomenon during training; the second fully connected layer contains 128 neurons and is ReLU-activated to further abstract the non-linear coupling relationship between features; the third fully connected layer contains 64 neurons and is ReLU-activated to achieve feature dimensionality reduction and key information concentration; The output layer contains 1 linearly activated neuron, which directly outputs the predicted image drift amount to achieve end-to-end regression mapping; S3. Use the training set to train the neural network model. Taking the mean square error as the training objective function, use the adaptive moment estimation algorithm to dynamically adjust the parameter update step size. After iterative calculation, the trained neural network model is obtained, specifically as follows: Use the mean square error to measure the difference between the prediction and the true drift amount, where is the true drift amount, is the predicted drift amount, and n is the number of samples; Use the Adam optimization algorithm to dynamically adjust the parameter update step size and maintain the first-order moment estimate , second-order moment estimate ,in To control the decay rate of the first-order moment estimate, To control the decay rate of the second-order moment estimate, is the gradient of the mean square error with respect to the parameter θ; The parameter update formula is: ,in is the learning rate (default 0.001), , , to prevent the denominator from being zero ; is the last moment ( ), which is the approximate value of the mean gradient up to the previous moment; is the last moment ( )’s second-order moment estimate; is the current moment ( ) The parameter value obtained after the update; is the last moment ( ) The parameter value obtained after the update; Yes The bias-corrected second moment estimates; Yes The first-order moment estimate after bias correction is performed; finally, the batch size is set to 32, and the training is stopped when the validation set loss does not decrease for 10 consecutive rounds using the early stopping method to obtain the trained model; S4. Use the trained neural network model to remove the thermal effect of the real-time measurement data to obtain non-standard static data. Set the static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period as the standard static data, and calculate the similarity between the non-standard static data and the standard static data (similarity evaluation), as follows: The temperature of the camera rises continuously after startup, and remains unchanged after reaching thermal equilibrium. The measured displacement obtained within a 10-minute period after the camera reaches thermal equilibrium and the temperature remains unchanged is the standard static data. The original measured displacement obtained within a 10-minute period when the camera has not fully reached thermal equilibrium and the temperature is still changing is fitted by the above neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data. The similarity between the standard static data and the non-standard static data is represented by KL divergence. , where x is the value of the displacement, It represents the probability of the random variable x appearing in the distribution P of the standard static data; It represents the probability of the random variable x appearing in the distribution Q of non-standard static data; S5. Establish the mapping relationship between the neural network model parameters and the similarity. Adjust the parameters of the neural network model (hyperparameter optimization) according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained. The specific steps are as follows: In the model construction and training process, the time series dataset is divided into a training set, a test set, and a validation set in a ratio of 70%, 15%, and 15% through cross-validation; during training, an automatic hyperparameter tuning algorithm is enabled, combining grid search and random search. For the weights, set the standard deviation range of the He normal distribution to [0.5, 2] to explore different initialization distributions; for the learning rate, set 10 trial values on a logarithmic scale from 1e-5 to 0.1; for the dropout rate, set 11 trial values from 0 to 1 at intervals of 0.1; for the number of neurons, in the first fully connected layer, it increases from 128 to 512 in increments of 128; in the second fully connected layer, it increases from 64 to 256 in increments of 64; in the third fully connected layer, it increases from 32 to 128 in increments of 32; each time the model is iteratively trained, record the KL divergence, the accuracy of the training set and the validation set synchronously; if the KL divergence increases, and if the update amplitude of the weights fluctuates by more than 20% due to unstable weight updates, switch the weight initialization method to the Xavier normal distribution; if the validation set loss increases for three consecutive rounds due to an overly large learning rate, reduce the learning rate to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, increase the dropout rate by 0.1 or reduce the number of neurons in the corresponding layer by 20%. Continuously search through the algorithm until the KL divergence converges to near the minimum value, and the model accuracy and training efficiency reach the best balance, thereby effectively improving the model performance; In summary, in this embodiment, for the image drift caused by the thermal effect during camera startup, taking the camera real-time temperature, ambient temperature, and measurement time as eigenvalue, predicting the image drift amount and eliminating it, significantly improving the measurement accuracy of the chain camera, so as to achieve the purpose of improving the accuracy of bridge spatial deformation monitoring.

[0022] Embodiment Two: This embodiment provides a camera startup thermal effect image drift compensation system for implementing the camera thermal-induced image drift compensation method based on multi-feature time series fusion described in Embodiment One, including: A dataset construction module, which is used to receive time, camera temperature, and ambient temperature data, perform preprocessing, form a time series dataset of drift caused by the thermal effect during camera startup, and divide the time series dataset into a training set, a test set, and a validation set; A model construction module, which is used to construct a neural network model, fit the curve of time-drift amount using the neural network model, and obtain the law of temperature drift during the camera startup to the thermal equilibrium stage. The neural network model includes an input layer, a feature fusion layer, a hidden layer, and an output layer; A training module, which is used to train a neural network model using a training set, with the mean squared error as the training objective function, and uses the adaptive moment estimation algorithm to dynamically adjust the parameter update step size, and obtains the trained neural network model after iterative calculation; A similarity calculation module, which is used to eliminate the thermal effect on the real-time measurement data using the trained neural network model to obtain non-standard static data. Assuming that the static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is the standard static data, it calculates the similarity between the non-standard static data and the standard static data; A parameter optimization and output module, which is used to establish the mapping relationship between the neural network model parameters and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained; An image drift compensation module, which is used to use the final thermal effect fitting prediction model to predict and eliminate the image drift amount, so as to compensate for the image drift caused by the thermal effect of the camera startup.

[0023] Specifically, the above dataset construction module, model construction module, training module, similarity calculation module, parameter optimization output module, and image drift compensation module can be embedded in a computer processing system. The computer calls the above modules according to the provided method for compensating camera thermal-induced image drift based on multi-feature time series fusion to complete the task of predicting and eliminating the image drift amount; the above dataset construction module, model construction module, training module, similarity calculation module, parameter optimization output module, and image drift compensation module can perform operations according to the specific steps given by the above method for compensating camera thermal-induced image drift based on multi-feature time series fusion.

[0024] It should be noted that it should be understood that the division of each module of the above system is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated, and these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the dataset construction module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device to perform the functions of the above signal processing module, and the implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or can be independently implemented. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the hardware of the processor element or the instruction in the form of software.

[0025] For example, these above-mentioned modules can be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or, one or more Digital Singnal Processors (DSPs), or, one or more Field Programmable Gate Arrays (FPGAs), etc. Again, when a certain above-mentioned module is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0026] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0027] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. When an element is referred to as being "assembled on", "mounted on", "fixed to" or "disposed on" another element, it can be directly on the other element or there may also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only implementation.

[0028] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0029] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A camera thermal image drift compensation method based on multi-feature temporal fusion, characterized in that: The following steps are involved: Receive the measurement time, camera temperature, and ambient temperature data and perform preprocessing to form a time series data set of drift caused by thermal effects of camera startup, and divide the time series data set into a training set, a test set, and a validation set; A neural network model was constructed and used to fit the time-drift curve to obtain the temperature drift law from camera startup to thermal equilibrium stage. The neural network model is trained using the training set, the mean square error is used as the training objective function, the adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size, and the trained neural network model is obtained after iterative calculation; The trained neural network model is used to remove the thermal effect of the real-time measurement data to obtain non-standard static data. The static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is set as the standard static data, and the similarity between the non-standard static data and the standard static data is calculated. Establish a mapping relationship between the neural network model parameters and similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculations until the final thermal effect fitting prediction model is obtained; The final thermal effect fitting prediction model is used to predict the image drift and eliminate it, thereby compensating for the image drift caused by the thermal effect when the camera is started.

2. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 1 is characterized in that: The time series data set that constitutes the drift caused by the thermal effect of camera startup is as follows: Step 2-1. Arrange temperature sensors on the camera surface and the surrounding environment, and use the temperature sensors to collect camera temperature and ambient temperature data; Step 2-2. Use a chain camera to perform multiple sets of static tests at different intervals. After each measurement, shut down the camera for a period of time until the camera is fully cooled, and collect the variation pattern of the chain camera thermal effect displacement drift results with the measurement time.

3. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 2 is characterized in that: The measurement time, camera temperature, and ambient temperature data were preprocessed using Z-score standardization.

4. The camera thermal image drift compensation method based on multi-feature temporal fusion according to claim 3 is characterized in that: The neural network model structure includes an input layer, a feature fusion layer, a hidden layer and an output layer. The number of neurons in the input layer is 3, which is used to receive input feature parameters, wherein the input feature parameters include measurement time, camera temperature and ambient temperature.

5. The camera thermal image drift compensation method based on multi-feature temporal fusion according to claim 4 is characterized in that: The feature fusion layer includes a first fully connected layer, and the hidden layer includes a Dropout layer, a second fully connected layer, and a third fully connected layer; The first fully connected layer contains 256 neurons and uses the ReLU activation function; The dropout rate of the Dropout layer is set to 0.3; The second fully connected layer contains 128 neurons and uses the ReLU activation function; The third fully connected layer contains 64 neurons and uses the ReLU activation function.

6. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 5, characterized in that: The training set is used to train the neural network model, the mean square error is used as the training objective function, and the adaptive moment estimation algorithm is used to dynamically adjust the parameter update step size. After iterative calculation, the trained neural network model is obtained, as follows: Step 6-1. Use mean square error (MSE) to measure the difference between the predicted and actual drift: in is the actual drift, To predict the drift, n is the number of samples; Step 6-2. Use the Adam optimization algorithm to dynamically adjust the parameter update step size and maintain the first-order moment estimate , second-order moment estimate ,in is an approximation of the mean gradient, is the approximation of the gradient variance, To control the decay rate of the first-order moment estimate, To control the decay rate of the second-order moment estimate, is the gradient of the mean square error with respect to the parameter θ; yes The first-order moment estimate at time The approximate value of the mean gradient up to time; yes The second moment estimate of the moment; Step 6-3. The parameter update formula is: In the formula is the learning rate, set to 0.001, , , To prevent the denominator from being zero, ; is current The parameter value obtained after the moment is updated; yes The parameter value obtained after the moment is updated; Yes The bias-corrected second moment estimates; Yes The first-order moment estimate after bias correction; Step 6-4. Finally, set the batch size to 32, and stop training when the validation set loss does not decrease for 10 consecutive rounds using the early stopping method to obtain the trained neural network model.

7. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 6 is characterized in that: The trained neural network model is used to remove the thermal effect of the real-time measurement data to obtain non-standard static data. The static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is set as the standard static data. The similarity between the non-standard static data and the standard static data is calculated as follows: Step 7-1. The temperature of the camera rises continuously after startup, and remains constant after reaching thermal equilibrium. The measured displacement obtained in any 10-minute period after the camera reaches thermal equilibrium and the temperature remains constant is defined as standard static data. The measured displacement obtained in any 10-minute period when the camera does not fully reach thermal equilibrium and the temperature still changes is defined as original measured displacement. The original measured displacement is fitted by the neural network model to obtain the fitted displacement, and the difference between the original measured displacement and the fitted displacement is the non-standard static data. Step 7-2. The similarity between standard static data and non-standard static data is characterized by KL divergence, as follows: Where x is the value of displacement, It represents the probability of the random variable x appearing in the distribution P of the standard static data; It represents the probability of a random variable x occurring in the distribution Q of non-standard static data.

8. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 7, characterized in that: Adjust the parameters of the neural network model, including weight, learning rate, dropout rate, number of neurons, and model training time.

9. The camera thermal image drift compensation method based on multi-feature time series fusion according to claim 8, characterized in that: The mapping relationship between the neural network model parameters and the KL divergence is established, the parameters of the neural network model are adjusted according to the validation set, and a new round of iterative calculation is performed until the final thermal effect fitting prediction model is obtained, as follows: In the model building and training process, the time series data set is divided into training set, test set and validation set in the ratio of 70%, 15% and 15% through cross-validation; When training, enable the automatic parameter tuning algorithm and combine grid search with random search; For the weights, the standard deviation of the He normal distribution is set to [0.5, 2] to explore different initialization distributions; The learning rate is set on a logarithmic scale from 1e-5 to 0.1 for 10 trials; the dropout rate is set from 0 to 1 for 11 trials with an interval of 0.1; The number of neurons in the first fully connected layer increases from 128 to 512 in the order of 128; the number of neurons in the second fully connected layer increases from 64 to 256 in the order of 64; the number of neurons in the third fully connected layer increases from 32 to 128 in the order of 32; Each time the model is iterated, the KL divergence, training set and validation set accuracy are recorded synchronously. If the KL divergence increases, if the update amplitude fluctuates by more than 20% due to unstable weight update, the weight initialization method is switched to Xavier normal distribution; if the learning rate is too large and the validation set loss increases for three consecutive rounds, the learning rate is reduced to 0.5 times the original value; if the difference between the training set accuracy and the validation set accuracy exceeds 15% due to overfitting, the discard rate is increased by 0.1 or the number of neurons in the corresponding layer is reduced by 20%; The automatic parameter adjustment algorithm continuously searches until the KL divergence converges to the minimum value, and the final thermal effect fitting prediction model is obtained.

10. A camera-activated thermal effect image drift compensation system, used to implement the camera thermal image drift compensation method based on multi-feature time series fusion as described in any one of claims 1 to 9, characterized in that: include: The data set construction module is used to receive and preprocess the time, camera temperature, and ambient temperature data to form a time series data set of drift caused by the thermal effect of camera startup, and divide the time series data set into a training set, a test set, and a validation set; The model building module is used to build a neural network model, and use the neural network model to fit the time-drift curve to obtain the temperature drift law from the camera startup to the thermal equilibrium stage; The training module is used to train the neural network model using the training set, taking the mean square error as the training objective function, and using the adaptive moment estimation algorithm to dynamically adjust the parameter update step size, and obtain the trained neural network model after iterative calculation; A similarity calculation module is used to use the trained neural network model to remove the thermal effect of the real-time measurement data to obtain non-standard static data. The static test data when the camera reaches thermal equilibrium and the ambient temperature is stable during the standard period is set as the standard static data, and the similarity between the non-standard static data and the standard static data is calculated; The parameter optimization and output module is used to establish the mapping relationship between the parameters of the neural network model and the similarity, adjust the parameters of the neural network model according to the validation set, and perform a new round of iterative calculation until the final thermal effect fitting prediction model is obtained; The image drift compensation module is used to use the final thermal effect fitting prediction model to predict the image drift amount and eliminate it, so as to compensate for the image drift caused by the thermal effect when the camera is started.

Images