Deep learning image data intelligent supervision system and method

Abstract

The application discloses a kind of deep learning's image data intelligent supervision system and method, it is related to image processing technical field, the present method includes the following steps: combination deployment high-temperature-resistant traditional camera and event camera, calibration obtains parameter, sets up acquisition parameter and carries out foundation pit image multimodal acquisition;To the acquisition data carries out time domain weighted average, asynchronous event fusion, calculates optical flow using algorithm and combines kalman filter tracking ROI;Build TSR-WGAN model architecture, construct sample set training model, input pre-processing sequence and carry out inference output correction image;The subset size of DIC algorithm is configured, such as parameter, and the displacement field and strain field data generated after analyzing correction image sequence are analyzed;Calculate absolute displacement and other indicators, set each index threshold, data over threshold triggers foundation pit abnormal early warning.The present application can effectively improve the situation that traditional image data supervision scheme is excessively dependent on environmental parameters in high-temperature scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a deep learning-based intelligent image data monitoring system and method. Background Technology

[0002] Safety monitoring of foundation pit projects is crucial for urban construction. Image data monitoring, by capturing structural surface displacement and strain, provides key evidence for risk assessment. Traditional methods, such as digital image correlation (DIC), rely on highly stable image sequences, but are susceptible to interference from factors such as air turbulence and sudden changes in illumination under high-temperature environments, leading to blurred images and increased noise. With the development of sensor technology, multimodal acquisition (such as a combination of traditional cameras and event cameras) can fuse temporal frames and high-frequency disturbance information, and combine deep learning models (such as generative adversarial networks) to intelligently correct complex disturbances, becoming an important direction for solving the monitoring challenges in high-temperature scenarios. This type of technology improves data reliability through spatiotemporal information complementarity and algorithm optimization, providing a new path for structural health monitoring.

[0003] Existing high-temperature scene image monitoring solutions rely excessively on environmental parameter inputs, requiring manual configuration of conditions such as temperature and light intensity to adjust HDR camera exposure parameters or filter algorithm weights. Parameter fluctuations in dynamic environments can easily lead to monitoring failures. Meanwhile, dedicated high-temperature HDR equipment relies on special optical components, resulting in high hardware costs and complex maintenance. Real-time preprocessing algorithms (such as global histogram equalization) are computationally intensive, making it difficult to meet the requirements for sub-second response times. Furthermore, traditional single-modal cameras have poor adaptability to extreme lighting conditions such as strong light / backlight, and insufficient imaging dynamic range, requiring environmental parameter compensation, further exacerbating the system's reliance on manual intervention. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based intelligent image data monitoring system and method to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a deep learning-based intelligent image data monitoring method, the method comprising the following steps:

[0006] Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters to perform multimodal acquisition of foundation pit images;

[0007] Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate optical flow using an algorithm, and combine it with Kalman filtering to track the ROI;

[0008] Step 3: Build the TSR-WGAN model architecture, construct a sample set to train the model, input the preprocessed sequence for inference, and output the corrected image;

[0009] Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, and analyze the corrected image sequence to generate displacement field and strain field data;

[0010] Step 5: Calculate the absolute displacement and displacement change rate indicators, set the threshold for each indicator, and trigger an early warning for foundation pit anomalies when the data exceeds the threshold.

[0011] In step 1, a combination of a high-temperature resistant conventional camera and an event camera is used to acquire multimodal image data of the foundation pit;

[0012] Traditional high-temperature resistant cameras need to cover the monitoring environment temperature range, be equipped with imaging elements with a resolution of no less than a certain resolution, have a frame rate of f frames per second, and support external trigger synchronous acquisition function to acquire high-resolution images;

[0013] Event cameras are characterized by low latency and high dynamic range, and are used to capture high-speed disturbance information. Instead of acquiring images at a fixed frame rate, event cameras respond independently to brightness changes (such as sudden changes in brightness) for each pixel, and output an "event stream". Each event contains coordinates (x, y), timestamp t, and polarity (increase / decrease in brightness).

[0014] High-temperature resistant conventional cameras and event cameras are mounted on the same synchronous bracket, maintaining a distance from the foundation pit within a set range;

[0015] The high-temperature resistant camera and the event camera were calibrated using a calibration board to obtain internal and external parameters, achieve sub-pixel level spatial coordinate transformation accuracy, and provide accurate geometric mapping relationships for subsequent data processing.

[0016] The high-temperature resistant traditional camera performs time-series acquisition at a rate of f frames per second, with each acquisition consisting of no less than (2N+1) frames; where N represents the half-length parameter of the time-series processing window, which is used for subsequent time-series feature extraction.

[0017] The automatic exposure adjustment function generates grayscale images based on ambient lighting.

[0018] The event camera asynchronously records all event streams within the time window and aligns them with the traditional frames acquired by the high-temperature resistant conventional camera to ensure the temporal consistency of multimodal data.

[0019] In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of image acquired by a high-temperature resistant conventional camera with subpixel precision;

[0020] For each ROI, construct a time window of length (2N+1) frames;

[0021] Perform a time-domain weighted average on the images within the time window, with weights w. i Follows a Gaussian distribution: w i =e^-i2 / 2σ 2 ;

[0022] Where i represents the index of the image frame within the time window, i∈[-N,N]; w i σ represents the weight coefficient of the i-th frame image within the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay rate of neighboring frames;

[0023] Through the weighted fusion formula:

[0024] ,

[0025] Enhance the stability of the center frame and suppress random perturbations; finally, convert the fused sequence into the model input format.

[0026] Where (x,y) represents the coordinates of a pixel on the image plane; I i (x,y) represents the pixel value of the i-th frame within the time window at coordinates (x,y); TWA (x,y) represents the pixel value at coordinates (x,y) after time-domain weighted averaging.

[0027] Within the same time window as a traditional camera (the duration corresponds to the acquisition cycle of (2N+1) frames), the event stream is accumulated according to spatial location: for each pixel (x,y), the total number of events occurring within the time window is counted to form an "event density map" E(x,y);

[0028] The physical meaning of E(x,y) is the intensity of dynamic disturbance at pixel position (x,y). The more events there are, the more drastic the brightness change in the region within the window time (such as displacement, vibration, etc.).

[0029] Events from the event camera within the time window are accumulated into an event density map E(x,y), which is then spatiotemporally weighted and fused with the TWA-processed image; the fusion formula is I. fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0030] Among them, I fusion (x,y) represents the pixel value at coordinate (x,y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors such as on-site vibration intensity to achieve complementary advantages of multimodal data and improve data quality; the value of E(x,y) at coordinate (x,y) reflects the density of events occurring at that location within the time window; max(E) represents the maximum value of all pixel values ​​in the event density map E(x,y), used to normalize the event density map;

[0031] The optical flow algorithm is used to calculate dense optical flow in the initial ROI region. When the optical flow distribution exceeds a preset threshold, it indicates that the target may have moved out of the original ROI. Combining the optical flow results with the ROI position in the previous frame, the center and size of the new ROI are estimated by the Kalman filter algorithm to achieve smooth tracking. When the deviation between the Kalman prediction and the TWA calibration mark displacement exceeds the preset deviation threshold ε, short-term field of view calibration is automatically triggered to correct the drift of intrinsic and extrinsic parameters, ensuring the continuous and accurate positioning of the ROI and providing stable input for subsequent processing.

[0032] In step 3, the TSR-WGAN model is adopted, and the generator network uses the fused image sequence I of (2N+1) frames. fusion As input, multi-scale feature extraction is performed through multiple convolutions with a feature dimension of [d1, d2, d3], where d1, d2, and d3 are incremental feature channel parameters used to capture multi-scale perturbation features. Cascaded multi-scale deformable convolutions (kernel size [k1, k2, k3]; k1, k2, and k3 are incremental spatial receptive field parameters) are used to correct geometric distortions. Finally, 3D convolutions (kernel size [k...]) are employed. w ,k h ,k t ];k t The network uses a time dimension span parameter and a time attention mechanism to distinguish between real motion and turbulent disturbances; it employs an encoding / decoding structure containing n residual blocks to preserve global and local image features and output a corrected image; where n represents the network depth parameter.

[0033] The discriminator network adopts a multi-scale discriminative architecture. The global discriminator is based on the PatchGAN structure, with an input size of [W,H], and judges the overall authenticity of the image. The local discriminator samples image patches of size [w,h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and each of the previous m frames, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter.

[0034] Perturbation samples of varying intensities are generated through physical simulation or algorithms, and a training set containing M pairs of real perturbation samples, a validation set containing Q pairs of samples, and a test set containing P pairs of samples are constructed; where M, Q, and P represent positive integers set according to the task complexity.

[0035] Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is conducted; among them, L WG L represents the WGAN-GP loss, which ensures training stability through a gradient penalty term; Pe Representing perceptual loss, it extracts features based on a pre-trained visual network and measures the semantic difference between generated and real images; L TVλ1 and λ2 represent the total variation loss, which suppresses noise in the generated image; λ1 and λ2 represent the weight coefficients of the corresponding loss terms.

[0036] The system takes a (2N+1) frame preprocessed image sequence as input, performs inference under GPU acceleration, and outputs the corrected center frame image.

[0037] In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements.

[0038] The overlap rate of adjacent subsets is represented by r%, which controls the calculation density of the displacement field.

[0039] The iterative convergence threshold is γ to ensure the accuracy of displacement calculation;

[0040] For the image sequence after deep learning model correction, the pixel grayscale change is analyzed by the DIC algorithm. Based on the conversion coefficient determined by camera calibration and the actual length corresponding to 1 pixel, displacement field and strain field data are generated.

[0041] In step 5, the monitoring indicators are calculated, including absolute displacement (daily cumulative displacement, reflecting the overall deformation of the structure), displacement change rate (displacement change rate per unit time, monitoring the deformation rate), and strain in key areas (assessing the stress state of materials), and the condition of the foundation pit is analyzed.

[0042] For each monitoring indicator, a historical normal sliding window dataset of length W is maintained using the generated displacement field and strain field data, etc.

[0043] Historical normal sliding window data refers to data collected when the foundation pit structure is in a stable state, ensuring the reliability of the baseline; the median med(t) and median absolute deviation MAD are calculated within the window, using the formula: MAD(t) = median(|x j -med(t)|);

[0044] Where med(t) represents the median of the monitored indicators within the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median deviation of the data within the sliding window from the median med(t); x j This represents the data of the j-th frame within the sliding window;

[0045] The median and median absolute deviation can robustly describe the central tendency and dispersion of data, avoiding the influence of extreme values;

[0046] The event density E(t) is obtained by using the disturbance information captured by the event camera to reflect the intensity of vibration / disturbance on site; the event density E(t) is normalized to g(t)=E(t) / max(E); the linear correction function is defined as: f(g(t))=1+β(g(t)-0.5);

[0047] Where E(t) represents the sum of global event densities after accumulating all event streams collected by the event camera within the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which can be set according to engineering experience; the function can automatically adjust the correction amplitude according to the intensity of environmental disturbance. When the environmental disturbance is strong (g(t)>0.5), the function value is greater than 1, and the threshold will be automatically relaxed; when the environment is stable (g(t)<0.5), the function value is less than 1, and the threshold is tightened, thereby improving the adaptability of the monitoring system to different environmental conditions.

[0048] Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t) = med(t) + kMAD(t) × f(g(t)); where k represents the statistical confidence factor, which can be set uniformly or differentiated slightly for different indicators. This formula allows the threshold to automatically adjust as historical data changes and the intensity of environmental disturbances vary. When environmental disturbances are strong, the threshold is relaxed to avoid false alarms caused by environmental interference; when the environment is stable, the threshold is tightened to improve sensitivity to abnormal changes in the foundation pit.

[0049] To further suppress sporadic noise and ensure the accuracy of early warning, an alarm is triggered when the time exceeds T(t) for Z1 consecutive frames or Z2 cumulative times, reminding relevant personnel to take measures to ensure the safety of foundation pit construction.

[0050] A deep learning-based intelligent monitoring system for image data includes a data acquisition module, a fusion tracking module, an image correction module, a displacement analysis module, and an early warning decision module.

[0051] The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters for multimodal acquisition of foundation pit images. The fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, calculate optical flow using algorithms, and track the ROI using Kalman filtering. The image correction module is used to build a TSR-WGAN model architecture, construct a sample set to train the model, input a preprocessed sequence for inference, and output corrected images. The displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data. The early warning decision module is used to calculate absolute displacement and displacement change rate indices, set thresholds for each index, and trigger an early warning of foundation pit anomalies when the data exceeds the threshold.

[0052] The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit, and a parameter configuration unit;

[0053] The dual-mode acquisition unit is used to deploy a high-temperature resistant camera and an event camera to achieve multimodal image acquisition; the calibration synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

[0054] The fusion tracking module includes an ROI extraction unit, a temporal fusion unit, and an optical flow tracking unit;

[0055] The ROI extraction unit is used to extract monitoring point regions from each frame of image; the temporal fusion unit is used to perform Gaussian weighted average and event density map fusion; the optical flow tracking unit is used to dynamically track ROIs through optical flow algorithm and Kalman filter, triggering calibration correction.

[0056] The image correction module includes a feature extraction unit, a distortion correction unit, and a temporal inference unit;

[0057] The feature extraction unit is used to extract perturbation features through multi-channel, multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion using cascaded deformable convolution; the temporal inference unit is used to distinguish between motion and turbulence using 3D convolution and attention mechanisms, and output a corrected image.

[0058] The displacement analysis module includes a DIC configuration unit and a strain calculation unit;

[0059] The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes.

[0060] The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit, and an alarm triggering unit;

[0061] The index calculation unit is used to calculate absolute displacement, rate of change, and strain; the baseline maintenance unit is used to maintain a historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold; the alarm triggering unit is used to trigger an abnormal alarm when the triggering conditions exceed the threshold.

[0062] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention uses a low-cost combination of "high-temperature resistant traditional camera + event camera" to replace a single high-end HDR camera: the traditional camera is selected as an industrial-grade high-temperature resistant model, and the resolution only needs to meet the basic imaging requirements, and the cost is far lower than that of a dedicated high-temperature HDR camera; this invention uses a calibration board to uniformly calibrate the internal and external parameters of the two cameras and generate a sub-pixel level conversion matrix, without the need for manual adjustment of the structured light system or HDR camera exposure parameters. The calibration process can be executed with one click through the program, and the technical requirements for operators are low; this invention uses TSR-WGAN to generate perturbation samples under different temperature and lighting conditions through physical simulation for training. The model can adaptively correct image distortion (such as blurring caused by air turbulence) and uneven lighting caused by high temperature, without the need for real-time input of environmental parameters such as temperature and light. Attached Figure Description

[0063] Figure 1 This is a schematic diagram illustrating the steps of a deep learning-based intelligent image data monitoring method according to the present invention.

[0064] Figure 2 This is a flowchart illustrating a deep learning-based intelligent image data monitoring system according to the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example: Figures 1-2 As shown, this invention provides a technical solution: a deep learning-based intelligent monitoring method for image data, which includes the following steps:

[0067] Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters to perform multimodal acquisition of foundation pit images;

[0068] Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate optical flow using an algorithm, and combine it with Kalman filtering to track the ROI;

[0069] Step 3: Build the TSR-WGAN model architecture, construct a sample set to train the model, input the preprocessed sequence for inference, and output the corrected image;

[0070] Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, and analyze the corrected image sequence to generate displacement field and strain field data;

[0071] Step 5: Calculate the absolute displacement and displacement change rate indicators, set the threshold for each indicator, and trigger an early warning for foundation pit anomalies when the data exceeds the threshold.

[0072] In step 1, a combination of a high-temperature resistant conventional camera and an event camera is used to acquire multimodal image data of the foundation pit;

[0073] Traditional high-temperature resistant cameras need to cover the monitoring environment temperature range, be equipped with imaging elements with a resolution of no less than a certain resolution, have a frame rate of f frames per second, and support external trigger synchronous acquisition to acquire high-resolution images; event cameras have the characteristics of low latency and high dynamic range to capture high-speed disturbance information.

[0074] High-temperature resistant conventional cameras and event cameras are mounted on the same synchronous bracket, maintaining a distance from the foundation pit within a set range;

[0075] The high-temperature resistant camera and the event camera were calibrated using a calibration board to obtain internal and external parameters, achieve sub-pixel level spatial coordinate transformation accuracy, and provide accurate geometric mapping relationships for subsequent data processing.

[0076] The high-temperature resistant traditional camera performs time-series acquisition at a rate of f frames per second, with each acquisition consisting of no less than (2N+1) frames; where N represents the half-length parameter of the time-series processing window, which is used for subsequent time-series feature extraction.

[0077] The automatic exposure adjustment function generates grayscale images based on ambient lighting.

[0078] The event camera asynchronously records all event streams within the time window and aligns them with the traditional frames acquired by the high-temperature resistant conventional camera to ensure the temporal consistency of multimodal data.

[0079] In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of image acquired by a high-temperature resistant conventional camera with subpixel precision;

[0080] For each ROI, construct a time window of length (2N+1) frames;

[0081] Perform a time-domain weighted average on the images within the time window, with weights w. i Follows a Gaussian distribution: w i =e^-i 2 / 2σ 2 ;

[0082] Where i represents the index of the image frame within the time window, i∈[-N,N]; w i σ represents the weight coefficient of the i-th frame image within the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay rate of neighboring frames;

[0083] Through the weighted fusion formula:

[0084] ,

[0085] Enhance the stability of the center frame and suppress random perturbations; finally, convert the fused sequence into the model input format.

[0086] Where (x,y) represents the coordinates of a pixel on the image plane; I i (x,y) represents the pixel value of the i-th frame within the time window at coordinates (x,y); TWA (x,y) represents the pixel value at coordinates (x,y) after time-domain weighted averaging.

[0087] Events from the event camera within the time window are accumulated into an event density map E(x,y), which is then spatiotemporally weighted and fused with the TWA-processed image; the fusion formula is I. fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0088] Among them, I fusion (x,y) represents the pixel value at coordinate (x,y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors such as on-site vibration intensity to achieve complementary advantages of multimodal data and improve data quality; the value of E(x,y) at coordinate (x,y) reflects the density of events occurring at that location within the time window; max(E) represents the maximum value of all pixel values ​​in the event density map E(x,y), used to normalize the event density map;

[0089] The optical flow algorithm is used to calculate dense optical flow in the initial ROI region. When the optical flow distribution exceeds a preset threshold, it indicates that the target may have moved out of the original ROI. Combining the optical flow results with the ROI position in the previous frame, the center and size of the new ROI are estimated by the Kalman filter algorithm to achieve smooth tracking. When the deviation between the Kalman prediction and the TWA calibration mark displacement exceeds the preset deviation threshold ε, short-term field of view calibration is automatically triggered to correct the drift of intrinsic and extrinsic parameters, ensuring the continuous and accurate positioning of the ROI and providing stable input for subsequent processing.

[0090] In step 3, the TSR-WGAN model is adopted, and the generator network uses the fused image sequence I of (2N+1) frames. fusionAs input, multi-scale feature extraction is performed through multiple convolutions with a feature dimension of [d1, d2, d3], where d1, d2, and d3 are incremental feature channel parameters used to capture multi-scale perturbation features. Cascaded multi-scale deformable convolutions (kernel size [k1, k2, k3]; k1, k2, and k3 are incremental spatial receptive field parameters) are used to correct geometric distortions. Finally, 3D convolutions (kernel size [k...]) are employed. w ,k h ,k t ];k t The network uses a time dimension span parameter and a time attention mechanism to distinguish between real motion and turbulent disturbances; it employs an encoding / decoding structure containing n residual blocks to preserve global and local image features and output a corrected image; where n represents the network depth parameter.

[0091] The discriminator network adopts a multi-scale discriminative architecture. The global discriminator is based on the PatchGAN structure, with an input size of [W,H], and judges the overall authenticity of the image. The local discriminator samples image patches of size [w,h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and each of the previous m frames, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter.

[0092] Perturbation samples of varying intensities are generated through physical simulation or algorithms, and a training set containing M pairs of real perturbation samples, a validation set containing Q pairs of samples, and a test set containing P pairs of samples are constructed; where M, Q, and P represent positive integers set according to the task complexity.

[0093] Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is conducted; among them, L WG L represents the WGAN-GP loss, which ensures training stability through a gradient penalty term; Pe Representing perceptual loss, it extracts features based on a pre-trained visual network and measures the semantic difference between generated and real images; L TV λ1 and λ2 represent the total variation loss, which suppresses noise in the generated image; λ1 and λ2 represent the weight coefficients of the corresponding loss terms.

[0094] The system takes a (2N+1) frame preprocessed image sequence as input, performs inference under GPU acceleration, and outputs the corrected center frame image.

[0095] In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements.

[0096] The overlap rate of adjacent subsets is represented by r%, which controls the calculation density of the displacement field.

[0097] The iterative convergence threshold is γ to ensure the accuracy of displacement calculation;

[0098] For the image sequence after deep learning model correction, the pixel grayscale change is analyzed by the DIC algorithm. Based on the conversion coefficient determined by camera calibration and the actual length corresponding to 1 pixel, displacement field and strain field data are generated.

[0099] In step 5, the monitoring indicators are calculated, including absolute displacement (daily cumulative displacement, reflecting the overall deformation of the structure), displacement change rate (displacement change rate per unit time, monitoring the deformation rate), and strain in key areas (assessing the stress state of materials), and the condition of the foundation pit is analyzed.

[0100] For each monitoring indicator, a historical normal sliding window dataset of length W is maintained using the generated displacement field and strain field data, etc.

[0101] Historical normal sliding window data refers to data collected when the foundation pit structure is in a stable state, ensuring the reliability of the baseline; the median med(t) and median absolute deviation MAD are calculated within the window, using the formula: MAD(t) = median(|x j -med(t)|);

[0102] Where med(t) represents the median of the monitored indicators within the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median deviation of the data within the sliding window from the median med(t); x j This represents the data of the j-th frame within the sliding window;

[0103] The median and median absolute deviation can robustly describe the central tendency and dispersion of data, avoiding the influence of extreme values;

[0104] The event density E(t) is obtained by using disturbance information captured by the event camera to reflect the intensity of vibration / disturbance on site. The event density E(t) is normalized to g(t) = E(t) / max(E); a linear correction function is defined as f(g(t)) = 1 + β(g(t) - 0.5).

[0105] Where E(t) represents the sum of global event densities after accumulating all event streams collected by the event camera within the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which can be set according to engineering experience; the function can automatically adjust the correction amplitude according to the intensity of environmental disturbance. When the environmental disturbance is strong (g(t)>0.5), the function value is greater than 1, and the threshold will be automatically relaxed; when the environment is stable (g(t)<0.5), the function value is less than 1, and the threshold is tightened, thereby improving the adaptability of the monitoring system to different environmental conditions.

[0106] Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t) = med(t) + kMAD(t) × f(g(t)); where k represents the statistical confidence factor, which can be set uniformly or differentiated slightly for different indicators. This formula allows the threshold to automatically adjust as historical data changes and the intensity of environmental disturbances vary. When environmental disturbances are strong, the threshold is relaxed to avoid false alarms caused by environmental interference; when the environment is stable, the threshold is tightened to improve sensitivity to abnormal changes in the foundation pit.

[0107] To further suppress sporadic noise and ensure the accuracy of early warning, an alarm is triggered when the time exceeds T(t) for Z1 consecutive frames or Z2 cumulative times, reminding relevant personnel to take measures to ensure the safety of foundation pit construction.

[0108] A deep learning-based intelligent monitoring system for image data includes a data acquisition module, a fusion tracking module, an image correction module, a displacement analysis module, and an early warning decision module.

[0109] The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters for multimodal acquisition of foundation pit images. The fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, calculate optical flow using algorithms, and track the ROI using Kalman filtering. The image correction module is used to build a TSR-WGAN model architecture, construct a sample set to train the model, input a preprocessed sequence for inference, and output corrected images. The displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data. The early warning decision module is used to calculate absolute displacement and displacement change rate indices, set thresholds for each index, and trigger an early warning of foundation pit anomalies when the data exceeds the threshold.

[0110] The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit, and a parameter configuration unit;

[0111] The dual-mode acquisition unit is used to deploy a high-temperature resistant camera and an event camera to achieve multimodal image acquisition; the calibration synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

[0112] The fusion tracking module includes an ROI extraction unit, a temporal fusion unit, and an optical flow tracking unit;

[0113] The ROI extraction unit is used to extract monitoring point regions from each frame of image; the temporal fusion unit is used to perform Gaussian weighted average and event density map fusion; the optical flow tracking unit is used to dynamically track ROIs through optical flow algorithm and Kalman filter, triggering calibration correction.

[0114] The image correction module includes a feature extraction unit, a distortion correction unit, and a temporal inference unit;

[0115] The feature extraction unit is used to extract perturbation features through multi-channel, multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion using cascaded deformable convolution; the temporal inference unit is used to distinguish between motion and turbulence using 3D convolution and attention mechanisms, and output a corrected image.

[0116] The displacement analysis module includes a DIC configuration unit and a strain calculation unit;

[0117] The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes.

[0118] The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit, and an alarm triggering unit;

[0119] The index calculation unit is used to calculate absolute displacement, rate of change, and strain; the baseline maintenance unit is used to maintain a historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold; the alarm triggering unit is used to trigger an abnormal alarm when the triggering conditions exceed the threshold.

[0120] In this embodiment, at the monitoring site of a subway foundation pit project, a high-temperature resistant traditional camera (operating temperature -20℃ to 120℃, resolution 1920×1080, frame rate 30fps) and an event camera (pixel resolution 346×260) were jointly mounted on a seismic synchronous support. The support was 15 meters from the edge of the foundation pit, with an elevation angle of 30°. The traditional camera was equipped with an autofocus lens (focal length 16mm, aperture F1.8), and the event camera used a wide-angle lens (field of view 120°). Using a 9×12 checkerboard calibration board, 8 calibration points were evenly distributed at the foundation pit site, and 50 sets of image pairs were collected. The intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (rotation matrix, translation vector) of the traditional camera were calculated using the Zhang Zhengyou calibration method. For the event camera, a spatiotemporal calibration algorithm based on event flow was used to map the event coordinates to the traditional camera coordinate system, achieving sub-pixel level (±0.3 pixels) alignment accuracy.

[0121] Step 1: A traditional camera captures time-series images at 30fps, continuously capturing 31 frames each time (N=15, time window approximately 1 second).

[0122] The event camera asynchronously records all pixel brightness change events with an event resolution of 1 μs, generating a four-tuple data stream (x, y, t, p);

[0123] Step 2: Time-domain weighted fusion and dynamic ROI tracking;

[0124] Extract all events within ±167ms (corresponding to 1 / 30 second) of the timestamp of the center frame of a traditional camera; for each pixel (x,y), count the number of events C(x,y) and normalize it; apply 3×3 Gaussian kernel smoothing to E(x,y) to suppress salt-and-pepper noise;

[0125] Time-domain weighted average (TWA): For a traditional image sequence Ii(x,y) (i∈[-15,15]), calculate:

[0126] ,

[0127] w i =e^-i 2 / 2σ 2 ;

[0128] Spacetime Fusion: I fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0129] ROI Tracking: Initialize the ROI as a 200×200 pixel key point area of ​​the foundation pit support structure; calculate the optical flow field within the ROI; when the average optical flow amplitude is >5 pixels, trigger Kalman filtering to predict the new ROI position; if the predicted position deviates from the TWA-corrected position by >3 pixels, start short-time field of view calibration (time <0.5 seconds).

[0130] Step 3: TSR-WGAN model construction and image correction;

[0131] Generator Network: Input layer: 31 frames × 200 × 200 × 1 fused image sequence; Feature extraction: 3 layers of 3D convolution (3×3×3 kernel, stride 1, feature dimensions [64, 128, 256]); Deformable convolution: 2-level deformable convolution module to correct geometric distortion caused by high temperature; Temporal attention: self-attention mechanism based on 3D convolution, weight dimensions [31, 200, 200]; Encoding / decoding structure: 8 residual blocks (each containing 2 3×3 convolutions + ReLU), outputting the corrected center frame.

[0132] Discriminator Network: Global Discriminator: PatchGAN structure, input 256×256 image patch, output 32×32 realism score; Local Discriminator: samples 64×64 image patch, uses dilation convolution (dilation=2) to enhance detail perception; Temporal Consistency: input the current frame and the three frames before and after, and use 3D convolution (kernel size 3×3×3) to determine temporal coherence;

[0133] Training process:

[0134] Dataset: Physical simulation: 1000 pairs of disturbance-realistic images under different temperature fields were generated using COMSOL thermal flux simulation; Field data collection: 1000 pairs of real data were collected at different construction stages of the foundation pit;

[0135] Loss function: L=L WG +λ1L Pe +λ2L TV ;

[0136] Step 4: DIC algorithm parameter configuration and displacement field analysis;

[0137] Parameter settings: Subset size: 21×21 pixels (corresponding to actual size 10.5mm×10.5mm); Subset overlap rate: 60% (step size 8 pixels); Iteration convergence threshold: γ=1e-4; Displacement calculation method: Least squares method based on Newton-Raphson iteration.

[0138] Calculation process: For the corrected image sequence, with frame t as a reference, calculate the displacement field of frame t+Δt; use subpixel interpolation (bicubic interpolation) to improve the displacement accuracy to 0.01 pixels; convert pixel displacement into actual physical displacement according to camera calibration parameters (1 pixel = 0.5 mm); calculate the strain field through displacement field gradient;

[0139] Step 5: Adaptive threshold early warning mechanism;

[0140] Historical baseline maintenance: window size W = 2000 frames (approximately 70 minutes of continuous monitoring data); only manually confirmed data on the stability of the foundation pit are included; when new data is added, a sliding window strategy is used to discard the oldest data.

[0141] Statistical feature calculation: For each monitoring index (absolute displacement D, displacement change rate v, principal strain εmax), calculate: med(t); calculate global event density, perform normalization and correction; construct adaptive threshold;

[0142] An alert is triggered if Z1=3 consecutive frames exceed the threshold, or if Z2=10 cumulative times exceed the threshold.

[0143] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A deep learning-based intelligent monitoring method for image data, characterized in that: The method includes the following steps: Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters to perform multimodal acquisition of foundation pit images; Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate optical flow using an algorithm, and combine it with Kalman filtering to track the ROI; Step 3: Build the TSR-WGAN model architecture. Generate perturbation samples of different intensities through physical simulation or algorithms to build a sample set to train the model. Input the preprocessed sequence for inference, and the model adaptively outputs the corrected image. Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, and analyze the corrected image sequence to generate displacement field and strain field data; Step 5: Calculate the absolute displacement and displacement change rate indicators, set the threshold for each indicator, and trigger an early warning for foundation pit anomalies when the data exceeds the threshold. In step 1, a combination of a high-temperature resistant conventional camera and an event camera is used to acquire multimodal image data of the foundation pit; High-temperature resistant conventional cameras operate within a temperature range covering the monitored ambient temperature, are equipped with imaging elements of a resolution no lower than a specific resolution, and have a frame rate of f frames per second; event cameras are used to capture disturbance information. High-temperature resistant conventional cameras and event cameras are mounted on the same synchronous bracket, maintaining a distance from the foundation pit within a set range; The high-temperature resistant camera and the event camera were calibrated using a calibration board to obtain internal and external parameters and achieve sub-pixel level spatial coordinate transformation accuracy. The high-temperature resistant conventional camera performs time-series acquisition at a rate of f frames per second, with each acquisition consisting of no less than (2N+1) frames; where N represents the half-length parameter of the time-series processing window. The automatic exposure adjustment function generates grayscale images based on ambient lighting. The event camera asynchronously records all event streams within the time window and aligns them with the traditional frames captured by the high-temperature resistant conventional camera.

2. The deep learning-based intelligent image data monitoring method according to claim 1, characterized in that: In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of image acquired by a high-temperature resistant conventional camera with subpixel precision; For each ROI, construct a time window of length (2N+1) frames; Perform a time-domain weighted average on the images within the time window, with weights w. i Follows a Gaussian distribution: w i =e^-i 2 / 2σ 2 ; Where i represents the index of the image frame within the time window, i∈[-N,N]; w i σ represents the weight coefficient of the i-th frame image within the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay rate of neighboring frames; Through the weighted fusion formula: ， Enhance center frame stability and suppress random disturbances; Where (x,y) represents the coordinates of a pixel on the image plane; I i (x,y) represents the pixel value of the i-th frame within the time window at coordinates (x,y); TWA (x,y) represents the pixel value at coordinates (x,y) after time-domain weighted averaging. Events from the event camera within the time window are accumulated into an event density map E(x,y), which is then spatiotemporally weighted and fused with the TWA-processed image; the fusion formula is I. fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E); Among them, I fusion (x,y) represents the pixel value at coordinates (x,y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors; max(E) represents the maximum value of all pixel values ​​in the event density map E(x,y), which is used to normalize the event density map; The optical flow algorithm is used to calculate the dense optical flow in the initial ROI region. When the optical flow distribution exceeds the preset threshold, it indicates that the target has moved out of the original ROI. Combining the optical flow result and the ROI position in the previous frame, the center and size of the new ROI are estimated by the Kalman filter algorithm. When the deviation between the Kalman prediction and the TWA calibration mark displacement exceeds the preset deviation threshold ε, short-time field of view calibration is automatically triggered to correct the drift of intrinsic and extrinsic parameters.

3. The deep learning-based intelligent image data monitoring method according to claim 2, characterized in that: In step 3, the TSR-WGAN model is adopted, and the generator network uses the fused image sequence I of (2N+1) frames. fusion As input, multi-scale feature extraction is performed through multi-layer convolution, with feature dimensions [d1, d2, d3], where d1, d2, and d3 are incremental feature channel parameters used to capture multi-scale perturbation features; cascaded multi-scale deformable convolution is used to correct geometric distortion; 3D convolution and temporal attention mechanisms are used to distinguish between real motion and turbulent perturbation; an encoding / decoding structure containing n residual blocks is adopted to preserve global and local image features and output the corrected image; where n represents the network depth parameter; The discriminator network adopts a multi-scale discriminative architecture. The global discriminator is based on the PatchGAN structure, with an input size of [W,H], and judges the overall authenticity of the image. The local discriminator samples image patches of size [w,h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and each of the previous m frames, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter. Perturbation samples of varying intensities are generated through physical simulation or algorithms, and a training set containing M pairs of real perturbation samples, a validation set containing Q pairs of samples, and a test set containing P pairs of samples are constructed; where M, Q, and P represent positive integers set according to the task complexity. Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is conducted; among them, L WG L represents the WGAN-GP loss, which ensures training stability through a gradient penalty term; Pe Representing perceptual loss, it extracts features based on a pre-trained visual network and measures the semantic difference between generated and real images; L TV λ1 and λ2 represent the total variation loss, which suppresses noise in the generated image; λ1 and λ2 represent the weight coefficients of the corresponding loss terms. The image sequence is preprocessed into (2N+1) frames, and the corrected center frame image is output.

4. The deep learning-based intelligent image data monitoring method according to claim 3, characterized in that: In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements. The overlap rate of adjacent subsets is represented by r%, which controls the calculation density of the displacement field. The iterative convergence threshold is γ to ensure the accuracy of displacement calculation; For the image sequence after deep learning model correction, the pixel grayscale change is analyzed by the DIC algorithm. Based on the conversion coefficient determined by camera calibration and the actual length corresponding to 1 pixel, displacement field and strain field data are generated. In step 5, the monitoring indicators are calculated. For each monitoring indicator, a historical normal sliding window dataset of length W is maintained using the generated displacement field and strain field data. The historical normal sliding window data refers to the data collected when the foundation pit structure is in a stable state. Calculate the median med(t) and median absolute deviation MAD within window W. The formula is: MAD(t) = median(|x j -med(t)|); Where med(t) represents the median of the monitored indicators within the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median deviation of the data within the sliding window from the median med(t); x j This represents the data of the j-th frame within the sliding window W; The event density E(t) is obtained using an event camera; the event density E(t) is normalized to g(t) = E(t) / max(E); a linear correction function is defined: f(g(t)) = 1 + β(g(t) - 0.5); Where E(t) represents the sum of global event densities after accumulating all event streams collected by the event camera within the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which is set according to engineering experience; Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t) = med(t) + kMAD(t) × f(g(t)); where k represents the statistical confidence factor; An alarm is triggered when the time exceeds T(t) for Z1 consecutive frames or Z2 cumulative times.

5. A deep learning-based intelligent image data monitoring system, applied to the deep learning-based intelligent image data monitoring method according to any one of claims 1-4, characterized in that: The system includes a data acquisition module, a fusion tracking module, an image correction module, a displacement analysis module, and an early warning decision module; The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate and acquire parameters, and set acquisition parameters for multimodal acquisition of foundation pit images. The fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, calculate optical flow using algorithms, and track the ROI using Kalman filtering. The image correction module is used to build a TSR-WGAN model architecture, construct a sample set to train the model, input a preprocessed sequence for inference, and output corrected images. The displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data. The early warning decision module is used to calculate absolute displacement and displacement change rate indices, set thresholds for each index, and trigger an early warning of foundation pit anomalies when the data exceeds the threshold.

6. The deep learning-based intelligent image data monitoring system according to claim 5, characterized in that: The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit, and a parameter configuration unit; The dual-mode acquisition unit is used to deploy a high-temperature resistant camera and an event camera to achieve multimodal image acquisition; the calibration synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

7. The deep learning-based intelligent image data monitoring system according to claim 6, characterized in that: The fusion tracking module includes an ROI extraction unit, a temporal fusion unit, and an optical flow tracking unit; The ROI extraction unit is used to extract the monitoring point region from each frame of image; The temporal fusion unit is used to perform Gaussian weighted average and event density map fusion; the optical flow tracking unit is used to dynamically track ROIs using optical flow algorithm and Kalman filter, triggering calibration correction.

8. The deep learning-based intelligent image data monitoring system according to claim 7, characterized in that: The image correction module includes a feature extraction unit, a distortion correction unit, and a temporal inference unit; The feature extraction unit is used to extract perturbation features through multi-channel, multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion using cascaded deformable convolution; the temporal inference unit is used to distinguish between motion and turbulence using 3D convolution and attention mechanisms, and output a corrected image.

9. The deep learning-based intelligent image data monitoring system according to claim 8, characterized in that: The displacement analysis module includes a DIC configuration unit and a strain calculation unit; The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes. The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit, and an alarm triggering unit; The index calculation unit is used to calculate absolute displacement, rate of change, and strain; the baseline maintenance unit is used to maintain a historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold. The alarm triggering unit is used to trigger an abnormal alarm when the triggering conditions exceed the threshold.

Images