Deep learning image data intelligent supervision system and method

Abstract

The invention discloses an intelligent image data supervision system and method for deep learning, and relates to the technical field of image processing, and the method comprises the following steps: carrying out the combined deployment of a high-temperature-resistant conventional camera and an event camera, calibrating and obtaining parameters, and setting collection parameters to carry out the multi-modal collection of a foundation pit image; performing time domain weighted averaging and asynchronous event fusion on the acquired data, calculating optical flow by using an algorithm, and tracking ROI in combination with Kalman filtering; building a TSR-WGAN model architecture, constructing a sample set training model, inputting a preprocessing sequence, reasoning, and outputting a corrected image; configuring parameters such as a subset size of a DIC algorithm, and analyzing the corrected image sequence to generate displacement field and strain field data; and indexes such as absolute displacement are calculated, threshold values of all the indexes are set, and foundation pit abnormity early warning is triggered when data exceed the threshold values. The situation that a traditional image data monitoring scheme excessively depends on environmental parameters in a high-temperature scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep learning-based image data intelligent supervision system and method. Background Art

[0002] Safety monitoring of foundation pit projects is crucial to urban construction. Image data monitoring provides a key basis for risk assessment by capturing surface displacements and strains of structures. Traditional methods such as digital image correlation (DIC) rely on highly stable image sequences, but are susceptible to interference from factors such as air turbulence and sudden changes in illumination in high-temperature environments, resulting in blurred imaging and increased noise. With the development of sensor technology, multimodal acquisition (such as a combination of traditional cameras and event cameras) can fuse time-series frames with high-frequency disturbance information, and combine deep learning models (such as generative adversarial networks) to intelligently correct complex disturbances, becoming an important direction for solving the monitoring challenges of high-temperature scenes. This type of technology improves data reliability through information complementarity in the spatiotemporal domain and algorithm optimization, providing a new path for structural health monitoring.

[0003] Existing image monitoring solutions for high-temperature scenes rely excessively on environmental parameter inputs, requiring manual configuration of conditions such as temperature and light intensity to adjust HDR camera exposure parameters or filter algorithm weights. Parameter fluctuations in dynamic environments can easily lead to monitoring failures. Furthermore, specialized high-temperature HDR equipment relies on specialized optical components, resulting in high hardware costs and complex maintenance. Real-time preprocessing algorithms (such as global histogram equalization) are computationally intensive and struggle to meet sub-second response requirements. Furthermore, traditional single-modality cameras have poor adaptability to extreme lighting conditions such as strong and backlit conditions, resulting in an insufficient dynamic range for imaging and requiring reliance on environmental parameter compensation, further increasing the system's reliance on manual intervention. Summary of the Invention

[0004] The purpose of the present invention is to provide a deep learning image data intelligent supervision system and method to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a deep learning image data intelligent supervision method, the method comprising the following steps:

[0006] Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images;

[0007] Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate the optical flow using an algorithm, and track the ROI in combination with the Kalman filter;

[0008] Step 3: Build the TSR-WGAN model architecture, construct a sample set training model, input the preprocessed sequence for inference, and output the corrected image;

[0009] Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data;

[0010] Step 5: Calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

[0011] In step 1, a combination of a high-temperature resistant traditional camera and an event camera is used to perform multimodal acquisition of foundation pit image data;

[0012] The operating temperature range of high-temperature resistant traditional cameras must cover the monitoring environment temperature, be equipped with an imaging element with a resolution not lower than a specific value, have a frame rate of f frames per second, and support external trigger synchronous acquisition function to obtain high-resolution images;

[0013] Event cameras offer low latency and high dynamic range, making them suitable for capturing high-speed disturbances. Rather than capturing images at a fixed frame rate, event cameras independently respond to brightness changes (such as sudden changes in brightness) for each pixel, outputting an "event stream." Each event contains coordinates (x, y), a timestamp t, and polarity (brightness increase / decrease).

[0014] The high-temperature resistant traditional camera and the event camera are mounted on the same synchronous bracket, and the distance from the foundation pit is kept within the set range;

[0015] Use the calibration plate to calibrate the high-temperature resistant camera and event camera respectively, obtain internal and external parameters, achieve sub-pixel spatial coordinate conversion accuracy, and provide accurate geometric mapping relationships for subsequent data processing;

[0016] The high-temperature resistant traditional camera performs time series acquisition at a rate of f frames per second, and each acquisition is no less than (2N+1) frames; where N represents the half-length parameter of the time series processing window, which is used for subsequent time series feature extraction;

[0017] Generate grayscale images based on ambient lighting through automatic exposure adjustment;

[0018] The event camera asynchronously records all event streams within the time window and time-aligns them with the traditional frames collected by the high-temperature resistant traditional camera to ensure the temporal consistency of multimodal data.

[0019] In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of the image captured by the high-temperature resistant conventional camera with sub-pixel accuracy;

[0020] With each ROI as the center, a time window of length (2N+1) frames is constructed;

[0021] Perform time domain weighted averaging on the images in the time window, with weight w i Follows Gaussian distribution: w i =e^-i2 / 2σ 2 ;

[0022] Where i represents the index of the image frame in the time window, i∈[-N,N]; w i represents the weight coefficient of the i-th frame image in the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay speed of adjacent frames;

[0023] Through the weighted fusion formula:

[0024] ,

[0025] Enhance the stability of the central frame and suppress random disturbances; finally, convert the fused sequence into the model input format;

[0026] Where (x, y) represents the coordinates of the pixel point on the image plane; I i (x, y) represents the pixel value of the i-th frame image at the coordinate (x, y) in the time window; I TWA (x,y) represents the pixel value at coordinate (x,y) after time-domain weighted averaging;

[0027] In the same time window as a traditional camera (the duration corresponds to the acquisition period of (2N+1) frames), the event stream is accumulated by spatial position: for each pixel (x, y), the total number of events that occurred in the time window is counted to form an "event density map" E(x, y);

[0028] The physical meaning of E(x,y) is the dynamic disturbance intensity at the pixel position (x,y). The more events there are, the more drastic the brightness change in the area within the window time (such as displacement, vibration, etc.).

[0029] The events of the event camera in the time window are accumulated into the event density map E(x, y), and the image processed by TWA is subjected to spatiotemporal weighted fusion; the fusion formula is I fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0030] Among them, I fusion (x, y) represents the pixel value at the coordinate (x, y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors such as on-site vibration intensity to achieve complementary advantages of multimodal data and improve data quality; the value of E(x, y) at the coordinate (x, y) reflects the density of events at that location within the time window; max(E) represents the maximum value of all pixel values ​​in the event density map E(x, y), which is used to normalize the event density map;

[0031] The optical flow algorithm is used to calculate dense optical flow for the initial ROI area. When the optical flow distribution exceeds the preset threshold, it indicates that the target may have moved out of the original ROI. Combining the optical flow results and the ROI position of the previous frame, the center and size of the new ROI are estimated through the Kalman filter algorithm to achieve smooth tracking. When the displacement deviation between the Kalman prediction and the TWA calibration marker exceeds the preset deviation threshold ε, the short-term field of view calibration is automatically triggered to correct the drift of internal and external parameters, ensuring the continuous and accurate positioning of the ROI and providing stable input for subsequent processing.

[0032] In step 3, the TSR-WGAN model is used, and the generator network is an image sequence I after (2N+1) frames of fusion fusion As input, multi-scale feature extraction is performed through multi-layer convolution, and the feature dimension is [d1, d2, d3], where d1, d2, and d3 are the increasing feature channel number parameters, which are used to capture multi-scale perturbation features; geometric distortion is corrected using cascaded multi-scale deformation convolution (the convolution kernel size is [k1, k2, k3]; k1, k2, and k3 are the increasing spatial receptive field parameters); with the help of 3D convolution (the convolution kernel size is [k w ,k h ,k t ];k t is the time dimension span parameter) and the temporal attention mechanism to distinguish real motion from turbulent disturbances; a codec structure containing n residual blocks is used to retain the global and local features of the image and output the corrected image; where n represents the network depth parameter;

[0033] The discriminator network adopts a multi-scale discriminant architecture. The global discriminator is based on the PatchGAN structure and takes an input of size [W, H] to judge the overall authenticity of the image. The local discriminator samples image patches of size [w, h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and the m frames before and after it, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter.

[0034] Generate perturbation samples of different intensities through physical simulation or algorithm, and construct a training set consisting of M pairs of perturbed real samples, a validation set of Q pairs of samples, and a test set of P pairs of samples; where M, Q, and P represent positive integers set according to the complexity of the task.

[0035] Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is performed; among them, L WG represents the WGAN-GP loss, which ensures training stability through the gradient penalty term; L Pe represents the perceptual loss, which extracts features based on the pre-trained visual network and measures the semantic difference between the generated image and the real image; L TVRepresents the total variation loss, which suppresses the generation of image noise; λ1 and λ2 represent the weight coefficients of the corresponding loss terms.

[0036] The (2N+1) frame preprocessed image sequence is used as input, inference is performed under GPU acceleration, and the corrected central frame image is output.

[0037] In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements;

[0038] The overlap ratio of adjacent subsets is expressed as r%, which controls the density of displacement field calculation;

[0039] The iterative convergence threshold is γ to ensure the accuracy of displacement calculation;

[0040] For the image sequence corrected by the deep learning model, the DIC algorithm is used to analyze the pixel grayscale changes. Based on the conversion coefficient determined by the camera calibration and the actual length corresponding to 1 pixel, the displacement field and strain field data are generated.

[0041] In step 5, the monitoring indicators are calculated, including absolute displacement (accumulated displacement per day, reflecting the overall deformation of the structure), displacement change rate (displacement change rate per unit time, monitoring deformation speed), key area strain (assessment of material stress state) and other indicators to analyze the foundation pit status;

[0042] For each monitoring indicator, the generated displacement field and strain field data are used to maintain a historical normal sliding window data set of length W;

[0043] The historical normal sliding window data refers to the data collected when the foundation pit structure is in a stable state to ensure the reliability of the baseline; the median med(t) and median absolute deviation MAD are calculated within the window. The calculation formula is: MAD(t)=median(|x j -med(t)|);

[0044] Where med(t) represents the median of the monitoring indicator in the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median of the deviation between the data in the sliding window and the median med(t); x j Represents the data of the jth frame in the sliding window;

[0045] The median and median absolute deviation can robustly describe the central tendency and dispersion of data and avoid the influence of extreme values;

[0046] The disturbance information captured by the event camera is used to obtain the event density E(t), which reflects the on-site vibration / disturbance intensity. The event density E(t) is normalized to g(t) = E(t) / max(E). A linear correction function is defined: f(g(t)) = 1 + β(g(t) - 0.5).

[0047] Among them, E(t) represents the sum of the global event density after the accumulation of all event streams collected by the event camera in the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which can be set according to engineering experience; the function can automatically adjust the correction amplitude according to the intensity of the environmental disturbance. When the environmental disturbance is strong (g(t)>0.5), the function value is greater than 1 and the threshold will be automatically relaxed; when the environment is stable (g(t)<0.5), the function value is less than 1 and the threshold is tightened, thereby improving the adaptability of the monitoring system to different environmental conditions.

[0048] Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t)=med(t)+kMAD(t)×f(g(t)); where k represents the statistical confidence factor, which can be set uniformly or slightly differentiated for different indicators. This formula allows the threshold to automatically adjust as historical data changes and the intensity of environmental disturbances vary. When the environmental disturbance is strong, the threshold is relaxed to avoid false alarms caused by environmental interference; when the environment is stable, the threshold is tightened to increase sensitivity to abnormal changes in the foundation pit.

[0049] In order to further suppress occasional noise and ensure the accuracy of early warning, when Z1 consecutive frames or Z2 cumulative times exceed T(t), an alarm is triggered to remind relevant personnel to take measures to ensure the safety of foundation pit construction.

[0050] A deep learning image data intelligent supervision system, the system includes a data acquisition module, a fusion tracking module, an image correction module, a displacement analysis module and an early warning decision module;

[0051] The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images; the fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, use an algorithm to calculate optical flow and combine it with Kalman filtering to track ROI; the image correction module is used to build a TSR-WGAN model architecture, construct a sample set training model, input a preprocessing sequence for inference and output a corrected image; the displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data; the early warning decision module is used to calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

[0052] The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit and a parameter configuration unit;

[0053] The dual-mode acquisition unit is used to deploy high-temperature resistant cameras and event cameras to achieve multimodal image acquisition; the calibration and synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

[0054] The fusion tracking module includes a ROI extraction unit, a time domain fusion unit and an optical flow tracking unit;

[0055] The ROI extraction unit is used to intercept the monitoring point area from each frame image; the time domain fusion unit is used to perform Gaussian weighted averaging and event density map fusion; the optical flow tracking unit is used to dynamically track the ROI through the optical flow algorithm and Kalman filtering to trigger calibration correction.

[0056] The image correction module includes a feature extraction unit, a distortion correction unit and a temporal reasoning unit;

[0057] The feature extraction unit is used to extract disturbance features through multi-channel multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion by using cascaded deformation convolution; the temporal reasoning unit is used to distinguish motion or turbulence by using 3D convolution and attention mechanism, and output a corrected image.

[0058] The displacement analysis module includes a DIC configuration unit and a strain calculation unit;

[0059] The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes;

[0060] The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit and an alarm triggering unit;

[0061] The indicator calculation unit is used to calculate absolute displacement, rate of change and strain; the baseline maintenance unit is used to maintain the historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold; the alarm triggering unit is used to trigger an abnormal alarm when the trigger condition exceeds the threshold.

[0062] Compared with the prior art, the present invention has the following advantages: the present invention adopts a low-cost combination of "high-temperature resistant traditional camera + event camera" to replace a single high-end HDR camera: the traditional camera selects an industrial-grade high-temperature resistant model with a resolution that meets basic imaging requirements, and the cost is much lower than that of a dedicated high-temperature HDR camera; the present invention uses a calibration plate to uniformly calibrate the internal and external parameters of the dual cameras to generate a sub-pixel conversion matrix, without the need for manual adjustment of the structured light system or HDR camera exposure parameters. The calibration process can be executed with one click through a program, and the technical requirements for the operator are relatively low; the present invention uses TSR-WGAN to generate disturbance samples under different temperature and lighting conditions through physical simulation for training. The model can adaptively correct image distortion caused by high temperature (such as blurring caused by air turbulence) and uneven lighting, without the need to input environmental parameters such as temperature and light in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a schematic diagram of the steps of a deep learning image data intelligent supervision method of the present invention;

[0064] Figure 2 This is a flow chart of a deep learning-based intelligent supervision system for image data according to the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a deep learning image data intelligent supervision method, the method comprising the following steps:

[0067] Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images;

[0068] Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate the optical flow using an algorithm, and track the ROI in combination with the Kalman filter;

[0069] Step 3: Build the TSR-WGAN model architecture, construct a sample set training model, input the preprocessed sequence for inference, and output the corrected image;

[0070] Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data;

[0071] Step 5: Calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

[0072] In step 1, a combination of a high-temperature resistant traditional camera and an event camera is used to perform multimodal acquisition of foundation pit image data;

[0073] High-temperature resistant traditional cameras must have an operating temperature range that covers the monitored ambient temperature, be equipped with an imaging element with a resolution of at least 10 frames per second, and support external triggering for synchronous acquisition to obtain high-resolution images. Event cameras feature low latency and a high dynamic range for capturing high-speed disturbance information.

[0074] The high-temperature resistant traditional camera and the event camera are mounted on the same synchronous bracket, and the distance from the foundation pit is kept within the set range;

[0075] Use the calibration plate to calibrate the high-temperature resistant camera and event camera respectively, obtain internal and external parameters, achieve sub-pixel spatial coordinate conversion accuracy, and provide accurate geometric mapping relationships for subsequent data processing;

[0076] The high-temperature resistant traditional camera performs time series acquisition at a rate of f frames per second, and each acquisition is no less than (2N+1) frames; where N represents the half-length parameter of the time series processing window, which is used for subsequent time series feature extraction;

[0077] Generate grayscale images based on ambient lighting through automatic exposure adjustment;

[0078] The event camera asynchronously records all event streams within the time window and time-aligns them with the traditional frames collected by the high-temperature resistant traditional camera to ensure the temporal consistency of multimodal data.

[0079] In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of the image captured by the high-temperature resistant conventional camera with sub-pixel accuracy;

[0080] With each ROI as the center, a time window of length (2N+1) frames is constructed;

[0081] Perform time domain weighted averaging on the images in the time window, with weight w i Follows Gaussian distribution: w i =e^-i 2 / 2σ 2 ;

[0082] Where i represents the index of the image frame in the time window, i∈[-N,N]; w i represents the weight coefficient of the i-th frame image in the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay speed of adjacent frames;

[0083] Through the weighted fusion formula:

[0084] ,

[0085] Enhance the stability of the central frame and suppress random disturbances; finally, convert the fused sequence into the model input format;

[0086] Where (x, y) represents the coordinates of the pixel point on the image plane; I i (x, y) represents the pixel value of the i-th frame image at the coordinate (x, y) in the time window; I TWA (x,y) represents the pixel value at coordinate (x,y) after time-domain weighted averaging;

[0087] The events of the event camera in the time window are accumulated into the event density map E(x, y), and the image processed by TWA is subjected to spatiotemporal weighted fusion; the fusion formula is I fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0088] Among them, I fusion (x, y) represents the pixel value at the coordinate (x, y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors such as on-site vibration intensity to achieve complementary advantages of multimodal data and improve data quality; the value of E(x, y) at the coordinate (x, y) reflects the density of events at that location within the time window; max(E) represents the maximum value of all pixel values ​​in the event density map E(x, y), which is used to normalize the event density map;

[0089] The optical flow algorithm is used to calculate dense optical flow for the initial ROI area. When the optical flow distribution exceeds the preset threshold, it indicates that the target may have moved out of the original ROI. Combining the optical flow results and the ROI position of the previous frame, the center and size of the new ROI are estimated through the Kalman filter algorithm to achieve smooth tracking. When the displacement deviation between the Kalman prediction and the TWA calibration marker exceeds the preset deviation threshold ε, the short-term field of view calibration is automatically triggered to correct the drift of internal and external parameters, ensuring the continuous and accurate positioning of the ROI and providing stable input for subsequent processing.

[0090] In step 3, the TSR-WGAN model is used, and the generator network is an image sequence I after (2N+1) frames of fusion fusionAs input, multi-scale feature extraction is performed through multi-layer convolution, and the feature dimension is [d1, d2, d3], where d1, d2, and d3 are the increasing feature channel number parameters, which are used to capture multi-scale perturbation features; geometric distortion is corrected using cascaded multi-scale deformation convolution (the convolution kernel size is [k1, k2, k3]; k1, k2, and k3 are the increasing spatial receptive field parameters); with the help of 3D convolution (the convolution kernel size is [k w ,k h ,k t ];k t is the time dimension span parameter) and the temporal attention mechanism to distinguish real motion from turbulent disturbances; a codec structure containing n residual blocks is used to retain the global and local features of the image and output the corrected image; where n represents the network depth parameter;

[0091] The discriminator network adopts a multi-scale discriminant architecture. The global discriminator is based on the PatchGAN structure and takes an input of size [W, H] to judge the overall authenticity of the image. The local discriminator samples image patches of size [w, h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and the m frames before and after it, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter.

[0092] Generate perturbation samples of different intensities through physical simulation or algorithm, and construct a training set consisting of M pairs of perturbed real samples, a validation set of Q pairs of samples, and a test set of P pairs of samples; where M, Q, and P represent positive integers set according to the complexity of the task.

[0093] Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is performed; among them, L WG represents the WGAN-GP loss, which ensures training stability through the gradient penalty term; L Pe represents the perceptual loss, which extracts features based on the pre-trained visual network and measures the semantic difference between the generated image and the real image; L TV Represents the total variation loss, which suppresses the generation of image noise; λ1 and λ2 represent the weight coefficients of the corresponding loss terms.

[0094] The (2N+1) frame preprocessed image sequence is used as input, inference is performed under GPU acceleration, and the corrected central frame image is output.

[0095] In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements;

[0096] The overlap ratio of adjacent subsets is expressed as r%, which controls the density of displacement field calculation;

[0097] The iterative convergence threshold is γ to ensure the accuracy of displacement calculation;

[0098] For the image sequence corrected by the deep learning model, the DIC algorithm is used to analyze the pixel grayscale changes. Based on the conversion coefficient determined by the camera calibration and the actual length corresponding to 1 pixel, the displacement field and strain field data are generated.

[0099] In step 5, the monitoring indicators are calculated, including absolute displacement (accumulated displacement per day, reflecting the overall deformation of the structure), displacement change rate (displacement change rate per unit time, monitoring deformation speed), key area strain (assessment of material stress state) and other indicators to analyze the foundation pit status;

[0100] For each monitoring indicator, the generated displacement field and strain field data are used to maintain a historical normal sliding window data set of length W;

[0101] The historical normal sliding window data refers to the data collected when the foundation pit structure is in a stable state to ensure the reliability of the baseline; the median med(t) and median absolute deviation MAD are calculated within the window. The calculation formula is: MAD(t)=median(|x j -med(t)|);

[0102] Where med(t) represents the median of the monitoring indicator in the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median of the deviation between the data in the sliding window and the median med(t); x j Represents the data of the jth frame in the sliding window;

[0103] The median and median absolute deviation can robustly describe the central tendency and dispersion of data and avoid the influence of extreme values;

[0104] Using the disturbance information captured by the event camera, we obtain the event density E(t), which reflects the on-site vibration / disturbance intensity. We normalize the event density E(t) to g(t) = E(t) / max(E). We define a linear correction function: f(g(t)) = 1 + β(g(t) - 0.5).

[0105] Among them, E(t) represents the sum of the global event density after the accumulation of all event streams collected by the event camera in the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which can be set according to engineering experience; the function can automatically adjust the correction amplitude according to the intensity of the environmental disturbance. When the environmental disturbance is strong (g(t)>0.5), the function value is greater than 1 and the threshold will be automatically relaxed; when the environment is stable (g(t)<0.5), the function value is less than 1 and the threshold is tightened, thereby improving the adaptability of the monitoring system to different environmental conditions.

[0106] Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t)=med(t)+kMAD(t)×f(g(t)); where k represents the statistical confidence factor, which can be set uniformly or slightly differentiated for different indicators. This formula allows the threshold to automatically adjust as historical data changes and the intensity of environmental disturbances vary. When the environmental disturbance is strong, the threshold is relaxed to avoid false alarms caused by environmental interference; when the environment is stable, the threshold is tightened to increase sensitivity to abnormal changes in the foundation pit.

[0107] In order to further suppress occasional noise and ensure the accuracy of early warning, when Z1 consecutive frames or Z2 cumulative times exceed T(t), an alarm is triggered to remind relevant personnel to take measures to ensure the safety of foundation pit construction.

[0108] A deep learning image data intelligent supervision system, the system includes a data acquisition module, a fusion tracking module, an image correction module, a displacement analysis module and an early warning decision module;

[0109] The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images; the fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, use an algorithm to calculate optical flow and combine it with Kalman filtering to track ROI; the image correction module is used to build a TSR-WGAN model architecture, construct a sample set training model, input a preprocessing sequence for inference and output a corrected image; the displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data; the early warning decision module is used to calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

[0110] The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit and a parameter configuration unit;

[0111] The dual-mode acquisition unit is used to deploy high-temperature resistant cameras and event cameras to achieve multimodal image acquisition; the calibration and synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

[0112] The fusion tracking module includes a ROI extraction unit, a time domain fusion unit and an optical flow tracking unit;

[0113] The ROI extraction unit is used to intercept the monitoring point area from each frame image; the time domain fusion unit is used to perform Gaussian weighted averaging and event density map fusion; the optical flow tracking unit is used to dynamically track the ROI through the optical flow algorithm and Kalman filtering to trigger calibration correction.

[0114] The image correction module includes a feature extraction unit, a distortion correction unit and a temporal reasoning unit;

[0115] The feature extraction unit is used to extract disturbance features through multi-channel multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion by using cascaded deformation convolution; the temporal reasoning unit is used to distinguish motion or turbulence by using 3D convolution and attention mechanism, and output a corrected image.

[0116] The displacement analysis module includes a DIC configuration unit and a strain calculation unit;

[0117] The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes;

[0118] The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit and an alarm triggering unit;

[0119] The indicator calculation unit is used to calculate absolute displacement, rate of change and strain; the baseline maintenance unit is used to maintain the historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold; the alarm triggering unit is used to trigger an abnormal alarm when the trigger condition exceeds the threshold.

[0120] In this example, at a subway foundation pit monitoring site, a high-temperature resistant conventional camera (operating temperature -20°C-120°C, resolution 1920×1080, frame rate 30fps) and an event camera (pixel resolution 346×260) were mounted on a seismic-resistant synchronous bracket, 15 meters from the edge of the foundation pit and at a 30° elevation angle. The conventional camera was equipped with an autofocus lens (focal length 16mm, aperture F1.8), while the event camera used a wide-angle lens (field of view 120°). A 9×12 checkerboard calibration plate was used, with eight calibration points evenly distributed across the foundation pit site, to capture 50 image pairs. The Zhang Zhengyou calibration method was used to calculate the intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (rotation matrix, translation vector) of the conventional camera. For the event camera, an event stream-based spatiotemporal calibration algorithm was used to map the event coordinates to the conventional camera coordinate system, achieving sub-pixel alignment accuracy (±0.3 pixels).

[0121] Step 1: The traditional camera captures time-series images at 30 fps, capturing 31 frames each time (N=15, with a time window of approximately 1 second).

[0122] The event camera asynchronously records all pixel brightness change events with an event resolution of 1μs and generates a four-tuple data stream (x, y, t, p);

[0123] Step 2: Time domain weighted fusion and ROI dynamic tracking;

[0124] Extract all events within ±167ms (corresponding to 1 / 30 second) of the traditional camera's center frame timestamp. For each pixel (x, y), count the number of events C(x, y) and normalize them. Smooth E(x, y) with a 3×3 Gaussian kernel to suppress salt and pepper noise.

[0125] Temporal weighted average (TWA): For the traditional image sequence Ii(x,y) (i∈[-15,15]), calculate:

[0126] ,

[0127] w i =e^-i 2 / 2σ 2 ;

[0128] Space-Time Fusion: I fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E);

[0129] ROI tracking: Initialize the ROI to a 200×200 pixel area representing the key points of the foundation pit support structure; calculate the optical flow field within the ROI. When the average optical flow amplitude is greater than 5 pixels, trigger the Kalman filter to predict the new ROI position; if the deviation between the predicted position and the TWA-corrected position is greater than 3 pixels, initiate short-term field of view calibration (taking less than 0.5 seconds);

[0130] Step 3: TSR-WGAN model construction and image correction;

[0131] Generator network: Input layer: 31 frames × 200 × 200 × 1 fused image sequence; Feature extraction: 3 layers of 3D convolution (convolution kernel 3 × 3 × 3, stride 1, feature dimension [64, 128, 256]); Deformation convolution: 2-level deformable convolution module to correct geometric distortion caused by high temperature; Temporal attention: Self-attention mechanism based on 3D convolution, weight dimension [31, 200, 200]; Encoder-decoder structure: 8 residual blocks (each containing 2 3 × 3 convolutions + ReLU), outputting the corrected center frame.

[0132] Discriminator network: Global discriminator: PatchGAN structure, input 256×256 image patches, output 32×32 authenticity scores; Local discriminator: samples 64×64 image patches and uses dilated convolution (dilation=2) to enhance detail perception; Temporal consistency: input the current frame and the three frames before and after, and judge the temporal coherence through 3D convolution (kernel size 3×3×3);

[0133] Training process:

[0134] Dataset: Physical simulation: COMSOL thermal flow simulation was used to generate 1000 pairs of disturbance-real images under different temperature fields; Field collection: 1000 pairs of real data were collected at different stages of foundation pit construction;

[0135] Loss function: L=L WG +λ1L Pe +λ2L TV ;

[0136] Step 4: DIC algorithm parameter configuration and displacement field analysis;

[0137] Parameter settings: subset size: 21×21 pixels (corresponding to the actual size of 10.5mm×10.5mm); subset overlap rate: 60% (step size 8 pixels); iterative convergence threshold: γ=1e-4; displacement calculation method: least squares method based on Newton-Raphson iteration.

[0138] Calculation process: For the corrected image sequence, the displacement field of frame t+Δt is calculated with frame t as the reference. Sub-pixel interpolation (bicubic interpolation) is used to improve the displacement accuracy to 0.01 pixel. Based on the camera calibration parameters (1 pixel = 0.5 mm), the pixel displacement is converted into actual physical displacement. The strain field is calculated from the displacement field gradient.

[0139] Step 5: Adaptive threshold warning mechanism;

[0140] Historical baseline maintenance: Window size W = 2000 frames (approximately 70 minutes of continuous monitoring data); only manually confirmed foundation pit stability data are included; when new data are added, the earliest data is discarded using a sliding window strategy.

[0141] Statistical feature calculation: For each monitoring indicator (absolute displacement D, displacement change rate v, principal strain εmax), calculate: med(t); calculate the global event density, perform normalization and correction; and construct an adaptive threshold;

[0142] An alarm is triggered when Z1=3 consecutive frames exceed the threshold, or Z2=10 cumulative times exceed the threshold.

[0143] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A deep learning-based intelligent supervision method for image data, characterized by: The method comprises the following steps: Step 1: Combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images; Step 2: Perform time-domain weighted averaging and asynchronous event fusion on the collected data, calculate the optical flow using an algorithm, and track the ROI in combination with the Kalman filter; Step 3: Build the TSR-WGAN model architecture, construct a sample set training model, input the preprocessed sequence for inference, and output the corrected image; Step 4: Configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data; Step 5: Calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

2. The deep learning image data intelligent supervision method according to claim 1, characterized in that: In step 1, a combination of a high-temperature resistant traditional camera and an event camera is used to perform multimodal acquisition of foundation pit image data; The operating temperature range of high-temperature resistant traditional cameras covers the monitoring environment temperature, and is equipped with imaging elements with a resolution not less than a specific value and a frame rate of f frames per second; event cameras are used to capture disturbance information; The high-temperature resistant traditional camera and the event camera are mounted on the same synchronous bracket, and the distance from the foundation pit is kept within the set range; Use the calibration plate to calibrate the high-temperature resistant camera and event camera respectively, obtain internal and external parameters, and achieve sub-pixel spatial coordinate conversion accuracy; The high-temperature resistant traditional camera performs time series acquisition at a rate of f frames per second, and each acquisition is no less than (2N+1) frames; where N represents the half-length parameter of the time series processing window; Generate grayscale images based on ambient lighting through automatic exposure adjustment; The event camera asynchronously records all event streams within a time window and time-aligns them with the traditional frames captured by the high-temperature resistant traditional camera.

3. The deep learning image data intelligent supervision method according to claim 2, characterized in that: In step 2, the region of interest (ROI) containing the monitoring point is extracted from each frame of the image captured by the high-temperature resistant conventional camera with sub-pixel accuracy; With each ROI as the center, a time window of length (2N+1) frames is constructed; Perform time domain weighted averaging on the images in the time window, with weight w i Follows Gaussian distribution: w i =e^-i 2 / 2σ 2 ; Where i represents the index of the image frame in the time window, i∈[-N,N]; w i represents the weight coefficient of the i-th frame image in the corresponding time window; σ represents the weight decay coefficient, which controls the weight decay speed of adjacent frames; Through the weighted fusion formula: ， Enhance center frame stability and suppress random disturbances; Where (x, y) represents the coordinates of the pixel point on the image plane; I i (x, y) represents the pixel value of the i-th frame image at the coordinate (x, y) in the time window; I TWA (x,y) represents the pixel value at coordinate (x,y) after time-domain weighted averaging; The events of the event camera in the time window are accumulated into the event density map E(x, y), and the image processed by TWA is subjected to spatiotemporal weighted fusion; the fusion formula is I fusion (x,y)=α·I TWA (x,y)+(1-α)·E(x,y) / max(E); Among them, I fusion (x, y) represents the pixel value at the coordinate (x, y) after multimodal data fusion; α represents the fusion coefficient, which is adaptively adjusted according to environmental factors such as the on-site vibration intensity; max(E) represents the maximum value of all pixel values ​​in the event density map E(x, y), which is used to normalize the event density map; The optical flow algorithm is used to calculate the dense optical flow of the initial ROI area. When the optical flow distribution exceeds the preset threshold, it indicates that the target has moved out of the original ROI. The center and size of the new ROI are estimated by combining the optical flow results and the ROI position of the previous frame through the Kalman filter algorithm. When the displacement deviation between the Kalman prediction and the TWA calibration marker exceeds the preset deviation threshold ε, the short-term field of view calibration is automatically triggered to correct the drift of the internal and external parameters.

4. The deep learning image data intelligent supervision method according to claim 3, characterized in that: In step 3, the TSR-WGAN model is used, and the generator network is an image sequence I after (2N+1) frames of fusion fusion The model takes d1 as input and extracts multi-scale features through multi-layer convolution. The feature dimension is [d1, d2, d3], where d1, d2, and d3 are parameters with increasing number of feature channels, which are used to capture multi-scale perturbation features. Cascaded multi-scale deformable convolution is used to correct geometric distortion. 3D convolution and temporal attention mechanism are used to distinguish between real motion and turbulent perturbations. A codec structure containing n residual blocks is used to retain global and local features of the image and output the corrected image. Where n represents the network depth parameter. The discriminator network adopts a multi-scale discriminant architecture. The global discriminator is based on the PatchGAN structure and takes an input of size [W, H] to judge the overall authenticity of the image. The local discriminator samples image patches of size [w, h] and enhances detail perception through dilated convolution. The input includes the temporal context of the current frame and the m frames before and after it, and achieves temporal consistency discrimination through 3D convolution. Here, m represents the temporal correlation parameter. Generate perturbation samples of varying intensities through physical simulation or algorithms, and construct a training set consisting of M pairs of perturbed real samples, a validation set of Q pairs of samples, and a test set of P pairs of samples; where M, Q, and P represent positive integers set according to the complexity of the task. Using the composite loss function L=L WG +λ1L Pe +λ2L TV Training is performed; among them, L WG represents the WGAN-GP loss, which ensures training stability through the gradient penalty term; L Pe represents the perceptual loss, which extracts features based on the pre-trained visual network and measures the semantic difference between the generated image and the real image; L TV Represents the total variation loss, which suppresses the generation of image noise; λ1 and λ2 represent the weight coefficients of the corresponding loss terms; The (2N+1) frame preprocessed image sequence is used as input and the corrected central frame image is output.

5. The deep learning image data intelligent supervision method according to claim 4, characterized in that: In step 4, the subset size of the DIC algorithm is set to (s×s) pixels; where s represents the spatial resolution parameter, which is set according to the measurement accuracy requirements; The overlap ratio of adjacent subsets is expressed as r%, which controls the density of displacement field calculation; The iterative convergence threshold is γ to ensure the accuracy of displacement calculation; For the image sequence corrected by the deep learning model, the DIC algorithm is used to analyze the pixel grayscale changes. Based on the conversion coefficient determined by the camera calibration and the actual length corresponding to 1 pixel, the displacement field and strain field data are generated. In step 5, the monitoring indicators are calculated. For each monitoring indicator, the generated displacement field and strain field data are used to maintain a historical normal sliding window data set of length W; the historical normal sliding window data refers to the data collected when the foundation pit structure is in a stable state; Calculate the median med(t) and median absolute deviation MAD in the window W. The calculation formula is: MAD(t)=median(|x j -med(t)|); Where med(t) represents the median of the monitoring indicator in the sliding window, t represents the timestamp when the current threshold is calculated; MAD(t) represents the median of the deviation between the data in the sliding window and the median med(t); x j Represents the data of the jth frame in the sliding window W; Use the event camera to obtain the event density E(t); normalize the event density E(t) to g(t) = E(t) / max(E); define the linear correction function: f(g(t)) = 1 + β(g(t) - 0.5); Where E(t) represents the total global event density of all event streams collected by the event camera in the current time window; β∈[0,1] represents the environmental sensitivity coefficient, which is set according to engineering experience; Combining historical baselines and environmental factors, an adaptive threshold formula is constructed: T(t)=med(t)+kMAD(t)×f(g(t)); where k represents the statistical confidence factor; When Z1 consecutive frames or Z2 cumulative times exceed T(t), an alarm is triggered.

6. A deep learning image data intelligent supervision system, applied to the deep learning image data intelligent supervision method according to any one of claims 1 to 5, characterized in that: The system includes data acquisition module, fusion tracking module, image correction module, displacement analysis module and early warning decision module; The data acquisition module is used to combine and deploy high-temperature resistant traditional cameras and event cameras, calibrate acquisition parameters, and set acquisition parameters for multimodal acquisition of foundation pit images; the fusion tracking module is used to perform time-domain weighted averaging and asynchronous event fusion on the acquired data, use an algorithm to calculate optical flow and combine it with Kalman filtering to track ROI; the image correction module is used to build a TSR-WGAN model architecture, construct a sample set training model, input a preprocessing sequence for inference and output a corrected image; the displacement analysis module is used to configure the subset size and overlap rate parameters of the DIC algorithm, analyze the corrected image sequence to generate displacement field and strain field data; the early warning decision module is used to calculate the absolute displacement and displacement change rate indicators, set the thresholds for each indicator, and trigger a foundation pit abnormality warning when the data exceeds the threshold.

7. The deep learning image data intelligent monitoring system according to claim 6, characterized in that: The data acquisition module includes a dual-mode acquisition unit, a calibration synchronization unit and a parameter configuration unit; The dual-mode acquisition unit is used to deploy high-temperature resistant cameras and event cameras to achieve multimodal image acquisition; the calibration and synchronization unit is used to perform camera calibration to obtain sub-pixel level parameters to ensure spatiotemporal synchronization; the parameter configuration unit is used to set the acquisition window and automatic exposure parameters.

8. The deep learning image data intelligent monitoring system according to claim 7, characterized in that: The fusion tracking module includes a ROI extraction unit, a time domain fusion unit and an optical flow tracking unit; The ROI extraction unit is used to intercept the monitoring point area from each frame of image; The time domain fusion unit is used to perform Gaussian weighted averaging and event density map fusion; the optical flow tracking unit is used to dynamically track ROI through optical flow algorithm and Kalman filtering to trigger calibration correction.

9. The deep learning image data intelligent monitoring system according to claim 8, characterized in that: The image correction module includes a feature extraction unit, a distortion correction unit and a temporal reasoning unit; The feature extraction unit is used to extract disturbance features through multi-channel multi-scale convolution; the distortion correction unit is used to eliminate geometric distortion by using cascaded deformation convolution; the temporal reasoning unit is used to distinguish motion or turbulence by using 3D convolution and attention mechanism, and output a corrected image.

10. The deep learning image data intelligent monitoring system according to claim 9, characterized in that: The displacement analysis module includes a DIC configuration unit and a strain calculation unit; The DIC configuration unit is used to set the subset size, overlap rate, and convergence threshold; the strain calculation unit is used to generate displacement field and strain field data based on pixel grayscale changes; The early warning decision module includes an indicator calculation unit, a baseline maintenance unit, a threshold calculation unit and an alarm triggering unit; The indicator calculation unit is used to calculate absolute displacement, rate of change and strain; the baseline maintenance unit is used to maintain a historical normal sliding window and calculate the median; the threshold calculation unit is used to receive event density and perform normalization and linear correction to generate an adaptive threshold; The alarm triggering unit is used to trigger an abnormality alarm when the triggering condition exceeds a threshold.

Images