A structural intelligent monitoring device based on deep learning

Abstract

The present invention discloses a structure intelligent monitoring device based on deep learning, which specifically relates to the field of structure measurement and survey, including a camera image acquisition module, an external environment information acquisition module, an external environment data comparison module, an internal environment information acquisition module, an internal environment data comparison module, a camera and algorithm synchronization module, and an abnormal state processing module. The device system generates an environmental assessment index through the external environment information acquisition module and the internal environment information acquisition module, and compares it with a preset safety threshold; when the evaluation index of the external environment or internal environment monitored or surveyed exceeds the safety range, the system will automatically suspend data processing or adjust the acquisition parameters to avoid monitoring and analysis under unstable or abnormal conditions; this mechanism effectively improves the adaptability and safety of the system in complex environments, ensuring that it can still maintain accurate monitoring and survey capabilities under harsh conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of structural measurement and surveying, and more specifically, to a structural intelligent monitoring device based on deep learning. Background Art

[0002] The intelligent structural monitoring device uses a camera as the main data acquisition device to monitor the state changes of structures (such as buildings or bridges) in real time through visual methods. The camera continuously collects image data of the structure and passes it to the deep learning algorithm to analyze the displacement, deformation or cracks of the structure in the image.

[0003] In the intelligent structural monitoring device based on deep learning, the synchronization problem between the camera and the algorithm is a crucial technical link. The existing technology relies on the image data collected by the camera, combined with the deep learning algorithm, to conduct real-time monitoring of structures such as buildings or bridges, and detect subtle displacements, deformations or cracks. Usually, the image data is continuously collected by the camera and transmitted to the processing unit. The deep learning model identifies the changes in the structural state through these continuous image sequences. However, the emergence of synchronization problems will cause confusion in data processing, leading to incorrect judgment of the structural state, and thus creating potential safety hazards. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a structural intelligent monitoring device based on deep learning, which generates an environmental assessment index through an external environment information collection module and an internal environment information collection module, and compares it with a preset safety threshold to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solutions: a structure intelligent monitoring device based on deep learning, comprising a camera image acquisition module, an external environment information acquisition module, an external environment data comparison module, an internal environment information acquisition module, an internal environment data comparison module, a camera and algorithm synchronization module and an abnormal state processing module;

[0006] The camera image acquisition module is used to acquire real-time images of the monitored object. Real-time image acquisition is used to capture the displacement, cracks or deformation of the monitored object, obtain image data through the camera device, and adjust the acquisition parameters in real time in combination with external environment information;

[0007] The external environment information acquisition module is used to collect external environment information that affects image quality and system synchronization. The external environment information includes system clock deviation rate information and ambient light intensity fluctuation rate information;

[0008] The external environment data comparison module is used to compare the collected external environment information with a preset threshold value 1; the external environment data comparison module generates an external environment evaluation index by using the collected external environment information; if the external environment evaluation index exceeds threshold value 1, the system will judge the external environment as high risk, trigger an alarm, and suspend subsequent processing; if the external environment evaluation index is lower than threshold value 1 or is within the range of threshold value 1, the external environment is judged as low risk, and the system is allowed to continue to collect and process internal environment information;

[0009] The internal environment information acquisition module is used to monitor the operating status of the device system; the internal environment information includes processor load rate information, data transmission delay rate information and signal transmission integrity rate information;

[0010] The internal environment data comparison module is used to analyze the collected processor load rate information, data transmission delay rate information and signal transmission integrity rate information, and generate an internal environment data evaluation index; compare the internal environment data evaluation index with the preset threshold value 2 to determine whether the device system is suitable for synchronization processing of the camera and the algorithm; if the internal environment data evaluation index exceeds the threshold value 2, the current environment is determined to be in an abnormal state, and the device system suspends the synchronization process and adjusts the parameters; if the internal environment data evaluation index is lower than the threshold value 2 or is within the range of the threshold value 2, the current environment is determined to be in a normal state, and the device system continues data processing;

[0011] The camera and algorithm synchronization module is used to synchronize the image data collected by the camera with the deep learning algorithm;

[0012] The abnormal state processing module is used to handle abnormal situations during the synchronization process between the camera and the algorithm; if the device system detects abnormal internal environmental data during the synchronization process, the abnormal state processing module will suspend the current processing and re-evaluate the external and internal environmental information.

[0013] In a preferred embodiment, the system clock deviation rate information includes a timestamp accuracy index; the ambient light intensity fluctuation rate information includes a light intensity variation index;

[0014] The proposed timestamp precision index is The timestamp accuracy index is used to evaluate the timestamp collected by the camera. and the time the processing unit receives The synchronization accuracy between the two frames is calculated by And introduce the cumulative time deviation square ratio ;

[0015] ;

[0016] in Indicates the timestamp precision index, which is used to reflect the accuracy of time synchronization in the system; is the adjustment coefficient, which is used to control the impact of time difference on the index; Indicates The time difference of the frames, equal ; Indicates the camera acquisition timestamp; Indicates the timestamp received by the processing unit; Indicates the square ratio of time deviation, which is used to amplify the impact of time difference on synchronization accuracy;

[0017] The proposed light intensity variation index is The light intensity variation index is used to evaluate the impact of light intensity fluctuations on image acquisition quality and calculate the light intensity variation rate of each frame. And the maximum and minimum ratio of light intensity ;

[0018] ;

[0019] in Indicates the light intensity variation index, reflecting the impact of light fluctuations on image quality; represents the adjustment coefficient, which controls the sensitivity of light fluctuations to the index; Indicates The frame illumination change rate is calculated as:

[0020] ;

[0021] in is the maximum value of light intensity, indicating the peak value of light fluctuation; is the minimum value of light intensity, indicating the lowest point of light fluctuation; It represents the ratio of the maximum to minimum light intensity, reflecting the amplitude of light fluctuation.

[0022] In a preferred embodiment, by combining the timestamp precision index and light intensity variation index To evaluate the comprehensive impact of the external environment on the system, and obtain the external environment evaluation index ;

[0023] ;

[0024] in It is the external environment evaluation index, reflecting the comprehensive impact of the external environment on the system; is the adjustment coefficient, which controls the weight of the impact of time synchronization and illumination changes on the external environment; It is the timestamp precision index, which reflects the accuracy of time synchronization; It is the light intensity variation index, reflecting the impact of light fluctuation on image acquisition; is the product of time synchronization and illumination change, indicating the joint influence of the two; is the Euclidean distance between time synchronization and illumination fluctuation, representing the joint deviation of the two.

[0025] In a preferred embodiment, the processor load rate information includes an algorithm processing delay index; the data transmission delay rate information includes a network jitter index; the signal transmission integrity rate information includes a data packet loss index;

[0026] The proposed algorithm processing delay index is The algorithm processing delay index is used to measure the load rate and processing delay of the system processing unit and calculate the processing delay of each frame. and processor load fluctuations ;

[0027] ;

[0028] in Indicates the algorithm processing delay index, which is used to reflect the severity of the processing delay; It represents the adjustment coefficient, which controls the impact of delay and load fluctuation on the index; Indicates the processing delay of each frame, indicating the length of processing time; Indicates the maximum value of the processor load, indicating the peak value in the load fluctuation; Indicates the minimum value of the processor load, indicating the lowest point in the load fluctuation; Indicates the load fluctuation ratio, which is used to evaluate the load fluctuation;

[0029] The proposed network jitter index is ; The network jitter index is used to evaluate the fluctuation of data transmission delay and calculate the transmission time change of each frame The maximum and minimum time ratio of network transmission ;

[0030] ;

[0031] in Indicates the network jitter index, which is used to reflect the volatility of network transmission delay; Indicates the adjustment coefficient, which controls the impact of jitter fluctuations on the index; Indicates The transmission delay of frames reflects the fluctuation of network transmission time; Indicates the maximum value of network transmission delay, reflecting the highest delay of network transmission; Indicates the minimum value of network transmission delay, reflecting the lowest delay of network transmission; It indicates the ratio of the maximum value to the minimum value of the transmission delay, reflecting the amplitude of the transmission jitter;

[0032] The proposed packet loss index is ,Packet loss index is used to evaluate the packet loss rate in network transmission;

[0033] ;

[0034] in is the packet loss index, which reflects the impact of the packet loss rate on system transmission; To adjust the coefficient, we control the impact of packet loss and fluctuation on the index; For the The packet loss rate of the frame indicates the proportion of lost data packets; is the maximum value of the expected data packet; is the minimum value of the received data packets; It is the packet loss fluctuation ratio, which reflects the fluctuation amplitude of the packet loss rate.

[0035] In a preferred embodiment, the delay index is processed by a nonlinear combination algorithm. , Network Jitter Index and Packet Loss Index To evaluate the health status of the system and obtain the internal environment evaluation index ; By combining the Euclidean distance and product of the three indexes, the comprehensive impact of various factors within the system is reflected;

[0036] ;

[0037] in Represents the internal environment evaluation index, which is used to reflect the health status of the internal operation of the system; It represents the adjustment coefficient, which controls the comprehensive influence weight of each internal factor; Indicates the algorithm processing delay index, reflecting the impact of processing delay; Indicates the network jitter index, reflecting the impact of data transmission delay fluctuations; Indicates the packet loss index, which reflects the packet loss rate in network transmission; It represents the Euclidean distance of the three, indicating the joint deviation of the various factors operating within the system; It represents the product of the three, amplifying the impact when multiple factors go wrong at the same time.

[0038] In a preferred embodiment, it also includes a survey system; the survey system includes a data acquisition and preprocessing module, a deep learning model selection and design module, a model training and optimization module, a model verification and testing module, and a model deployment and real-time monitoring module;

[0039] The data acquisition and preprocessing module collects multi-dimensional data through lidar, camera, inertial measurement unit, and accelerometer; the collected data is filtered, denoised, and standardized before being input into the deep learning model for processing;

[0040] The deep learning model selection and design module uses convolutional neural networks to process image data and extract structural surface features according to the structural monitoring task; for time series data, long short-term memory networks are used to analyze time series features;

[0041] The model training and optimization module trains the model using labeled structural health history data, optimizes the model weights using the back-propagation algorithm, uses the cross-entropy loss function to predict errors, and ensures model convergence during training through hyperparameter tuning;

[0042] Model validation and testing module: the performance of the trained model is evaluated on the validation set and the test set. The evaluation indicators include accuracy, mean square error, precision and recall.

[0043] Model deployment and real-time monitoring module: The trained deep learning model is deployed on edge devices or the cloud to process the monitoring data from sensors in real time, analyze and judge the health status of the structure; it responds quickly through edge computing and triggers an early warning immediately when a structural abnormality is detected.

[0044] Technical effects and advantages of the present invention:

[0045] The device system generates an environmental assessment index through the external environment information collection module and the internal environment information collection module, and compares it with the preset safety threshold; when the external environment or internal environment assessment index is monitored or surveyed to exceed the safety range, the system will automatically suspend data processing or adjust the collection parameters to avoid monitoring and analysis under unstable or abnormal conditions; this mechanism effectively improves the adaptability and safety of the system in complex environments, ensures that it can still maintain accurate monitoring and surveying capabilities under harsh conditions, and improves the synchronization problem between the camera and the algorithm;

[0046] The system uses five key indexes to comprehensively monitor the operating status of the equipment and changes in the external environment. These five indexes include timestamp accuracy index, light intensity change index, processor load rate index, data transmission delay rate index and signal transmission integrity index. Through these indexes, the system can dynamically adjust its working parameters to ensure the stable operation of the equipment in different environments, and timely discover and deal with potential equipment operation problems through adaptive mechanisms. This multi-dimensional monitoring and adjustment mechanism enables the system to have higher reliability and accuracy in long-term use. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a module flow chart of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Refer to the instruction manual Figure 1 , a structure intelligent monitoring device based on deep learning according to an embodiment of the present invention comprises a camera image acquisition module, an external environment information acquisition module, an external environment data comparison module, an internal environment information acquisition module, an internal environment data comparison module, a camera and algorithm synchronization module, and an abnormal state processing module;

[0050] Camera image acquisition module (for linkage with external environment assessment):

[0051] The camera image acquisition module is used to acquire real-time images of the monitored object. Real-time image acquisition is used to capture structural anomalies such as displacement, cracks or deformation of the monitored object. Image data is obtained through camera equipment, and acquisition parameters such as frame rate and exposure are adjusted in real time in combination with external environmental information. Camera image acquisition is the starting point of the entire monitoring process, ensuring that the captured data has clarity and time consistency, providing an accurate and synchronized data source for subsequent processing.

[0052] External environment information collection module (used for dependencies with internal processing triggers):

[0053] The external environment information acquisition module is used to collect external environment information that affects image quality and system synchronization. The external environment information includes system clock deviation rate information and ambient light intensity fluctuation rate information. The external environment information is used to ensure that the image data captured by the camera is not interfered by external factors such as illumination changes and clock errors. The goal of this module is to ensure that the system performs further data acquisition and processing under appropriate external conditions.

[0054] External environment data comparison module (used for linkage with high-risk early warning):

[0055] The external environment data comparison module is used to compare the collected external environment information with a preset threshold value one; the external environment data comparison module generates an external environment evaluation index by using the collected external environment information; if the external environment evaluation index exceeds threshold value one, the system will judge the external environment as high risk, trigger an alarm, and suspend subsequent processing; if the external environment evaluation index is lower than threshold value one or within the range of threshold value one, the external environment is judged as low risk, allowing the system to continue to collect and process internal environment information; this module is used to filter external conditions that are not suitable for monitoring to ensure the reliability of image data acquisition;

[0056] Internal environment information collection module (external environment assessment is the premise of internal environment assessment):

[0057] The internal environment information acquisition module is used to monitor the operating status of the device system; the internal environment information includes processor load rate information, data transmission delay rate information and signal transmission integrity rate information. The internal environment information reflects the health of the system during image data transmission and processing, ensuring the stable operation of the data transmission chain between the camera and the algorithm. This module is the prerequisite for ensuring that the subsequent algorithm can smoothly receive and process image data, avoiding excessive internal system load or data delay affecting the overall performance;

[0058] Internal environment data comparison module (for deterministic association with synchronization status):

[0059] The internal environment data comparison module is used to analyze the collected processor load rate information, data transmission delay rate information and signal transmission integrity rate information, and generate an internal environment data evaluation index. By analyzing these parameters, the system can determine whether the internal environment can support normal data transmission and processing; this module ensures that the various operating indicators of the internal system are within a reasonable range to avoid internal bottlenecks affecting data synchronization and the accuracy of deep learning algorithms; the internal environment data evaluation index is compared with the preset threshold value 2 to determine whether the device system is suitable for synchronization processing of the camera and the algorithm; if the internal environment data evaluation index exceeds the threshold value 2, the current environment is judged to be in an abnormal state, and the device system suspends the synchronization process and adjusts the parameters; if the internal environment data evaluation index is lower than the threshold value 2 or is within the range of the threshold value 2, the current environment is judged to be in a normal state, and the device system continues data processing; this is the core module to ensure the health of the internal state of the device system;

[0060] Camera and algorithm synchronization module:

[0061] The camera and algorithm synchronization module is used to synchronize the image data collected by the camera with the deep learning algorithm. Through the camera and algorithm synchronization module, the device system ensures that the data transmission and algorithm processing maintain a consistent time sequence to avoid data analysis errors caused by timing disorder. The module can dynamically adjust the camera frame rate and processor load to ensure the real-time and accuracy of data synchronization. The camera and algorithm synchronization module is the core of the monitoring status of the entire device system, ensuring that the image data can be transmitted to the algorithm processing unit in a timely and effective manner.

[0062] Abnormal state processing module (used for closed-loop linkage with all front-end modules):

[0063] The abnormal state processing module is used to handle abnormal situations during the synchronization process of the camera and the algorithm; if the device system detects abnormal internal environmental data during the synchronization process, the abnormal state processing module will suspend the current processing and re-evaluate the external and internal environmental information; the system will adjust the corresponding parameters according to the evaluation results until the synchronization returns to normal; this module is an important guarantee to ensure that the system can self-repair and resume normal operation under abnormal circumstances.

[0064] The system clock deviation rate information includes a timestamp accuracy index; the ambient light intensity fluctuation rate information includes a light intensity variation index;

[0065] The proposed timestamp precision index is The timestamp accuracy index is used to evaluate the timestamp collected by the camera. and the time the processing unit receives The synchronization accuracy between the two frames is calculated by calculating the time difference between each frame. And introduce the cumulative time deviation square ratio ,The timestamp accuracy index amplifies the time difference and is used to reflect the ,accuracy of the time synchronization of the device system. The larger the time difference, the faster the index decreases;

[0066] ;

[0067] in Indicates the timestamp precision index, which is used to reflect the accuracy of time synchronization in the system; is the adjustment coefficient, which is used to control the impact of time difference on the index; Indicates The time difference of the frames, equal ; Indicates the camera acquisition timestamp; Indicates the timestamp received by the processing unit; Indicates the square ratio of time deviation, which is used to amplify the impact of time difference on synchronization accuracy;

[0068] The proposed light intensity variation index is The illumination intensity variation index is used to evaluate the impact of illumination intensity fluctuations on image acquisition quality by calculating the illumination intensity variation rate of each frame. And the maximum and minimum ratio of light intensity , which amplifies the effect of light fluctuations, and the index drops rapidly when the light changes dramatically;

[0069] ;

[0070] in Indicates the light intensity variation index, reflecting the impact of light fluctuations on image quality; represents the adjustment coefficient, which controls the sensitivity of light fluctuations to the index; Indicates The frame illumination change rate is calculated as:

[0071] ;

[0072] in is the maximum value of light intensity, indicating the peak value of light fluctuation; is the minimum value of light intensity, indicating the lowest point of light fluctuation; It represents the ratio of the maximum to minimum light intensity, reflecting the amplitude of light fluctuation.

[0073] By combining the timestamp precision index and light intensity variation index To evaluate the comprehensive impact of the external environment on the system, and obtain the external environment evaluation index ; Through nonlinear combination, the joint effect of time synchronization and illumination changes is taken into account. If there are problems with both time synchronization and illumination fluctuations, the index will drop rapidly. In addition, when time synchronization and illumination fluctuations do not have problems at the same time, the solution is to monitor and process these two factors separately. When only time synchronization problems occur, the system should immediately trigger the time synchronization correction mechanism to adjust the time deviation between the camera and the processing unit to ensure timestamp synchronization without making any adjustments to the illumination. Similarly, when the illumination fluctuations are large but the time synchronization is normal, the system should give priority to adjusting the camera's exposure settings or illumination compensation to ensure image quality without processing time synchronization.

[0074] In the formula, the Euclidean distance part Allows the system to evaluate the impact of time synchronization and light fluctuations separately. If one of the indices drops, the system will focus on dealing with the problem without amplifying the impact. This ensures that the system can adjust to changes in a single factor, rather than waiting for problems with both at the same time before taking action.

[0075] ;

[0076] in It is the external environment evaluation index, reflecting the comprehensive impact of the external environment on the system; is the adjustment coefficient, which controls the weight of the impact of time synchronization and illumination changes on the external environment; It is the timestamp precision index, which reflects the accuracy of time synchronization; is the light intensity variation index, reflecting the impact of light fluctuations on image acquisition; is the product of time synchronization and illumination change, indicating the joint influence of the two; is the Euclidean distance between time synchronization and illumination fluctuation, representing the joint deviation of the two.

[0077] The processor load rate information includes an algorithm processing delay index; the data transmission delay rate information includes a network jitter index; the signal transmission integrity rate information includes a data packet loss index;

[0078] The proposed algorithm processing delay index is The algorithm processing delay index is used to measure the load rate and processing delay of the system processing unit. and processor load fluctuations ,The algorithm processing delay index amplifies the delay impact under high load, and the larger the processing delay and load fluctuations, the faster the index decreases;

[0079] ;

[0080] in Indicates the algorithm processing delay index, which is used to reflect the severity of the processing delay; It represents the adjustment coefficient, which controls the impact of delay and load fluctuation on the index; Indicates the processing delay of each frame, indicating the length of processing time; Indicates the maximum value of the processor load, indicating the peak value in the load fluctuation; Indicates the minimum value of the processor load, indicating the lowest point in the load fluctuation; Indicates the load fluctuation ratio, which is used to evaluate the severe fluctuation of load;

[0081] The proposed network jitter index is ; The network jitter index is used to evaluate the fluctuation of data transmission delay by calculating the transmission time change of each frame The maximum and minimum time ratio of network transmission ,This index reflects the volatility of network delay. The greater the jitter, the faster the index decreases;

[0082] ;

[0083] in Indicates the network jitter index, which is used to reflect the volatility of network transmission delay; Indicates the adjustment coefficient, which controls the impact of jitter fluctuations on the index; Indicates The transmission delay of frames reflects the fluctuation of network transmission time; Indicates the maximum value of network transmission delay, reflecting the highest delay of network transmission; Indicates the minimum value of network transmission delay, reflecting the lowest delay of network transmission; It indicates the ratio of the maximum value to the minimum value of the transmission delay, reflecting the amplitude of the transmission jitter;

[0084] The proposed packet loss index is The packet loss index is used to evaluate the packet loss rate in network transmission; it calculates the ratio of the number of received and expected packets. and its volatility. The index amplifies the impact of packet loss and volatility. The larger the packet loss rate, the faster the index decreases;

[0085] ;

[0086] in is the packet loss index, which reflects the impact of the packet loss rate on system transmission; To adjust the coefficient, we control the impact of packet loss and fluctuation on the index; For the The packet loss rate of the frame indicates the proportion of lost data packets; is the maximum value of the expected data packet; is the minimum value of the received data packets; It is the packet loss fluctuation ratio, which reflects the fluctuation amplitude of the packet loss rate.

[0087] Processing delay index through nonlinear combination algorithm , Network Jitter Index and Packet Loss Index To evaluate the health status of the system and obtain the internal environment evaluation index ; By combining the Euclidean distance and product of the three indexes, the comprehensive impact of various factors within the system is reflected;

[0088] ;

[0089] in Represents the internal environment evaluation index, which is used to reflect the health status of the internal operation of the system; It represents the adjustment coefficient, which controls the comprehensive influence weight of each internal factor; Indicates the algorithm processing delay index, reflecting the impact of processing delay; Indicates the network jitter index, reflecting the impact of data transmission delay fluctuations; Indicates the packet loss index, which reflects the packet loss rate in network transmission; It represents the Euclidean distance of the three, indicating the joint deviation of the various factors operating within the system; It represents the product of the three, amplifying the impact when multiple factors have problems at the same time; the index evaluates the internal operating status of the system by combining processing delay, network jitter and packet loss. The Euclidean distance part is used to measure their independent impact, and the product part captures the interaction between them. If multiple factors in the system have problems at the same time, the index will drop rapidly, indicating that the internal environment has deteriorated;

[0090] When the external environment evaluation index is obtained, the system will rate it according to the size of the index. If the external environment evaluation index exceeds threshold one, the external environment is judged to be high risk, and the system will immediately trigger an alarm and suspend all image acquisition and processing operations. At this time, the system should initiate external environment adjustment measures, such as adjusting the camera's exposure settings to cope with light fluctuations, or recalibrating the time synchronization between the camera and the processing unit until the external environment returns to normal; if the external environment evaluation index is lower than or within the range of threshold one, the external environment is judged to be low risk or normal, and the system allows the collection and processing of internal environment information to continue;

[0091] For the internal environment evaluation index, the system compares it with the threshold 2. If the internal environment evaluation index exceeds the threshold 2, it means that there is an abnormality inside the system, such as excessive processor load, increased data transmission delay or serious data packet loss. The system should suspend the synchronization of the camera and the algorithm and make internal adjustments in time; the adjustment measures include reducing the processor load, optimizing the network transmission path or reallocating system resources to ensure that the system can resume operation under appropriate conditions; when the internal environment evaluation index is lower than or within the range of the threshold 2, the system determines that the internal environment is normal and continues the synchronization of the camera and the algorithm and data processing;

[0092] During the implementation of measures after the rating, it is necessary to ensure that the risk assessment of the external environment takes precedence over the processing of the internal environment. Only after the external environment is assessed as low risk can the system conduct internal assessment and adjustment. The system should regularly reassess the external and internal environments to ensure that parameters are adjusted in time when the environment changes to avoid potential risks. In particular, when an abnormal state is detected, the system must have the ability to automatically repair and restore the normal monitoring process to ensure the continuity and accuracy of structural monitoring.

[0093] It also includes a survey system; the survey system includes a data acquisition and preprocessing module, a deep learning model selection and design module, a model training and optimization module, a model verification and testing module, and a model deployment and real-time monitoring module;

[0094] The data acquisition and preprocessing module collects multi-dimensional data through lidar, camera, inertial measurement unit, and accelerometer. The collected data is filtered, denoised, and standardized before being input into the deep learning model for processing. The radar is used to obtain distance and three-dimensional point cloud data, providing spatial position and shape information on the surface of the structure. The camera is responsible for capturing image data for analyzing surface features such as cracks and deformation. The inertial measurement unit (IMU) measures changes in azimuth and angle, providing attitude, tilt, and rotation information of the structure. The accelerometer is used to measure vibration and acceleration data, and detect the dynamic response of the structure, such as vibration frequency and intensity. These sensors are combined to provide comprehensive data support for monitoring or surveying the structure. In the survey system, the data acquisition and preprocessing module is the key first step. Each sensor is responsible for collecting a specific type of information. After data collection is completed, these raw data may contain a lot of inaccurate information due to external environmental influences (such as noise, vibration, etc.). Therefore, the system will use filtering technology (such as Gaussian filtering or Kalman filtering) to remove these noises, so as to ensure that the data input to the model is more accurate and clean; the data is normalized and standardized to ensure that the data from different sensors can be scaled uniformly for subsequent model processing; this process is crucial for efficient survey operations;

[0095] The deep learning model selection and design module uses convolutional neural networks (CNN) to process image data and extract structural surface features according to the structural monitoring task. For time series data, such as vibration or stress changes, long short-term memory networks (LSTM) are used to analyze time series features to ensure that complex dynamic changes can be accurately captured. After the data is ready, the system needs to select a suitable deep learning model according to different data types and tasks. For image data on the surface of the structure, the images taken by the camera may contain subtle cracks, deformations, and surface defects. Convolutional neural networks (CNN) are the best choice for processing image data. CNN can capture local features in the image through its unique convolutional layer design. Subtle features such as cracks or surface changes can be gradually amplified and identified by multiple convolutional layers. Compared with traditional fully connected neural networks, CNN greatly reduces the number of parameters and computing costs, and is particularly suitable for large-scale image processing.

[0096] For time series data, such as structural vibration, displacement or angle change information provided by IMU and accelerometer, long short-term memory network (LSTM) is a more appropriate choice; LSTM can capture data changes over a period of time through its "memory" mechanism, and will not forget more distant historical data; this enables LSTM to maintain high prediction accuracy when dealing with long-term structural changes and vibration mode analysis; compared with traditional recurrent neural networks (RNNs), LSTM solves the problem of gradient disappearance, ensuring that it can still maintain efficient learning capabilities in long-term series data;

[0097] The model training and optimization module trains the model through labeled structural health history data, optimizes the model weights using the back propagation algorithm, uses the cross entropy loss function to predict errors, and ensures that the model converges quickly during the training process through hyperparameter tuning to achieve optimal performance; model training is the core link for the system to learn to identify the structural health status from the data; during the training phase, the system guides the learning of the model by providing a large amount of labeled structural health history data; the labeled data usually includes samples of the structure in normal state, slightly damaged state and severely damaged state; the system optimizes the model weights through the back propagation algorithm so that the model prediction results are as close to the actual situation as possible; the back propagation process depends on the definition of the loss function, and the system uses the cross entropy loss function to deal with classification problems. The cross entropy can measure the difference between the predicted category and the actual category, thereby optimizing the output of the model;

[0098] It should be further explained that during the model training process, the system continuously adjusts the model's weights through the back-propagation algorithm so that the model can better predict the health of the structure; in each training, the model will generate a prediction result based on the input data, and then compare this prediction result with the actual label; in order to measure the difference between the model prediction and the actual result, the system uses the cross-entropy loss function, which can calculate the degree of error of the model based on the probability of each category in the classification problem; the cross-entropy compares the probability given by the model with the label value of the actual category (usually 0 or 1) in each category. If the model's prediction in a certain category is very different from the true label, the cross-entropy loss value will increase significantly; so the value of the loss function It can reflect the overall error of the model; then the system uses the back propagation algorithm to pass this error back to each layer of the neural network; specifically, back propagation calculates the derivative of the loss function with respect to each parameter of the model (i.e., weight and bias), which represents the sensitivity of the model output to each parameter; through these derivatives, the system can gradually update the weight of each neuron so that the output of the model is closer to the actual label; the process of updating weights usually uses the gradient descent algorithm to adjust the weights by reducing the error gradient to ensure that the model error gradually decreases after each iteration; in this way, after multiple trainings, the model can better predict different structural states, ensuring high classification accuracy and reliability in actual monitoring;

[0099] In addition, in order to ensure that the model does not overfit (the model over-learns the training data and loses the ability to generalize) or underfit (the model does not fully learn the rules of the data) during the training process, the system will optimize the training effect by fine-tuning the hyperparameters; hyperparameters include learning rate, batch size, and regularization coefficient, etc.; the learning rate determines the step size when updating the weight each time. A step size that is too large may cause the optimal solution to be missed, while a step size that is too small may cause the convergence speed to be too slow. Therefore, the system needs to dynamically adjust the learning rate to find a value suitable for the current model; the batch size affects the amount of data received by the model during each training. A larger batch helps to smooth gradient changes, while a smaller batch can update weights faster, but may introduce noise; regularization parameters (such as L2 regularization) are used to prevent the model from overfitting during training. It penalizes larger weights and forces the model weights to remain within a reasonable range, thereby reducing excessive reliance on training data;

[0100] Hyperparameter adjustment is usually achieved through cross-validation or random search. Through cross-validation, the system divides the data into multiple subsets and uses different subsets for validation each time to ensure that the hyperparameter combination performs well on different data sets. Random search randomly selects multiple parameter combinations in a given parameter space for testing to find the best combination that can converge quickly and improve model performance. Through this hyperparameter adjustment and optimization, the model can effectively avoid overfitting or underfitting, maintain strong generalization capabilities, and can still accurately judge the health of the structure when processing unseen structural data.

[0101] Model validation and testing module: The performance of the trained model is evaluated on the validation set and test set. The evaluation indicators include accuracy, mean square error (MSE), precision and recall, to ensure that the model can generalize to unseen structural data, conduct accurate monitoring and prediction, and continuously optimize the model to improve performance. The trained model needs to be evaluated through independent validation and test sets to evaluate its effect in practical applications. During the model validation process, the system will use a validation data set different from the training data to evaluate the performance of the model. The data in the validation set will not be used for training, but to check the generalization ability of the model, that is, whether it can maintain good performance on unseen data. The system calculates multiple evaluation indicators such as accuracy (measures the proportion of correct predictions made by the model), precision (the accuracy of the model when predicting positive examples) and recall (the ability of the model to find all positive examples) by comparing the model prediction results with the actual results of the validation set. These evaluation indicators not only help detect the overall performance of the model, but also help identify whether there is overfitting (the model performs well on the training data but performs poorly on new data) or underfitting (the model does not fully learn the patterns in the training data).

[0102] If the model performs poorly on the validation set, the system will adjust the model's hyperparameters, such as modifying the learning rate, batch size, or increasing the regularization strength; the role of the validation set is to provide real-time feedback for each stage of model training to ensure that the model remains stable on different data sets without relying on specific training data;

[0103] The test set is used to finally verify the actual effect of the model. The data in the test set is completely independent of the training set and the validation set. It is used to simulate new data in the real environment to confirm whether the model's predictive ability is sufficient to cope with actual scenarios. The test set usually includes various types of structural health status data, and the system will test the performance of the model on these data. If the model can effectively identify abnormal conditions or health changes in the structure, such as cracks or abnormal displacements, it means that the model has good practical application capabilities. If the model's performance on the test set is not ideal, the system will consider incremental learning, gradually adding new data to continue training, or adjusting the model structure to improve its accuracy and reliability in handling complex tasks.

[0104] Model deployment and real-time monitoring module: The trained deep learning model is deployed on the edge device or cloud to process the monitoring data from the sensor in real time and analyze and judge the health status of the structure. It responds quickly through edge computing and triggers an early warning immediately when a structural abnormality is detected. Once the model passes verification and testing, the system will deploy it to the edge device or cloud for real-time monitoring of the health status of the structure. In actual survey scenarios, the edge device is directly connected to the sensor and can receive and process sensor data from lidar, camera, IMU and accelerometer in real time. The advantage of edge computing is that it can process data quickly, reduce the delay caused by data transmission, and ensure that the system can judge whether the structure is abnormal in a very short time. Through the real-time reasoning ability of the deep learning model, the system can quickly analyze the health status of the structure. When an abnormal state is detected (such as structural displacement exceeding the safety threshold or abnormal vibration frequency), the system will immediately issue an early warning signal and generate a detailed health report for the operator's reference. The system is not limited to real-time monitoring, but can also self-optimize the model through continuously collected new data to improve the accuracy and robustness of the model, ensuring that the system always has a high judgment ability during long-term monitoring.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A structure intelligent monitoring device based on deep learning, including a camera image acquisition module, characterized in that: It also includes an external environment information acquisition module, an external environment data comparison module, an internal environment information acquisition module, an internal environment data comparison module, a camera and algorithm synchronization module and an abnormal state processing module; The camera image acquisition module is used to acquire real-time images of the monitored object. Real-time image acquisition is used to capture the displacement, cracks or deformation of the monitored object, obtain image data through the camera device, and adjust the acquisition parameters in real time in combination with external environment information; The external environment information acquisition module is used to collect external environment information that affects image quality and system synchronization. The external environment information includes system clock deviation rate information and ambient light intensity fluctuation rate information; The external environment data comparison module is used to compare the collected external environment information with a preset threshold value 1; the external environment data comparison module generates an external environment evaluation index by using the collected external environment information; if the external environment evaluation index exceeds threshold value 1, the system will judge the external environment as high risk, trigger an alarm, and suspend subsequent processing; if the external environment evaluation index is lower than threshold value 1 or is within the range of threshold value 1, the external environment is judged as low risk, and the system is allowed to continue to collect and process internal environment information; The internal environment information acquisition module is used to monitor the operating status of the device system; the internal environment information includes processor load rate information, data transmission delay rate information and signal transmission integrity rate information; The internal environment data comparison module is used to analyze the collected processor load rate information, data transmission delay rate information and signal transmission integrity rate information, and generate an internal environment data evaluation index; compare the internal environment data evaluation index with the preset threshold value 2 to determine whether the device system is suitable for synchronization processing of the camera and the algorithm; if the internal environment data evaluation index exceeds the threshold value 2, the current environment is determined to be in an abnormal state, and the device system suspends the synchronization process and adjusts the parameters; if the internal environment data evaluation index is lower than the threshold value 2 or is within the range of the threshold value 2, the current environment is determined to be in a normal state, and the device system continues data processing; The camera and algorithm synchronization module is used to synchronize the image data collected by the camera with the deep learning algorithm; The abnormal state processing module is used to handle abnormal situations during the synchronization process between the camera and the algorithm; if the device system detects abnormal internal environmental data during the synchronization process, the abnormal state processing module will suspend the current processing and re-evaluate the external and internal environmental information.

2. The structure intelligent monitoring device based on deep learning according to claim 1, characterized in that: The system clock deviation rate information includes a timestamp accuracy index; The ambient light intensity fluctuation rate information includes a light intensity variation index; The proposed timestamp precision index is The timestamp accuracy index is used to evaluate the timestamp collected by the camera. and the time the processing unit receives The synchronization accuracy between the two frames is calculated by And introduce the cumulative time deviation square ratio ; ； in Indicates the timestamp precision index, which is used to reflect the accuracy of time synchronization in the system; is the adjustment coefficient, which is used to control the impact of time difference on the index; Indicates The time difference of the frames, equal ; Indicates the camera acquisition timestamp; Indicates the timestamp received by the processing unit; Indicates the square ratio of time deviation, which is used to amplify the impact of time difference on synchronization accuracy; The proposed light intensity variation index is The light intensity variation index is used to evaluate the impact of light intensity fluctuations on image acquisition quality and calculate the light intensity variation rate of each frame. And the maximum and minimum ratio of light intensity ; ； in Indicates the light intensity variation index, reflecting the impact of light fluctuations on image quality; represents the adjustment coefficient, which controls the sensitivity of light fluctuations to the index; Indicates The frame illumination change rate is calculated as: ； in is the maximum value of light intensity, indicating the peak value of light fluctuation; is the minimum value of light intensity, indicating the lowest point of light fluctuation; It represents the ratio of the maximum to minimum light intensity, reflecting the amplitude of light fluctuation.

3. The structure intelligent monitoring device based on deep learning according to claim 2, characterized in that: By combining the timestamp precision index and light intensity variation index To evaluate the comprehensive impact of the external environment on the system, and obtain the external environment evaluation index ; ； in It is the external environment evaluation index, reflecting the comprehensive impact of the external environment on the system; is the adjustment coefficient, which controls the weight of the impact of time synchronization and illumination changes on the external environment; It is the timestamp precision index, which reflects the accuracy of time synchronization; is the light intensity variation index, reflecting the impact of light fluctuations on image acquisition; is the product of time synchronization and illumination change, indicating the joint influence of the two; is the Euclidean distance between time synchronization and illumination fluctuation, representing the joint deviation of the two.

4. The structure intelligent monitoring device based on deep learning according to claim 3 is characterized by: The processor load rate information includes an algorithm processing delay index; the data transmission delay rate information includes a network jitter index; the signal transmission integrity rate information includes a data packet loss index; The proposed algorithm processing delay index is The algorithm processing delay index is used to measure the load rate and processing delay of the system processing unit and calculate the processing delay of each frame. and processor load fluctuations ; ； in Indicates the algorithm processing delay index, which is used to reflect the severity of the processing delay; It represents the adjustment coefficient, which controls the impact of delay and load fluctuation on the index; Indicates the processing delay of each frame, indicating the length of processing time; Indicates the maximum value of the processor load, indicating the peak value in the load fluctuation; Indicates the minimum value of the processor load, indicating the lowest point in the load fluctuation; Indicates the load fluctuation ratio, which is used to evaluate the load fluctuation; The proposed network jitter index is ; The network jitter index is used to evaluate the fluctuation of data transmission delay and calculate the transmission time variation of each frame. The maximum and minimum time ratio of network transmission ; ； in Indicates the network jitter index, which is used to reflect the volatility of network transmission delay; Indicates the adjustment coefficient, which controls the impact of jitter fluctuations on the index; Indicates The transmission delay of frames reflects the fluctuation of network transmission time; Indicates the maximum value of network transmission delay, reflecting the highest delay of network transmission; Indicates the minimum value of network transmission delay, reflecting the lowest delay of network transmission; It indicates the ratio of the maximum value to the minimum value of the transmission delay, reflecting the amplitude of the transmission jitter; The proposed packet loss index is ,Packet loss index is used to evaluate the packet loss rate in network transmission; ； in is the packet loss index, which reflects the impact of the packet loss rate on system transmission; To adjust the coefficient, we control the impact of packet loss and fluctuation on the index; For the The packet loss rate of the frame indicates the proportion of lost data packets; is the maximum value of the expected data packet; is the minimum value of the received data packets; It is the packet loss fluctuation ratio, which reflects the fluctuation amplitude of the packet loss rate.

5. The structure intelligent monitoring device based on deep learning according to claim 4, characterized in that: Processing delay index through nonlinear combination algorithm , Network Jitter Index and Packet Loss Index To evaluate the health status of the system and obtain the internal environment evaluation index ; By combining the Euclidean distance and product of the three indexes, the comprehensive impact of various factors within the system is reflected; ； in Represents the internal environment evaluation index, which is used to reflect the health status of the internal operation of the system; It represents the adjustment coefficient, which controls the comprehensive influence weight of each internal factor; Indicates the algorithm processing delay index, reflecting the impact of processing delay; Indicates the network jitter index, reflecting the impact of data transmission delay fluctuations; Indicates the packet loss index, which reflects the packet loss rate in network transmission; It represents the Euclidean distance of the three, indicating the joint deviation of the various factors operating within the system; It represents the product of the three, amplifying the impact when multiple factors go wrong at the same time.

6. The structure intelligent monitoring device based on deep learning according to claim 5, characterized in that: It also includes a survey system; the survey system includes a data acquisition and preprocessing module, a deep learning model selection and design module, a model training and optimization module, a model verification and testing module, and a model deployment and real-time monitoring module; The data acquisition and preprocessing module collects multi-dimensional data through lidar, camera, inertial measurement unit, and accelerometer; the collected data is filtered, denoised, and standardized before being input into the deep learning model for processing; The deep learning model selection and design module uses convolutional neural networks to process image data and extract structural surface features according to the structural monitoring task; for time series data, long short-term memory networks are used to analyze time series features; The model training and optimization module trains the model using labeled structural health history data, optimizes the model weights using the back-propagation algorithm, uses the cross-entropy loss function to predict errors, and ensures model convergence during training through hyperparameter tuning; Model validation and testing module: the performance of the trained model is evaluated on the validation set and the test set. The evaluation indicators include accuracy, mean square error, precision and recall. Model deployment and real-time monitoring module: The trained deep learning model is deployed on edge devices or the cloud to process the monitoring data from sensors in real time and analyze and judge the health status of the structure; Through rapid response through edge computing, an early warning is triggered immediately when structural anomalies are detected.

Images