Structure multi-measuring-point displacement visual perception system and method considering temperature self-compensation

Abstract

The invention belongs to the technical field of visual intelligent monitoring, and particularly discloses a structure multi-measuring-point displacement visual perception system and method considering temperature self-compensation, and the method comprises the following steps: collecting a target structure multi-target image and environment temperature data; structure multi-point displacement extraction based on a multi-target detection algorithm; temperature-induced error compensation based on time sequence deep learning; and carrying out edge calculation and real-time monitoring and early warning. According to the method, high-precision real-time monitoring of multi-measuring-point displacement of the structure is achieved, the long-term monitoring reliability of visual perception is improved through the temperature self-compensation technology, and the method has great significance in improving the health monitoring level of the structure.

Description

Technical Field

[0001] The present invention relates to the technical field of visual intelligent monitoring, and in particular to a system and method for visually perceiving displacement of multiple measuring points of a structure considering temperature self-compensation. Background Art

[0002] With the rapid development of optical cameras and semiconductor chip technologies, the application of computer vision in structural health monitoring has garnered widespread attention. However, the impact of temperature fluctuations on measurement accuracy in long-term monitoring has become a critical issue that needs to be addressed. Temperature changes can cause measurement drift through various pathways, including thermal expansion and contraction of camera components, variations in atmospheric refraction, and increased sensor thermal noise. These factors not only degrade signal integrity but also lead to significant deviations in measurement results, severely impacting the reliability and long-term stability of vision-based monitoring systems.

[0003] While traditional monitoring methods perform well in short-term monitoring, they are also affected by temperature fluctuations and struggle to achieve high spatial resolution and non-contact measurement. In contrast, computer vision technology offers new solutions for long-term structural monitoring through high-resolution imaging, advanced feature tracking algorithms, and multi-point simultaneous monitoring capabilities. However, effectively compensating for measurement errors caused by temperature fluctuations remains a major challenge in practical applications.

[0004] In recent years, researchers both domestically and internationally have conducted extensive research on thermal compensation techniques, proposing a variety of nonlinear temperature drift compensation models. Despite progress, existing models still suffer from significant limitations, such as oversimplification of the multivariate coupling relationships of thermal effects, inadequate treatment of the cumulative effects of nonlinear thermal drift, and a lack of validation of their adaptability to changing environmental conditions. These issues severely limit the application of vision-based monitoring systems for long-term structural health monitoring. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a visual perception system and method for structural multi-point displacement considering temperature self-compensation. The system and method have the advantages of high precision, temperature self-compensation, and synchronous monitoring of multiple measuring points. It can be used for displacement measurement and analysis of moving structures, and can eliminate camera measurement errors caused by ambient temperature changes, thereby realizing accurate assessment of structural status and real-time early warning, solving the problems mentioned in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a structural multi-point displacement visual perception system considering temperature self-compensation, comprising two modules: software and hardware. The software module includes a multi-point displacement extraction module based on deep learning and a temperature-induced error compensation module. The hardware module consists of a high-resolution industrial camera, a precision optical lens, an edge computing module, a high-precision temperature sensing module, and an Internet of Things module.

[0007] The multi-point displacement extraction module uses a target detection algorithm without a priori anchor box to extract target features, combines a prediction algorithm with an adaptive search method to track the target position, and obtains high-precision target spatial coordinates through three-dimensional position calculation;

[0008] The temperature-induced error compensation module uses the temperature sensing module to obtain ambient temperature data, adopts the time domain convolutional network model to model the temperature and camera displacement measurement errors, and accurately compensates for the camera measurement errors caused by temperature through data enhancement and time series processing technology;

[0009] The hardware module acquires structural motion displacement images through industrial cameras and lenses, uses the edge computing module to process and identify the target position and calculate the displacement change data, combines the temperature sensing module to collect ambient temperature information, and finally realizes wireless data transmission through the Internet of Things module.

[0010] On the other hand, to achieve the above-mentioned purpose, the present invention further provides the following technical solution: a method for visually perceiving displacement of multiple measuring points of a structure considering temperature self-compensation, comprising the following steps:

[0011] Step S1, collecting target structure multi-target images and ambient temperature data;

[0012] Step S2: extracting multi-point displacement of the structure based on a multi-target detection algorithm;

[0013] Step S3: temperature-induced error compensation based on time series deep learning;

[0014] Step S4: edge computing and real-time monitoring and early warning.

[0015] Preferably, in step S1, the target structure multi-target image and ambient temperature data acquisition specifically includes the following steps:

[0016] S11, based on the high-resolution industrial camera, precision optical lens and high-precision temperature sensor module array in the hardware module, activates the temperature sensor module system through an adaptive sampling strategy, sets the sampling frequency parameters, performs multi-point real-time collection and digital processing of the ambient temperature, and temporarily stores the temperature data in a local database;

[0017] S12. Activate the industrial camera system, automatically optimize the camera imaging parameters based on scene features, achieve high-quality image acquisition and preprocessing of dynamic displacement of the structure, and temporarily store the processed image sequence in the local database;

[0018] S13. Use the timing alignment algorithm to accurately synchronize and cross-calibrate the temperature and image data to achieve spatiotemporal consistency of multi-source data.

[0019] Preferably, in step S2, the structure multi-point displacement extraction based on the multi-target detection algorithm specifically includes the following steps:

[0020] S21. Use a camera to obtain real-time infrared target images, and calculate the displacement and deformation of the target structure based on the target pose changes in the infrared target images; first, use the target detection algorithm without prior anchor frames, and use the deep aggregation DLA framework as the backbone network to deeply extract and aggregate features;

[0021] S22. Use the prediction tracking technology to predict the true value of the target center in the current frame, and perform the spot search for the next frame by calculating the gate size. The new gate size is:

[0022]

[0023] Among them, A i is the pixel area of ​​any point A in the i-th frame, point B is the point closest to point A in a certain direction, point C is the point closest to point A in the normal direction of the line connecting points A and B, r A,i 、r B,i 、r C,i are the actual radius of points A, B, and C in the i-th frame, respectively. AC,i 、D AB,i is the actual distance from point A to point B and point C;

[0024] By updating the gate size, the spot search range is greatly reduced. Then, with the help of Kalman or extended Kalman filtering technology, the actual movement of the spot is predicted. The comprehensive predictor formula is:

[0025] F[k+1]=τf l [k+1]+(1-τ)f c [k+1]

[0026] Among them, f l is the linear predictor function, f c is the arc predictor function, and the two are combined into a comprehensive predictor through the weighting coefficient τ, thereby obtaining the actual motion information of the light spot;

[0027] S23. According to the camera pinhole imaging principle, the imaging process is:

[0028] sp=KTP

[0029] Where s is regarded as the target scale coefficient at a certain moment, p is the homogeneous pixel coordinate of the point projected in the image, K is the camera intrinsic parameter matrix, the translation matrix T is expressed as the displacement components of the three-dimensional point in the horizontal, vertical and distance directions, and P is the homogeneous coordinate of any point in the world coordinate system;

[0030] In order to obtain the translation matrix T, the IPPE pose solving algorithm is used. Based on the coplanarity of the observation points, the unit orthogonality of the rotation matrix is ​​used to solve the relative pose relationship between the plane where the cooperative target is located and the camera coordinate system. The translation matrix T is estimated and nonlinear optimization is performed using the virtual visual servoing algorithm to obtain the high-precision world coordinate system coordinates of the target.

[0031] S24. Obtain the pixel coordinates of the target in each frame of the image through the target detection method without prior anchor box. This process generates a geometrically consistent corresponding set: {x p,t ,y p,t}, where superscript t represents the time frame index;

[0032] Through each frame I t The displacement field is quantified by differential coordinate analysis between reference frames I0:

[0033]

[0034] where Δu t represents the pixel-level displacement vector of the target at time t, Δx t and Δy t Respectively represent the pixel displacement of the target in the horizontal and vertical directions at time t;

[0035] The acquired pixel-level displacement measurements are converted into physical displacement values ​​using a calibrated scaling factor, which is determined by geometric calibration using a reference target of known size in the imaging plane and is expressed as:

[0036]

[0037] D t =SF×Δu t

[0038] Among them, Ω physical represents the prior physical size of the calibration target, Ω pixel Represents the corresponding pixel measurement value on the image plane, SF is used as the conversion factor, and D t represents the actual physical displacement vector of the target at time t.

[0039] Preferably, in step S3, the temperature-induced error compensation based on time series deep learning specifically includes the following steps:

[0040] S31. The measured temperature-induced displacement error sequence is combined with the ambient temperature data to construct the initial data set. The data set is expanded using the time sliding window technique, and the data diversity is further improved through the hybrid data enhancement method.

[0041] S32. The processed data is then input into the time domain convolutional network model for training to establish a temperature-induced displacement error self-compensation model:

[0042] M=f(T,D,θ)

[0043] Where M is the temperature-induced displacement error self-compensation model, T is the ambient temperature sequence, D is the measured temperature-induced displacement error sequence, and θ is the network parameter of the time domain convolutional network model;

[0044] S33. Input the collected real-time temperature sequence into the prediction result of the trained time-domain convolutional network model, and subtract the predicted temperature-induced displacement error from the measured displacement to obtain the actual structural displacement after compensation. The compensation process is expressed as:

[0045] D compensated =D measured -D predicted

[0046] Among them, D compensated is the final displacement value after compensation, D measured is the measured displacement value, D predicted is the temperature-induced displacement error value predicted by the time-domain convolutional network model.

[0047] Preferably, in step S32, the temperature-induced displacement error self-compensation model training process adopts the Adam optimization algorithm, and the loss function uses the mean square error MSE:

[0048]

[0049] where y i is the true value, f(x i ) is the model prediction value.

[0050] Preferably, in step S33, the collected real-time temperature sequence is input into the trained time domain convolutional network model:

[0051] X emb =Embedding(X in )

[0052] Among them, X emb ∈R M×D×N Represents input embedding; block embedding is achieved through the full convolution method, specifically by converting X in The shape is changed to X in ∈RM×1×L , which is then fed into a one-dimensional convolutional layer with kernel size P and stride S; subsequently, the backbone network extracts temporal features:

[0053] Z=Backbone(X emb )

[0054] Among them, Backbone (.) consists of stacked time-domain convolutional network modules; finally, the prediction result is obtained through the linear head:

[0055]

[0056] Among them, the final prediction output Represents the predicted value of M variables in a time series of length T.

[0057] Preferably, in step S4, the edge computing and real-time monitoring and early warning specifically include the following steps:

[0058] S41. Input the acquired precise spatial displacement data of the target object after temperature-induced error compensation into the edge computing module for processing;

[0059] S42. By analyzing the displacement change characteristics of multiple measuring points, the structural response characteristics of the moving structure are identified, key information on the vibration characteristics and deformation trends of the structure are obtained, and the dynamic response behavior of the structure is accurately reflected;

[0060] S43. Then, the structural status assessment is performed on the analysis results:

[0061] When an abnormal state is detected, an early warning signal is triggered immediately and the warning result is sent to the cloud database;

[0062] When the status is normal, the system will transmit the warning analysis results wirelessly to the cloud database, providing key data support for the maintenance decision of the structure and ensuring the safe operation of the structure.

[0063] The present invention has the following beneficial effects: Through modular design and multi-dimensional data fusion, the perception system of the present invention effectively improves monitoring stability and reliability under complex working conditions while maintaining micron-level measurement accuracy (processing time for 20 measurement points on a single node does not exceed 200 milliseconds) and real-time response capabilities. The perception method of the present invention enables high-precision real-time monitoring of structural displacement at multiple measurement points, and improves the long-term monitoring reliability of visual perception through temperature self-compensation technology, which is of great significance for improving the level of structural health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of a method and system flow for visually perceiving displacements of multiple measuring points of a structure considering temperature self-compensation in Example 1 of the present invention;

[0065] Figure 2 This is a diagram of the temperature-induced error compensation network architecture for time series deep learning in Example 1 of the present invention;

[0066] Figure 3 This is a schematic diagram of a structural multi-point displacement extraction method of a multi-target detection algorithm in Example 1 of the present invention;

[0067] Figure 4 Schematic diagram of multi-point displacement visual data acquisition in Example 2 of the present invention;

[0068] Figure 5 Schematic diagram of the displacement of a bridge pier monitored by the non-contact visual measurement system in Example 2 of the present invention;

[0069] Figure 6 Schematic diagram of reference point displacement monitored by the non-contact visual measurement system in Example 2 of the present invention;

[0070] Figure 7 This is a schematic diagram of ambient temperature change data monitored by the temperature sensing module in Example 2 of the present invention;

[0071] Figure 8 Schematic diagram of the original structural deformation of the bridge before temperature compensation in Example 2 of the present invention;

[0072] Figure 9 Schematic diagram of the structural deformation of the bridge before temperature compensation in Example 2 of the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] Example 1

[0075] See also Figure 1 The present invention provides a technical solution: a structural multi-point displacement visual perception system considering temperature self-compensation, including two modules: software and hardware; wherein, the software module includes a multi-point displacement extraction module based on deep learning and a temperature-induced error compensation module; the hardware module consists of a high-resolution industrial camera, a precision optical lens, an edge computing module, a high-precision temperature sensing module and an Internet of Things module.

[0076] The multi-point displacement extraction module uses a target detection algorithm without a priori anchor frame to extract target features, combines a prediction algorithm with an adaptive search method to track the target position, and obtains high-precision target spatial coordinates through three-dimensional position calculation.

[0077] The temperature-induced error compensation module uses a temperature sensing module to obtain ambient temperature data, adopts a time-domain convolutional network model to model the temperature and camera displacement measurement errors, and accurately compensates for the camera measurement errors caused by temperature through data enhancement and time series processing technology.

[0078] The hardware module acquires structural motion displacement images through industrial cameras and lenses, uses the edge computing module to process and identify the target position and calculate the displacement change data, combines the temperature sensing module to collect ambient temperature information, and finally realizes wireless data transmission through the Internet of Things module.

[0079] A visual perception method of structural multi-point displacement considering temperature self-compensation, such as Figure 1 As shown, the following steps are included:

[0080] Step S1, collecting target structure multi-target images and ambient temperature data; specifically includes the following steps:

[0081] S11. Establish a structural displacement visual monitoring data acquisition system based on multi-source sensor fusion, including a high-resolution industrial camera, precision optical lens, and a high-precision temperature sensor module array. Based on the high-resolution industrial camera, precision optical lens, and high-precision temperature sensor module array in the hardware module, activate the temperature sensor module system through an adaptive sampling strategy, set the optimal sampling frequency parameters, perform multi-point real-time acquisition and digital processing of ambient temperature, and temporarily store the temperature data in a local database;

[0082] S12. Activate the industrial camera system, automatically optimize the camera imaging parameters based on scene features, achieve high-quality image acquisition and preprocessing of dynamic displacement of the structure, and temporarily store the processed image sequence in the local database;

[0083] S13. Use the timing alignment algorithm to accurately synchronize and cross-calibrate the temperature and image data to achieve spatiotemporal consistency of multi-source data.

[0084] Step S2, extracting multi-point displacement of the structure based on a multi-target detection algorithm; specifically includes the following steps:

[0085] S21. Use a camera to obtain real-time infrared target images, and calculate the displacement and deformation of the target structure based on the target pose changes in the infrared target image. First, use the target detection algorithm without prior anchor frames, and use the deep aggregation DLA framework as the backbone network to deeply extract and aggregate features; the principle diagram of the method is shown in Figure 3The DLA framework consists of two parts: hierarchical deep aggregation (HAD) and iterative deep aggregation (IAD). The former is an iterative method between stages, recursively combining blocks within each stage and fusing each stage together, retaining and combining channel features at a deep level. The latter is a sequential structure that takes each stage of the former as input and efficiently combines the features of each stage. Inference is performed through the Head layer to obtain the target center point, length, width, and offset information. The target detection method without a priori anchor box can quickly and in real time locate the positions of multiple targets, but its accuracy needs to be improved.

[0086] S22. Use the prediction tracking technology to predict the true value of the target center in the current frame, and perform the spot search for the next frame by calculating the gate size. The new gate size is:

[0087]

[0088] Among them, A i is the pixel area of ​​any point A in the i-th frame, point B is the point closest to point A in a certain direction, point C is the point closest to point A in the normal direction of the line connecting points A and B, r A,i 、r B,i 、r C,i are the actual radius of points A, B, and C in the i-th frame, respectively. AC,i 、D AB,i is the actual distance from point A to point B and point C;

[0089] By updating the gate size, the spot search range is greatly reduced. Then, with the help of Kalman or extended Kalman filtering technology, the actual movement of the spot is predicted. The comprehensive predictor formula is:

[0090] F[K+1]=τf l [k+1]+(1-τ)f c [k+1]

[0091] Among them, f l is the linear predictor function, f c is the arc predictor function, and the two are combined into a comprehensive predictor through the weighting coefficient τ, thereby obtaining the actual motion information (pixels) of the light spot;

[0092] S23. According to the camera pinhole imaging principle, the imaging process is:

[0093] sp=KTP

[0094] Where s is regarded as the target scale coefficient at a certain moment, p is the homogeneous pixel coordinate of the point projected in the image, K is the camera intrinsic parameter matrix, the translation matrix T is expressed as the displacement components of the three-dimensional point in the horizontal, vertical and distance directions, and P is the homogeneous coordinate of any point in the world coordinate system;

[0095] To obtain the translation matrix T, the IPPE (Infinitesimal Plane-based Pose Estimation) pose solution algorithm is used. Based on the coplanarity of the observation points, the unit orthogonality of the rotation matrix is ​​used to solve the relative pose relationship between the plane where the cooperative target is located and the camera coordinate system. The translation matrix T is estimated and nonlinear optimization is performed using the virtual visual servoing algorithm to obtain the high-precision target world coordinate system coordinates.

[0096] S24. Obtain the pixel coordinates of the target in each frame of the image through the target detection method without prior anchor box. This process generates a geometrically consistent corresponding set: {x p,t ,y p,t}, where superscript t represents the time frame index;

[0097] Through each frame I t The displacement field is quantified by differential coordinate analysis between reference frames I0:

[0098]

[0099] where Δu t represents the pixel-level displacement vector of the target at time t, Δx t and Δy t Respectively represent the pixel displacement of the target in the horizontal and vertical directions at time t;

[0100] The acquired pixel-level displacement measurements are converted into physical displacement values ​​using a calibrated scaling factor, which is determined by geometric calibration using a reference target of known size in the imaging plane and is expressed as:

[0101]

[0102] D t =SF×Δu t

[0103] Among them, Ω physical represents the prior physical size of the calibration target, Ω pixel Represents the corresponding pixel measurement value on the image plane, SF is used as the conversion factor, and D t represents the actual physical displacement vector of the target at time t.

[0104] Step S3, temperature-induced error compensation based on time series deep learning, specifically includes the following steps:

[0105] S31. The measured temperature-induced displacement error sequence is combined with the ambient temperature data to construct the initial data set. To expand the data scale, the time sliding window technology is used to expand the data set, and the hybrid data enhancement method is used to further improve the data diversity.

[0106] S32. The processed data is then input into the time domain convolutional network model for training to establish a temperature-induced displacement error self-compensation model:

[0107] M=f(T,D,θ)

[0108] Where M is the temperature-induced displacement error self-compensation model, T is the ambient temperature sequence, D is the measured temperature-induced displacement error sequence, and θ is the network parameter of the time domain convolutional network model;

[0109] The model training process uses the Adam optimization algorithm, and the loss function uses the mean square error MSE:

[0110]

[0111] where y i is the true value, f(x i ) is the model prediction value. To comprehensively evaluate the model performance, four evaluation indicators are introduced: mean absolute error (MAE), root mean square error (RMSE), correlation coefficient (R), and cosine similarity.

[0112] After obtaining the trained model, the compensation model is applied to the compensation process in actual engineering:

[0113] Input the collected real-time temperature sequence into the trained time domain convolutional network model. The network architecture diagram is shown in Figure 2 :

[0114] X emb =Embedding(X n )

[0115] Among them, X emb ∈R M×D×N Represents input embedding; block embedding is achieved through the full convolution method, specifically by converting X in The shape is changed to X in ∈R M×1×L , which is then fed into a one-dimensional convolutional layer with kernel size P and stride S; subsequently, the backbone network extracts temporal features:

[0116] Z=Backbone(X emb )

[0117] Among them, Backbone (.) consists of stacked time-domain convolutional network modules; finally, the prediction result is obtained through the linear head:

[0118]

[0119] Among them, the final prediction output Represents the predicted value of M variables in a time series of length T.

[0120] S33. Input the collected real-time temperature sequence into the prediction result of the trained time-domain convolutional network model, and subtract the predicted temperature-induced displacement error from the measured displacement to obtain the actual structural displacement after compensation. Since the normalization of the measurement distance has been considered during model training, the prediction result can be directly used for compensation without additional scale conversion. The compensation process is expressed as:

[0121] D compensated =D measured -D predicted

[0122] Among them, D compensated is the final displacement value after compensation, D measured is the measured displacement value, D predicted is the temperature-induced displacement error value predicted by the time-domain convolutional network model.

[0123] Step S4, edge computing and real-time monitoring and early warning, specifically includes the following steps:

[0124] S41. Input the acquired precise spatial displacement data of the target object after temperature-induced error compensation into the edge computing module for processing;

[0125] S42. By analyzing the displacement change characteristics of multiple measuring points, the structural response characteristics of the moving structure are identified, and key information on the vibration characteristics and deformation trends of the structure are obtained, thereby accurately reflecting the dynamic response behavior of the structure;

[0126] S43. Then, the structural status assessment is performed on the analysis results:

[0127] When an abnormal state is detected, an early warning signal is triggered immediately and the warning result is sent to the cloud database;

[0128] When the status is normal, the system will transmit the warning analysis results wirelessly to the cloud database, providing key data support for the maintenance decision of the structure and ensuring the safe operation of the structure.

[0129] The invention adopts a modular, layered design, with an industrial camera and autofocus lens as core imaging components, equipped with an ambient light compensation device to achieve clear, all-weather photography. A local processing unit integrates a high-performance processor and a dedicated AI chip, running an intelligent target recognition algorithm and dynamic tracking program to extract target features and measure micron-level displacements. A multi-parameter sensing network, composed of densely distributed temperature monitoring elements, miniature vibration detectors, and a fiber-optic temperature measurement system, simultaneously collects environmental variables. The data hub uses a high-speed serial interface and precise clock synchronization technology to achieve zero-latency transmission of collected data and alignment with nanosecond time scales, and cooperates with a long-range wireless module for data backhaul. The device adopts a fully enclosed protective structure, adapts to a wide operating temperature range (-40°C to 85°C), and features a built-in fault self-detection mechanism and dual power backup system. Each component forms a closed "acquisition-processing-transmission" loop through standardized interfaces: the imaging system continuously acquires structural images, the AI ​​unit analyzes the spatial coordinates of multiple measurement points in real time, the environmental sensor simultaneously records temperature changes, and the intelligent compensation model corrects measurement errors based on temperature drift patterns. Finally, high-precision displacement data is uploaded to the monitoring center via a wireless network, enabling dynamic analysis of structural status and risk warning. Through modular design and multi-dimensional data fusion, the system effectively improves the monitoring stability and reliability under complex working conditions while maintaining micron-level measurement accuracy (the processing time of 20 measurement points on a single node does not exceed 200 milliseconds) and real-time response capabilities.

[0130] Example 2

[0131] Combine Figures 2 to 9 A typical long-span continuous steel bridge case is used to illustrate a method for visually perceiving displacements of multiple measuring points of a structure considering temperature self-compensation. The specific steps are as follows:

[0132] 1) Target structure multi-target image and ambient temperature data acquisition: According to the shooting field size, determine the appropriate camera resolution and acquisition frame rate, control the high-speed camera to capture the vibration image sequence of the long-span continuous rigid frame bridge under vehicle driving and different lighting conditions (such as Figure 4 (as shown in the figure); establish a vibration image database of large-span bridges under different environmental conditions, construct a lightweight deep neural network suitable for infrared target semantic segmentation and image target tracking, and adjust the real-time collected image data by training a massive image database under different environmental conditions to obtain a bridge vibration image sequence with obvious image brightness changes and image features. The pre-processed image data is then temporarily stored in a local database; at the same time, a suitable acquisition frequency is set for the temperature sensor module, and the ambient temperature is collected and digitized at multiple points in real time. The temperature data is temporarily stored in a local database. Finally, a time alignment algorithm is used to accurately synchronize and cross-calibrate the temperature and image data to achieve spatiotemporal consistency of multi-source data.

[0133] The superstructure of the large-span continuous rigid frame bridge selected for this case study is a cast-in-place prestressed concrete box girder with variable height (4.0 meters at midspan and 11.5 meters at the piers), constructed using the balanced cantilever method. The substructure utilizes double-legged solid piers and hollow thin-walled piers, supported by pile group foundations. The bridge spans 1,172 meters, with a main span interval of 180 meters, side spans of 96 meters, a deck height of 16.25 meters, and a maximum height of 209.57 meters. For this test, a synchronized visual measurement system was implemented using two industrial cameras with a resolution of 2448 × 2048 pixels and a sampling rate of 0.1 Hz. Two wireless temperature sensor modules were used to monitor ambient temperature fluctuations, achieving a temperature measurement accuracy of 0.5°.

[0134] 2) Structural multi-point displacement extraction based on multi-target detection algorithm: In this test, camera 1 tracks the displacement of the bridge pier, while camera 2 tracks the stable geodetic reference stake. Three circular targets and one checkerboard target are set on the bridge pier, and one circular target and one checkerboard target are set on the geodetic reference stake. The bridge is monitored by the non-contact vision measurement system. The displacement of the bridge pier and the displacement of the reference point are as follows: Figure 5 、 Figure 6 As shown in Figure 1, it can be seen that the bridge pier displacement obtained from camera 1 increases nonlinearly with the experimental time, while the reference point displacement obtained from camera 2 decreases with the experimental time. This is reasonable because the two optical cameras are installed on the same platform with opposite imaging directions. At the same time, the temperature sensing module records the change of ambient temperature, and its data changes as shown in Figure 1. Figure 7 As shown, it can be clearly seen that the trends of temperature change and displacement change are roughly similar, indicating that there is a certain connection between the two.

[0135] 3) Temperature-induced error compensation based on time series deep learning: After the acquired temperature and displacement data are made into a temperature compensation data set, they are input into the time domain convolutional network model for training and saving, and then the measured ambient temperature data (such as Figure 8 As shown in the figure, the model is saved to obtain the predicted temperature-induced displacement sequence, and finally the actual original displacement is compensated for the error. The compensation result is shown in the figure below. Figure 9 As shown in the figure, the proposed method achieved excellent performance in terms of both correlation coefficient (R = 0.9998) and cosine similarity index (Cos_Sim = 0.9998), and also achieved significant improvement in error statistics (MAE: 0.0020mm, RMSE: 0.0022mm). These results jointly verify the proposed method's improved accuracy and ability to compensate for temperature-induced measurement errors. It can be seen that the camera's temperature-induced displacement predicted by the proposed method is consistent with the actual temperature-induced displacement, further verifying the correctness and effectiveness of the proposed method.

[0136] 4) Edge computing and real-time monitoring and early warning: Based on the precise spatial displacement data of the target object after temperature compensation, it is input into the edge computing module for processing. By analyzing the displacement change characteristics of multiple measuring points, the structural response characteristics of the moving structure are identified, and key information such as the structure's vibration characteristics and deformation trends are obtained, thereby accurately reflecting the dynamic response behavior of the structure. The system then conducts a structural status assessment based on the analysis results: when an abnormal status is detected, an early warning signal is immediately triggered and the warning results are sent to the cloud database; when the status is normal, the system transmits the early warning analysis results to the cloud database via wireless means, providing key data support for maintenance decisions of the structure and ensuring the safe operation of the structure.

[0137] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0138] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0139] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0140] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0141] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0142] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0143] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A structural multi-point displacement visual perception system considering temperature self-compensation, characterized by: It includes two modules: software and hardware. The software module includes a multi-point displacement extraction module based on deep learning and a temperature-induced error compensation module. The hardware module consists of a high-resolution industrial camera, a precision optical lens, an edge computing module, a high-precision temperature sensor module, and an Internet of Things module. The multi-point displacement extraction module uses a target detection algorithm without a priori anchor box to extract target features, combines a prediction algorithm with an adaptive search method to track the target position, and obtains high-precision target spatial coordinates through three-dimensional position calculation; The temperature-induced error compensation module uses the temperature sensor module to obtain ambient temperature data, adopts the time domain convolutional network model to model the temperature and camera displacement measurement errors, and accurately compensates for the camera measurement errors caused by temperature through data enhancement and time series processing technology; The hardware module acquires structural motion displacement images through industrial cameras and lenses, uses the edge computing module to process and identify the target position and calculate the displacement change data, combines the temperature sensing module to collect ambient temperature information, and finally realizes wireless data transmission through the Internet of Things module.

2. A perception method based on the structure multi-point displacement visual perception system considering temperature self-compensation as described in claim 1, characterized in that: The steps include: Step S1, collecting target structure multi-target images and ambient temperature data; Step S2: extracting multi-point displacement of the structure based on a multi-target detection algorithm; Step S3: temperature-induced error compensation based on time series deep learning; Step S4: edge computing and real-time monitoring and early warning.

3. The method for visually perceiving displacement of a structure at multiple measuring points with consideration of temperature self-compensation according to claim 2, characterized in that: In step S1, the target structure multi-target image and ambient temperature data acquisition specifically includes the following steps: S11, based on the high-resolution industrial camera, precision optical lens and high-precision temperature sensor module array in the hardware module, activates the temperature sensor module system through an adaptive sampling strategy, sets the sampling frequency parameters, performs multi-point real-time collection and digital processing of the ambient temperature, and temporarily stores the temperature data in a local database; S12. Activate the industrial camera system, automatically optimize the camera imaging parameters based on scene features, achieve high-quality image acquisition and preprocessing of the dynamic displacement of the structure, and temporarily store the processed image sequence in the local database; S13. Use the timing alignment algorithm to accurately synchronize and cross-calibrate the temperature and image data to achieve spatiotemporal consistency of multi-source data.

4. The method for visually perceiving displacement of a structure at multiple measuring points with temperature self-compensation according to claim 2, characterized in that: In step S2, the structure multi-point displacement extraction based on the multi-target detection algorithm specifically includes the following steps: S21. Use a camera to obtain real-time infrared target images, and calculate the displacement and deformation of the target structure based on the target pose changes in the infrared target image; first, use the target detection algorithm without prior anchor frames, and use the deep aggregation DLA framework as the backbone network to deeply extract and aggregate features; S22. Use the prediction tracking technology to predict the true value of the target center in the current frame, and perform the spot search for the next frame by calculating the gate size. The new gate size is: Among them, A i is the pixel area of ​​any point A in the i-th frame, point B is the point closest to point A in a certain direction, point C is the point closest to point A in the normal direction of the line connecting points A and B, r A,i 、r B,i 、r C,i are the actual radius of points A, B, and C in the i-th frame, respectively. AC,i 、D AB,i is the actual distance from point A to point B and point C; By updating the gate size, the spot search range is greatly reduced. Then, with the help of Kalman or extended Kalman filtering technology, the actual movement of the spot is predicted. The comprehensive predictor formula is: F[k+1]=τf l [k+1]+(1-τ)f c [k+1] Among them, f l is the linear predictor function, f c is the arc predictor function, and the two are combined into a comprehensive predictor through the weighting coefficient τ, thereby obtaining the actual motion information of the light spot; S23. According to the camera pinhole imaging principle, the imaging process is: sp=KTP Where s is regarded as the target scale coefficient at a certain moment, p is the homogeneous pixel coordinate of the point projected in the image, K is the camera intrinsic parameter matrix, the translation matrix T is expressed as the displacement components of the three-dimensional point in the horizontal, vertical and distance directions, and P is the homogeneous coordinate of any point in the world coordinate system; In order to obtain the translation matrix T, the IPPE pose solving algorithm is used. Based on the coplanarity of the observation points, the unit orthogonality of the rotation matrix is ​​used to solve the relative pose relationship between the plane where the cooperative target is located and the camera coordinate system. The translation matrix T is estimated and nonlinear optimization is performed using the virtual visual servoing algorithm to obtain the high-precision world coordinate system coordinates of the target. S24. Obtain the pixel coordinates of the target in each frame of the image through the target detection method without prior anchor box. This process generates a geometrically consistent corresponding set: {x p,t ,y p,t }, where superscript t represents the time frame index; Through each frame I t The displacement field is quantified by differential coordinate analysis between reference frames I0: where Δu t represents the pixel-level displacement vector of the target at time t, Δx t and Δy t Respectively represent the pixel displacement of the target in the horizontal and vertical directions at time t; The acquired pixel-level displacement measurements are converted into physical displacement values ​​using a calibrated scaling factor, which is determined by geometric calibration using a reference target of known size in the imaging plane and is expressed as: D t =SF×Δu t Among them, Ω physical represents the prior physical size of the calibration target, Ω pixel Represents the corresponding pixel measurement value on the image plane, SF is used as the conversion factor, and D t represents the actual physical displacement vector of the target at time t.

5. The method for visually perceiving displacement of a structure at multiple measuring points with consideration of temperature self-compensation according to claim 2, characterized in that: In step S3, the temperature-induced error compensation based on time series deep learning specifically includes the following steps: S31. The measured temperature-induced displacement error sequence is combined with the ambient temperature data to construct the initial data set. The data set is expanded using the time sliding window technique, and the data diversity is further improved through the hybrid data enhancement method. S32. The processed data is then input into the time domain convolutional network model for training to establish a temperature-induced displacement error self-compensation model: M=f(T,D,θ) Where M is the temperature-induced displacement error self-compensation model, T is the ambient temperature sequence, D is the measured temperature-induced displacement error sequence, and θ is the network parameter of the time domain convolutional network model; S33. Input the collected real-time temperature sequence into the prediction result of the trained time-domain convolutional network model, and subtract the predicted temperature-induced displacement error from the measured displacement to obtain the actual structural displacement after compensation. The compensation process is expressed as: D compensated =D measured -D predicted Among them, D compensated is the final displacement value after compensation, D measured is the measured displacement value, D predicted is the temperature-induced displacement error value predicted by the time-domain convolutional network model.

6. The method for visually perceiving displacement of a structure at multiple measuring points with consideration of temperature self-compensation according to claim 5, characterized in that: In step S32, the temperature-induced displacement error self-compensation model training process adopts the Adam optimization algorithm, and the loss function uses the mean square error MSE: where y i is the true value, f(x i ) is the model prediction value.

7. The method for visually perceiving displacement of a structure at multiple measuring points with consideration of temperature self-compensation according to claim 5, characterized in that: In step S33, the collected real-time temperature sequence is input into the trained time-domain convolutional network model: X emb =Embedding(X in ) Among them, X emb ∈R M×D×N Represents input embedding; block embedding is achieved through the full convolution method, specifically by converting X in The shape is changed to X in ∈R M×1×L , which is then fed into a one-dimensional convolutional layer with kernel size P and stride S; subsequently, the backbone network extracts temporal features: Z=Backbone(X emb ) Among them, Backbone (.) consists of stacked time-domain convolutional network modules; finally, the prediction result is obtained through the linear head: Among them, the final prediction output Represents the predicted value of M variables in a time series of length T.

8. The method for visually perceiving displacement of a structure at multiple measuring points with consideration of temperature self-compensation according to claim 2, characterized in that: In step S4, the edge computing and real-time monitoring and early warning specifically include the following steps: S41. Input the acquired precise spatial displacement data of the target object after temperature-induced error compensation into the edge computing module for processing; S42. By analyzing the displacement change characteristics of multiple measuring points, the structural response characteristics of the moving structure are identified, key information on the vibration characteristics and deformation trends of the structure are obtained, and the dynamic response behavior of the structure is accurately reflected; S43. Then, the structural status assessment is performed on the analysis results: When an abnormal state is detected, an early warning signal is triggered immediately and the warning result is sent to the cloud database; When the status is normal, the system will transmit the warning analysis results wirelessly to the cloud database, providing key data support for the maintenance decision of the structure and ensuring the safe operation of the structure.

Images