House structure safety edge visual monitoring method and device, equipment and storage medium

Abstract

The application discloses a kind of house structure safety edge visual monitoring method and device, equipment and storage medium, it is related to building safety monitoring technical field, comprising: acquisition multimodal data, wherein, multimodal data includes visible light video stream, thermal infrared video stream and point cloud data;Multi-modal data is handled with parameter synchronization extraction, and first displacement sequence, first inclination angle, first crack width and first hollow mask are obtained;First displacement sequence, first inclination angle, first crack width and first hollow mask are encapsulated, and first feature data package is obtained;And first feature data package is structured numerical compression, and compression vector is obtained, and compression vector is sent to cloud platform, so that cloud platform updates digital twin model in real time according to first feature data package and triggers early warning mechanism, completes house structure safety monitoring, realizes the high-precision, non-contact, real-time monitoring of the multi-parameter such as building structure inclination, settlement, vibration, crack and hollow.

Description

Technical Field

[0001] This invention relates to the field of building safety monitoring technology, and in particular to a method, device, equipment and storage medium for visual monitoring of the safety edges of building structures. Background Technology

[0002] Building structural safety is directly related to urban public safety and the protection of residents' lives and property. With the stock of existing buildings increasing year by year, management departments urgently need to conduct continuous, real-time, and low-cost deformation and damage monitoring of large-scale buildings in order to detect potential hazards such as tilting, settlement, cracks, and hollow areas as early as possible, and achieve risk warning and precise maintenance.

[0003] Traditional monitoring methods primarily rely on contact sensors or high-precision measuring instruments, which are complex to deploy, expensive to maintain, and highly susceptible to environmental interference. For example, the deployment cost of contact sensors is approximately 5,000-10,000 yuan per building, and they are difficult to cover large-scale building complexes. Cloud-based vision solutions, on the other hand, face bottlenecks such as high bandwidth and long processing latency in raw video transmission. Taking the transmission of 1080p high-definition video as an example, the bitrate is typically 5-10 Mbps, and transmitting video data from tens of thousands of buildings requires at least 50-100 Gbps of bandwidth. Meanwhile, cloud processing latency can reach several seconds to tens of seconds, making it difficult to meet real-time monitoring requirements.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, device, equipment, and storage medium for visual monitoring of building structure safety edges, aiming to solve the technical problem of how to perform multi-parameter synchronous monitoring of building structures.

[0006] To achieve the above objectives, the present invention provides a method for visual monitoring of the safety edges of building structures, the method comprising the following steps:

[0007] Collect multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data;

[0008] The multimodal data is subjected to parameter synchronous extraction processing to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask;

[0009] The first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask are encapsulated to obtain a first feature data packet;

[0010] The first feature data packet is subjected to structured numerical compression to obtain a compressed vector, which is then sent to the cloud platform so that the cloud platform can update the digital twin model in real time based on the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring.

[0011] In one embodiment, the step of performing parameter synchronous extraction processing on the multimodal data to obtain a first displacement sequence, a first tilt angle, a first crack width, and a first hollow mask includes:

[0012] Subpixel displacement calculations are performed on the visible light video stream to obtain a first displacement sequence;

[0013] The visible light video stream is subjected to monocular 3D reconstruction and planar fitting to obtain the first tilt angle;

[0014] The visible light video stream is segmented to obtain a first crack width;

[0015] Temperature gradient analysis was performed on the thermal infrared video stream to obtain the first hollow mask.

[0016] In one embodiment, the step of performing sub-pixel displacement calculation on the multimodal data to obtain a first displacement sequence includes:

[0017] The visible light video stream is subjected to pyramid layering processing to obtain a multi-level pyramid image;

[0018] Phase consistency features are extracted from the multi-level pyramid image to obtain a first phase feature map;

[0019] Inter-frame optical flow estimation is performed on the first phase feature map to obtain the first optical flow field;

[0020] The first optical flow field is scaled to obtain the first displacement sequence.

[0021] In one embodiment, the step of performing inter-frame optical flow estimation on the first phase feature map to obtain the first optical flow field includes:

[0022] Perform a Fourier transform on the first phase feature map to obtain the first frequency domain map;

[0023] The phase difference is calculated on the first frequency domain map to obtain the first phase difference map;

[0024] Subpixel displacement mapping is performed on the first phase difference map to obtain the first optical flow field.

[0025] In one embodiment, the step of performing monocular 3D reconstruction and planar fitting on the visible light video stream to obtain a first tilt angle includes:

[0026] Keyframes are extracted from the visible light video stream to obtain a second video frame sequence;

[0027] Feature point extraction and matching are performed on the second video frame sequence to obtain the first matching point pair;

[0028] Motion recovery structure calculation is performed on the first matched point pair to obtain the first sparse point cloud;

[0029] The first sparse point cloud is reconstructed by densification to obtain the first point cloud data;

[0030] Random sampling consistency screening is performed on the first point cloud data to obtain the first principal plane point set;

[0031] The normal vector of the first principal plane point set is estimated to obtain the first normal vector, and the angle between the normal vector and the preset reference normal vector is calculated to obtain the first tilt angle.

[0032] In one embodiment, the step of performing densification reconstruction on the first sparse point cloud to obtain the first point cloud data includes:

[0033] The first sparse point cloud is subjected to surface diffusion processing to obtain the first diffusion point set;

[0034] A depth map is generated from the first set of diffusion points to obtain the first depth map;

[0035] Point cloud back projection is performed on the first depth map to obtain the first point cloud data.

[0036] In one embodiment, after the step of performing monocular 3D reconstruction and planar fitting on the visible light video stream to obtain the first tilt angle, the method further includes:

[0037] The first point cloud data and the first sparse point cloud are globally optimized and registered to obtain optimized point cloud data.

[0038] The first principal plane point set is recalculated based on the optimized point cloud data;

[0039] The normal vectors of the recalculated principal plane point set are estimated to obtain the corrected normal vectors.

[0040] The corrected first tilt angle is calculated based on the corrected normal vector and the preset reference normal vector.

[0041] In one embodiment, the step of performing intelligent crack identification on the multimodal data to obtain a first crack width includes:

[0042] The visible light video stream is segmented to obtain a first binary image of the crack;

[0043] The skeleton of the first crack is extracted from the binary image of the first crack to obtain the center line of the first crack.

[0044] The pixel length of the first crack is obtained by performing pixel statistics on the center line of the first crack.

[0045] The pixel length of the first crack is converted into the width of the first crack according to a preset pixel scale.

[0046] In one embodiment, the step of performing void detection processing on the multimodal data to obtain a first void mask includes:

[0047] Temperature gradient calculation is performed on the thermal infrared video stream to obtain a first gradient map;

[0048] The first gradient map is binarized to obtain a first binary gradient map.

[0049] Perform connected component analysis on the first binary gradient graph to obtain the first set of connected regions;

[0050] The area of ​​the first connected region set is filtered, and regions with an area greater than a preset area threshold are retained to obtain the first hollow mask.

[0051] In one embodiment, the step of performing structured numerical compression on the first feature data packet to obtain a compressed vector, and sending the compressed vector to the cloud platform, includes:

[0052] The first feature data packet is input into the autoencoder network to obtain the first compression vector;

[0053] The first compressed vector is sent to the cloud platform via a wireless communication network, so that the cloud platform can reconstruct the first feature data packet through a decoder and perform monitoring and analysis.

[0054] Receive the warning level identifier returned by the cloud platform and write the warning level identifier to local storage.

[0055] In one embodiment, the method further includes:

[0056] Receive a second video frame sequence from the drone platform and use the second video frame sequence as supplementary input to the multimodal data;

[0057] The second video frame sequence is processed using the same procedure as the multimodal data to obtain the second displacement sequence and the second tilt angle.

[0058] By using precise time synchronization and global optimized registration, the second displacement sequence, the second tilt angle, the first displacement sequence, and the first tilt angle are fused into the first feature data packet to obtain the second feature data packet.

[0059] The second feature data packet is sent to the cloud platform so that the cloud platform can perform building structure safety monitoring based on the second feature data packet.

[0060] In one embodiment, the method further includes:

[0061] Read the preset time window length and slice the first feature data packet according to the preset time window length to obtain the first data slice;

[0062] The first data slice is temporally encoded to obtain the first temporal feature;

[0063] The first temporal feature is input into the spatiotemporal graph convolutional network to obtain the first prediction parameters;

[0064] The first prediction parameter is added to the first feature data packet, and the first feature data packet containing the first prediction parameter is sent to the cloud platform so that the cloud platform can predict the building structure safety trend based on the first prediction parameter.

[0065] Furthermore, to achieve the above objectives, the present invention also proposes a visual monitoring device for the safety edge of a building structure, the device comprising:

[0066] The acquisition module is used to acquire multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data.

[0067] The parameter extraction module is used to perform synchronous parameter extraction processing on the multimodal data to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask.

[0068] The data encapsulation module is used to encapsulate the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask to obtain a first feature data packet;

[0069] The monitoring module is used to perform structured numerical compression on the first feature data packet to obtain a compressed vector, and send the compressed vector to the cloud platform so that the cloud platform can update the digital twin model in real time according to the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring.

[0070] Furthermore, to achieve the above objectives, the present invention also proposes a visual monitoring device for the safety edge of a building structure, the device comprising: a memory, a processor, and a visual monitoring program for the safety edge of a building structure stored in the memory and executable on the processor, the visual monitoring program for the safety edge of a building structure configured to implement the steps of the visual monitoring method for the safety edge of a building structure as described above.

[0071] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a building structure safety edge visual monitoring program, wherein the building structure safety edge visual monitoring program, when executed by a processor, implements the steps of the building structure safety edge visual monitoring method described above.

[0072] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the building structure safety edge visual monitoring method described above.

[0073] One or more technical solutions proposed in this application have at least the following technical effects:

[0074] By directly performing subpixel displacement calculation and 3D reconstruction-plane fitting on the monocular visible light video stream in parallel at the edge, the displacement sequence and tilt angle are generated, packaged, and uploaded synchronously within the same calculation cycle. This eliminates the need for multi-sensor deployment and raw video transmission, thereby reducing the amount of system hardware and the amount of data transmitted, and thus lowering the monitoring cost. Real-time calculation and instant transmission shorten the information link. After receiving the feature data packet, the platform can update the digital twin model and output the monitoring results. The overall response time is reduced from minutes in traditional solutions to seconds, enabling low-cost and low-latency building structure safety monitoring. Attached Figure Description

[0075] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart illustrating an embodiment of the visual monitoring method for building structural safety edges provided in this application.

[0078] Figure 2 This is a structural diagram provided for Embodiment 1 of the visual monitoring method for building structural safety edges in this application;

[0079] Figure 3 This is a flowchart illustrating Embodiment 2 of the visual monitoring method for building structural safety edges provided in this application;

[0080] Figure 4 This is a flowchart illustrating Embodiment 3 of the visual monitoring method for building structural safety edges provided in this application;

[0081] Figure 5 This is a schematic diagram of the module structure of the visual monitoring device for building structural safety edges according to an embodiment of this application;

[0082] Figure 6 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the visual monitoring method for building structure safety edges in this application embodiment.

[0083] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0084] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0085] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0086] It should be noted that the main body of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a building structure safety edge visual monitoring device. The following description uses a building structure safety edge visual monitoring device as an example to illustrate this embodiment and the subsequent embodiments.

[0087] Traditional monitoring systems often employ a single data acquisition method, resulting in blind spots and limitations in accuracy. To address this issue, this system innovatively integrates three monitoring platforms, achieving comprehensive monitoring from millimeter to centimeter level and from local to overall coverage through an intelligent collaborative mechanism.

[0088] This solution uses an intelligent task scheduling system to coordinate the work of fixed nodes, drones, and laser scanners, achieving full-coverage monitoring from "millimeter-level local damage" to "centimeter-level overall deformation". It controls the time synchronization deviation of multiple platforms within <5ms and achieves a registration accuracy of 1.5mm, which is 50% better than the traditional ICP algorithm.

[0089] Based on this, embodiments of this application provide a method for visual monitoring of the safety edges of building structures, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the visual monitoring method for building structural safety edges according to this application.

[0090] In this embodiment, the visual monitoring method for the safety edge of a building structure includes steps S10 to S40:

[0091] Step S10: Collect multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data;

[0092] It should be noted that the purpose of this step is to provide uncompressed raw image sources for all subsequent calculations, so as to achieve a one-to-one correspondence between "time" and "image".

[0093] "Visible light video stream" refers to everyday color images captured by ordinary cameras; "multimodal data" refers to a collection of individual images split in chronological order.

[0094] Understandably, the edge computing unit continuously reads images from the camera buffer, numbers each frame at a fixed frame rate (e.g., 25 frames per second), and writes them into a memory queue, thus completing the conversion of the video stream into a frame sequence.

[0095] Figure 2 The architecture of a visual monitoring system for building structural safety edges was demonstrated, covering the entire process from data acquisition to final application.

[0096] The perception layer includes fixed ground monitoring stations, mobile drone inspections, and lidar scanning. These are responsible for collecting visible light video streams, thermal infrared video streams, and point cloud data to provide comprehensive information on the building structure.

[0097] The edge processing layer performs displacement monitoring, tilt calculation, crack identification, and void detection. Through data preprocessing, data fusion compression, and multi-parameter parallel computation, this layer can efficiently extract key features from the acquired data, such as displacement sequences, tilt angles, crack widths, and void masks.

[0098] The cloud platform layer utilizes digital twin modeling and multi-source data fusion technologies to further process and analyze data from the edge layer. Through an intelligent analysis and early warning system, the cloud platform can update the digital twin model in real time and trigger warnings when potential risks are detected.

[0099] The application layer provides a real-time visualization interface, a multi-level early warning mechanism, and decision support functions, enabling users to intuitively monitor the structural safety of buildings and make timely decisions based on the monitoring data.

[0100] Overall, the system architecture enables real-time and efficient monitoring of building structural safety, ensures rapid data processing and accurate analysis, and provides strong technical support for building safety management.

[0101] The preprocessing steps in this solution include:

[0102] Ground-based fixed monitoring nodes. Utilizing high-resolution cameras with zoom lenses, supporting visible light and thermal infrared imaging, covering monitoring needs from close-up components to long-range overall structures. The visible light video stream provides high-resolution texture information, while the thermal infrared video stream captures temperature distribution data. After acquisition, the visible light video stream undergoes adaptive contrast enhancement (CLAHE) and optical distortion correction.

[0103]

[0104] Non-uniformity correction (NUC) and temperature calibration of the thermal infrared video stream:

[0105]

[0106] in: Represents the temperature value (°C); This represents the raw output value of the infrared sensor; k and b are correction coefficients determined through blackbody calibration. This process provides an accurate multimodal data foundation for subsequent analysis.

[0107] Mobile drone inspection platform: can select cameras and lidar to scan building facades according to preset routes and generate high-precision point cloud data.

[0108] Ground-based 3D laser scanner: performs periodic precision measurements.

[0109] Step S20: Perform parameter synchronous extraction processing on the multimodal data to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask;

[0110] It should be noted that the multimodal data includes visible light video streams, thermal infrared video streams, and point cloud data.

[0111] These data provide information on the building's visual appearance, temperature distribution, and three-dimensional spatial information.

[0112] The first displacement sequence is obtained by performing subpixel displacement calculation on the visible light video stream; the first tilt angle is obtained by performing monocular 3D reconstruction and plane fitting on the visible light video stream; the first crack width is obtained by crack segmentation on the visible light video stream; and the first hollow mask is obtained by performing temperature gradient analysis on the thermal infrared video stream. The purpose of this step is to ensure that the required parameters are extracted simultaneously from different types of input data, laying the foundation for subsequent analysis and monitoring.

[0113] Step S30: Encapsulate the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask to obtain the first feature data packet;

[0114] It should be noted that these parameters are integrated into a single data structure and organized in an ordered and recognizable format so that subsequent processing modules can efficiently read and parse this data.

[0115] The purpose of this step is to transform the complex and diverse raw data into a structured feature data package, which facilitates subsequent compression, transmission, and further analysis and processing, thereby supporting real-time monitoring and early warning of building structural safety.

[0116] Step S40: The first feature data packet is subjected to structured numerical compression to obtain a compressed vector. The compressed vector is then sent to the cloud platform so that the cloud platform can update the digital twin model in real time based on the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring.

[0117] It should be noted that the purpose of this step is to compress and upload core results such as displacement and tilt, so that the cloud can monitor the safety status of the house in real time.

[0118] The “first characteristic data packet”, “first compressed vector”, and “early warning level identifier” represent the risk levels of the original packet, the compressed data, and the data transmitted back from the cloud, respectively.

[0119] Understandably, the data packets are compressed to 1 / 150 of their original size using an autoencoder, uploaded via the 4G / 5G MQTT protocol, and immediately updated in the cloud after decoding, with a "yellow / orange / red" warning written back. The edge device stores the returned identifier in its local log, completing a monitoring loop.

[0120] It should be understood that data collected from multiple platforms suffers from inconsistencies in spatiotemporal references and large data volumes. This system employs extended Kalman filtering for data fusion, significantly reducing transmission bandwidth requirements while ensuring data integrity.

[0121] To improve the accuracy and reliability of system state estimation, this system employs Extended Kalman Filter (EKF) for deep fusion of multi-source heterogeneous data. EKF uses a state-space model to optimally estimate observation data (such as visual displacement, laser point cloud tilt angle, and infrared void index) from different platforms, frequencies, and accuracies. Its core lies in a recursive two-step process: first, predicting the current state based on the system's physical model; then, using the latest observation data to weight and correct the predicted value, with the Kalman gain dynamically adjusting its confidence level in both the predicted and observed values.

[0122]

[0123] in: The state estimation vector represents parameters such as displacement, tilt, crack width, and hollowness. Represents the state transition matrix; Represents Kalman gain; Represents the observed value; Represents the observation matrix.

[0124] This algorithm can effectively handle noise and uncertainty in sensor data, and adapt to complex changes in structural behavior through nonlinear approximation. Finally, it outputs a smoother and more reliable optimal state estimate that evolves over time (including displacement, tilt, crack width, degree of voiding, etc.), providing a unified and accurate data foundation for subsequent digital twin-driven and intelligent early warning systems.

[0125] To overcome the bandwidth bottleneck of remote transmission of massive monitoring data, this system adopts an intelligent compression algorithm based on a deep convolutional autoencoder. This algorithm acts as an "intelligent encoder" at the edge, not simply and crudely compressing the original image, but performing nonlinear dimensionality reduction on the processed high-level feature data (such as feature point coordinates, displacement sequences, fitting parameters, etc.), encoding it into an extremely low-dimensional "latent representation" vector.

[0126] The essence of this process is to extract and retain the most informative key patterns from the data while discarding redundant details. This latent representation vector has an extremely small data size and a compression ratio of up to 150:1, thereby reducing transmission bandwidth requirements by more than 99% and enabling the system to operate stably even in low-bandwidth network environments. After the data is transmitted to the cloud, a dedicated "decoder" network can reconstruct the original feature data with high fidelity based on this latent representation, providing data for subsequent in-depth analysis and archiving, effectively ensuring information integrity. Compared to traditional compression algorithms (such as JPEG and ZIP), this method has higher compression efficiency and error resistance for structured numerical features, providing key technical support for achieving efficient edge-cloud collaboration.

[0127] In one feasible implementation, step S40 includes steps A11 to A13:

[0128] A11: Input the first feature data packet into the autoencoder network to obtain the first compression vector;

[0129] In one feasible implementation, step S40 includes steps A21 to A23:

[0130] It should be noted that the purpose of this sub-step is to compress the data packet size to 1 / 150 of the original size, thereby reducing 4G / 5G data charges.

[0131] The “first compressed vector” is a floating-point array with a dimension that is only 1 / 150 of the original features.

[0132] Understandably, the edge box concatenates displacement, tilt, and other numerical values ​​and inputs them into the pre-trained convolutional autoencoder, outputting a 128-dimensional first compressed vector to ensure that the decoding error is less than 1%.

[0133] A12: The first compressed vector is sent to the cloud platform via a wireless communication network, so that the cloud platform can reconstruct the first feature data packet through the decoder and perform monitoring and analysis;

[0134] It should be noted that the purpose of this step is to enable the cloud to obtain complete monitoring data in milliseconds.

[0135] "Wireless communication network" refers to the 4G / 5G MQTT protocol.

[0136] Understandably, the edge device publishes the first compressed vector to the topic / home / id via MQTT; after the cloud subscribes, it immediately uses a decoder to restore it into the first feature data packet and updates the digital twin model to complete the monitoring and analysis.

[0137] A13: Receive the warning level identifier returned by the cloud platform and write the warning level identifier to local storage.

[0138] It should be noted that the purpose of this step is to leave a record on site, which will facilitate the use of sound and light alarms or inspections.

[0139] The "warning level indicator" is a string of "yellow / orange / red" transmitted back from the cloud.

[0140] Understandably, after the cloud compares the threshold, it sends back the identifier. The edge box receives the identifier through the same MQTT link and writes the identifier and the current timestamp into the local SQLite log, which is then used by the HMI interface for instant pop-ups or SMS gateway calls.

[0141] It should be understood that traditional monitoring systems stop at data display and lack deep integration with the early warning process. This system constructs a multi-scale digital twin model, combines it with a spatiotemporal graph convolutional network for trend prediction, and establishes a three-level early warning mechanism to achieve comprehensive monitoring from real-time early warning to medium- and long-term forecasting.

[0142] To enable in-depth analysis and intuitive presentation of monitoring data, this system constructs a multi-scale digital twin model that is linked in real time with the physical entity. This model uses the building's BIM (Building Information Modeling) as its geometric and information foundation, seamlessly integrating real-time IoT monitoring data, historical data, and manual inspection information to form a dynamically updated virtual mirror.

[0143] The core of the model lies in its "multi-scale" expressive capability. At the macro scale (building as a whole), it dynamically visualizes overall tilt, settlement cloud maps, and vibration modes, providing managers with a comprehensive overview of the safety situation. At the meso scale (floors, components), it can precisely locate uneven settlement on a specific floor or tilt changes in a load-bearing wall. At the micro scale (local nodes), it supports high-magnification viewing and trend tracing of details such as the evolution of specific crack widths and the expansion of hollow areas.

[0144] Leveraging the powerful rendering and interactive capabilities of the Unity3D game engine, this model achieves deep integration and dynamic mapping between monitoring data and the 3D model. Users can click on any component in the model to query all current monitoring parameters, historical data curves, and health status assessment reports in real time. This not only greatly improves the interpretability of the data but also provides intuitive spatial positioning and decision-making context for the three-level early warning system, making intelligent operation and maintenance—"virtual-real linkage, controlling the real with the virtual"—a reality. It serves as the core platform for the entire system to achieve visualized and intelligent management.

[0145] Traditional early warning systems rely on static comparisons of current data with fixed thresholds, failing to predict risk evolution trends. Structural response data exhibits significant spatiotemporal correlations: spatially, parameters at different monitoring points influence each other; temporally, the current state is closely linked to historical states. To address this challenge, this system introduces a Spatiotemporal Graph Convolutional Network (ST-GCN) to deeply mine the spatiotemporal patterns in monitoring data and achieve forward-looking predictions. Its general model is as follows:

[0146]

[0147] in: Represents the predicted value; Represents historical data; Represents network parameters.

[0148] ST-GCN models building structures by abstracting them into graphs: nodes represent monitoring points and their multi-parameter data, while edges represent their physical connections. The model aggregates neighbor node information through spatial graph convolution to capture the transmission and spatial dependencies of internal forces; it also extracts the historical change patterns of each node through temporal convolution. This combination allows for the simultaneous learning of complex spatiotemporal features. Its core value lies in achieving a leap from "describing the present" to "predicting the future."

[0149] After inputting the fused historical time-series data stream from multiple parameters and monitoring points into the ST-GCN model, it can not only predict the future trend curves of key parameters (such as tilt angle and crack width) at multiple future time points, but also provide early warnings based on whether the predicted future trends will reach the danger threshold, providing a valuable time window for risk management. This is the core decision engine for the three-level intelligent early warning mechanism to achieve proactive prevention. For example, if the model predicts that, based on the current trend, the building tilt angle will exceed the danger threshold in 48 hours, it will trigger an early warning, allowing valuable response time for personnel evacuation and reinforcement maintenance.

[0150] This system innovatively designs a three-level early warning mechanism based on multi-parameter fusion decision-making. This mechanism not only considers the absolute threshold of a single parameter, but also focuses on the correlation and changing trends between parameters. Through the model of "real-time edge judgment + comprehensive cloud prediction", it achieves refined risk classification and accurate early warning response.

[0151] Furthermore, this solution also includes: receiving a second video frame sequence from the UAV platform and using the second video frame sequence as supplementary input to the multimodal data;

[0152] The second video frame sequence is processed using the same procedure as the multimodal data to obtain the second displacement sequence and the second tilt angle.

[0153] By using precise time synchronization and global optimized registration, the second displacement sequence, the second tilt angle, the first displacement sequence, and the first tilt angle are fused into the first feature data packet to obtain the second feature data packet;

[0154] The second feature data packet is sent to the cloud platform so that the cloud platform can perform building structure safety monitoring based on the second feature data packet.

[0155] It should be noted that the purpose of this sub-step is to address the blind spots of fixed camera views, provide overhead / side-view data, and expand the monitoring coverage.

[0156] "Second video frame sequence" refers to the aerial images transmitted back by the drone; "second displacement sequence" and "second tilt angle" are incremental data obtained using the same algorithm.

[0157] Understandably, the drone hovers and takes pictures along a preset route, then sends the JPEG stream back to the edge box via 5G. The box treats this as new "multimodal data," producing a second displacement and a second tilt. The two sets of data are then merged into the same first feature data packet using a weighted average before being uploaded, achieving complementarity between air and ground data.

[0158] This system innovatively designs a three-level early warning mechanism based on multi-parameter fusion decision-making. This mechanism not only considers the absolute threshold of a single parameter, but also focuses on the correlation and changing trends between parameters. Through the model of "real-time edge judgment + comprehensive cloud prediction", it achieves refined risk classification and accurate early warning response.

[0159]

[0160] The detailed design of this mechanism ensures that the system not only "sees" but also "understands, judges accurately, and reacts quickly," truly achieving a leap from passive monitoring to proactive early warning and intelligent prevention and control. The core innovations of the mechanism include:

[0161] From "single-point threshold" to "multi-parameter fusion": This approach changes the traditional simplistic judgment that relies solely on a single parameter exceeding a threshold. It introduces multi-parameter correlation analysis, which significantly reduces the false alarm rate and improves the scientific rigor and accuracy of early warnings.

[0162] From "static alarm" to "dynamic prediction": It deeply integrates time-series prediction models such as ST-GCN, not only focusing on the current state, but also predicting the future development trend of parameters, realizing "early warning" and gaining valuable time for intervention.

[0163] From "manual prediction" to "intelligent decision-making": The early warning logic is based on fuzzy reasoning or machine learning models, which incorporate expert knowledge and historical cases. It can automatically complete the judgment from data to risk level and provide preliminary handling suggestions, reducing the burden of manual analysis and improving response speed.

[0164] Closed-loop management: Each level of early warning corresponds to a clear response process and handling suggestions, forming an intelligent closed-loop management of "monitoring-early warning-handling-feedback", which ensures that risks are effectively controlled.

[0165] Furthermore, addressing the blind spots and insufficient accuracy inherent in single monitoring methods, this invention innovatively integrates three technology platforms: fixed ground monitoring stations, mobile UAV inspections, and precision lidar scanning, constructing an integrated air-ground collaborative monitoring network. Through intelligent task scheduling and the IEEE 1588 precision time synchronization protocol, the time consistency of multi-source heterogeneous data is ensured (synchronization deviation <5ms), resolving the pain points of difficult data stitching and inconsistent spatiotemporal benchmarks in traditional methods. An improved Go-ICP algorithm is employed to achieve high-precision registration of multi-period point clouds (accuracy up to 1.5mm, 50% improvement over traditional ICP), significantly enhancing the completeness and reliability of monitoring.

[0166] To address the challenges of high cost, system complexity, and data synchronization difficulties associated with discrete deployment of multiple sensors, this invention pioneers a multi-threaded parallel processing architecture on an edge computing platform. Employing a CPU+GPU heterogeneous computing model, it achieves simultaneous extraction of four types of parameters: displacement, tilt, cracks, and voids. By integrating improved phase-based optical flow, lightweight DeepCrack crack segmentation, and temperature gradient void identification adaptive algorithms, complex calculations are performed directly at the edge, resulting in a 4x increase in computational efficiency and a 40% reduction in power consumption. This provides a novel technological paradigm for real-time multi-parameter monitoring.

[0167] To address the challenges of large data volumes, heterogeneous formats, and high bandwidth requirements across multiple platforms, this invention proposes a multi-source data fusion technology based on an improved Go-ICP algorithm and extended Kalman filter (EKF) to achieve high-precision automatic registration and fusion of laser point clouds, visual images, and thermal infrared data. Simultaneously, a deep learning compression algorithm based on Autoencoder was developed, achieving a compression ratio of 150:1 and reducing bandwidth requirements by 99%, significantly enhancing the system's applicability in weak network environments.

[0168] Breaking through the limitations of traditional monitoring systems that prioritize data over decision-making, this system constructs a multi-scale dynamic digital twin deeply integrated with BIM models, enabling real-time driving and visual mapping between monitoring data and virtual models. It innovatively designs a multi-parameter fusion decision engine based on fuzzy logic and a three-level early warning mechanism, combined with a spatiotemporal graph convolutional network (ST-GCN) spatiotemporal prediction model, achieving a leap from real-time alarms to medium- and long-term safety predictions, providing comprehensive and intelligent decision support for building safety management.

[0169] This embodiment provides a visual monitoring method for building structure safety at the edge. It directly performs sub-pixel displacement calculation and 3D reconstruction-plane fitting on the monocular visible light video stream in parallel at the edge. The displacement sequence and tilt angle are generated, packaged, and uploaded synchronously within the same calculation cycle, eliminating the need for multi-sensor deployment and original video transmission. This reduces the amount of system hardware and the amount of data transmitted, thereby lowering the monitoring cost. Real-time calculation and instant transmission shorten the information link. After receiving the feature data packet, the platform can update the digital twin model and output the monitoring results. The overall response time is reduced from minutes in traditional solutions to seconds, achieving low-cost and low-latency building structure safety monitoring.

[0170] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S20 includes steps S201 to S204:

[0171] Step S201: Perform subpixel displacement calculation on the visible light video stream to obtain the first displacement sequence;

[0172] It should be noted that the purpose of this step is to measure the relative displacement of the building with an accuracy of "less than 1 pixel", so as to provide a quantitative basis for structural safety.

[0173] "Subpixel" is a virtual coordinate unit that subdivides a pixel by a factor of 100; "first displacement sequence" is the offset time sequence corresponding to each frame.

[0174] Understandably, by first reducing the image through pyramidal layering, then extracting phase consistency features, then estimating inter-frame optical flow, and finally multiplying by the on-site calibration scale, "0.3 pixels" can be converted into "0.5 millimeters" and written into the displacement sequence.

[0175] It should be understood that traditional monitoring methods require multiple dedicated sensors to measure different parameters, resulting in system complexity and high costs. This system adopts a multi-threaded processing architecture and a CPU+GPU heterogeneous computing mode to achieve simultaneous extraction of four types of parameters, thus solving the cost and synchronization problems caused by the separate deployment of multiple sensors.

[0176] Outdoor lighting variations and weather conditions significantly impact the stability of traditional visual algorithms. Phase-based optical flow algorithms utilize Fourier phase information and possess illumination invariance, but suffer from high computational complexity. This proposed algorithm maintains accuracy while meeting real-time performance requirements.

[0177]

[0178] in, SF represents the displacement change (mm); SF represents the scale factor (mm / pixel), obtained through camera calibration. Represents the coordinates of feature points in the reference frame; Represents the coordinates of the feature points in the current frame; N represents the number of valid feature points.

[0179] Building tilt is a crucial indicator of structural safety, but traditional tilt sensors require contact installation and are costly. This technical solution utilizes high-resolution cameras at fixed ground nodes or video sequences of building facades captured by drones during inspections as input for monocular 3D reconstruction. By extracting and matching natural feature points (such as window corners, brick joints, and decorative lines) from images acquired from different perspectives or time points, a 3D point cloud model of the building surface is reconstructed. Based on the point cloud data, the building's main plane is fitted, achieving non-contact, high-precision tilt measurement.

[0180]

[0181] Where: θ represents the tilt angle (°); These represent the point cloud normal vectors at different times.

[0182] The solution makes full use of low-cost vision sensors to achieve "non-contact, low-cost, and high-precision" three-dimensional geometric measurement, which is one of the key technologies for realizing "one-sensor-multiple-knowledge" (using a camera to simultaneously perceive multiple parameters).

[0183] Cracks are a direct manifestation of structural damage in buildings. Traditional edge detection methods are not robust to complex backgrounds, while deep learning methods have high computational complexity. This system uses a lightweight DeepCrack network to achieve pixel-level segmentation, enabling real-time crack identification and quantification analysis on edge devices.

[0184]

[0185] in: Represents the crack width (mm); Represents the pixel area of ​​the crack region; The pixel value representing the crack length; Represents the width scale factor (mm / pixel).

[0186] Hollow areas in exterior walls pose a significant safety hazard to buildings. Traditional tapping methods are inefficient and highly subjective. This algorithm, based on thermal infrared video stream temperature gradient analysis, identifies internal hollow areas by measuring surface temperature differences, achieving non-contact, non-destructive testing.

[0187]

[0188] in: Represents the temperature gradient; τ represents the temperature value; τ represents the adaptive threshold, which is dynamically adjusted according to the ambient temperature.

[0189] In one feasible implementation, step S201 includes steps A21 to A24:

[0190] A21: Perform pyramid layering processing on the visible light video stream to obtain a multi-level pyramid image;

[0191] It should be noted that the purpose of this step is to perform multi-scale processing on the visible light video stream in order to extract features at different resolutions, reduce computational complexity, and enhance the robustness of the features.

[0192] The video stream is decomposed into multi-level pyramid images. Pyramid layering is an image preprocessing technique that generates a series of image levels by progressively reducing the image resolution, thereby enabling multi-scale analysis.

[0193] A22: Phase consistency feature extraction is performed on the multi-level pyramid image to obtain the first phase feature map;

[0194] It should be noted that the goal of this step is to extract phase consistency features from the pyramid image that are sensitive to displacement calculations.

[0195] Phase consistency feature extraction is performed on the multi-level pyramid image to obtain the first phase feature map. Phase consistency feature extraction is a technique used to highlight edge and texture features in an image. It enhances the edge detection effect by analyzing the phase information of the image signal, providing a reliable feature basis for subsequent displacement calculation.

[0196] A23: Perform inter-frame optical flow estimation on the first phase feature map to obtain the first optical flow field;

[0197] It should be noted that the goal of this step is to calculate the optical flow field between frames based on the phase feature map, thereby estimating the motion vector of the pixel.

[0198] Inter-frame optical flow estimation is performed on the first phase feature map to obtain the first optical flow field.

[0199] Further, step A23 includes: performing a Fourier transform on the first phase feature map to obtain a first frequency domain map;

[0200] The phase difference is calculated on the first frequency domain map to obtain the first phase difference map;

[0201] Subpixel displacement mapping is performed on the first phase difference map to obtain the first optical flow field.

[0202] It should be noted that the Fourier transform is a mathematical tool used to analyze the frequency components of an image.

[0203] Phase difference calculation is used to determine the phase change of pixels in an image between different frames, which is very important for calculating pixel motion.

[0204] Subpixel displacement mapping is a technique that provides more accurate displacement estimates than pixels, resulting in a more accurate optical flow field.

[0205] A24: Scale transformation is performed on the first optical flow field to obtain the first displacement sequence.

[0206] It should be noted that the goal of this step is to convert the displacement information in the optical flow field into an actual physical displacement sequence.

[0207] The first optical flow field is scaled to obtain the first displacement sequence. The scale transformation is used to convert the displacement information in the optical flow field to the actual physical scale, thereby obtaining the real displacement sequence.

[0208] Step S202: Perform monocular 3D reconstruction and planar fitting on the visible light video stream to obtain the first tilt angle;

[0209] It should be noted that the purpose of this step is to restore the three-dimensional pose of the house using only "one camera" and to calculate the tilt angle relative to the vertical direction, replacing the traditional tilt meter.

[0210] "Monocular" refers to a single ordinary camera; "first point cloud data" is a collection of a large number of XYZ coordinate points on the surface of the building; "first tilt angle" is the angle between the principal plane and the vertical line.

[0211] Understandably, the computational load is reduced by extracting keyframes first, and then sparse point clouds are obtained by feature matching and motion recovery. After surface diffusion and densification, the largest plane is selected by random sampling consistency, its normal vector is estimated, and finally the angle between it and the preset reference normal vector is calculated to complete the angle output.

[0212] In one feasible implementation, step S202 includes steps A31 to A36:

[0213] A31: Extract keyframes from the visible light video stream to obtain the second video frame sequence;

[0214] It should be noted that the purpose of this sub-step is to select a small number of frames with "large and clear perspective changes" from continuous video to reduce the amount of subsequent 3D calculations.

[0215] The "second video frame sequence" is the set of keyframes that have been selected.

[0216] Understandably, the system calculates the motion amplitude between adjacent frames. When the displacement exceeds a preset threshold and the ambiguity is lower than a preset threshold, the frame is marked as a key frame and added to the sequence.

[0217] A32: Extract and match feature points from the second video frame sequence to obtain the first matching point pair;

[0218] It should be noted that the purpose of this sub-step is to find "unique" corner points on keyframes and establish cross-frame correspondences to provide "control points" for 3D positioning.

[0219] "First matching point pair" refers to the pixel coordinates of the same physical corner point in two photos.

[0220] Understandably, corner points are detected and descriptors are generated, and then cross-frame matching is completed through nearest neighbor search to form a list of matching point pairs.

[0221] It should be noted that the purpose of this step is to convert the spatial orientation into a readable angle.

[0222] The "preset reference normal vector" can be vertically upward [0,0,1].

[0223] Understandably, the angle value is obtained by using the formula for calculating the angle between vectors by dot product, and then written into "first tilt angle" for subsequent encapsulation.

[0224] A33: Perform motion recovery structure calculation on the first matched point pair to obtain the first sparse point cloud;

[0225] It should be noted that the purpose of this sub-step is to use the matching point pairs to infer the camera pose and 3D coordinates, thereby obtaining the initial sparse geometry.

[0226] The “first sparse point cloud” contains only a few thousand three-dimensional points.

[0227] Understandably, the camera parameters and spatial point coordinates are solved iteratively to output a sparse point cloud.

[0228] A34: Dense reconstruction is performed on the first sparse point cloud to obtain the first point cloud data;

[0229] It should be noted that the purpose of this sub-step is to expand sparse points into millions of dense points, thereby improving the reliability of plane fitting.

[0230] "First Diffuse Point Set", "First Depth Map", and "First Point Cloud Data" are respectively the intermediate results of interpolation and the final dense point set.

[0231] Understandably, a surface is first generated by interpolation around sparse points, then a depth map is generated by camera projection, and finally each pixel is back-projected to obtain a dense point cloud.

[0232] Further, step A34 includes: performing surface diffusion processing on the first sparse point cloud to obtain a first diffusion point set;

[0233] A depth map is generated from the first set of diffusion points to obtain the first depth map;

[0234] The first depth map is back-projected into the point cloud to obtain the first point cloud data.

[0235] It should be noted that the purpose of this sub-step is to expand sparse points into millions of dense points, thereby improving the reliability of plane fitting.

[0236] "First Diffuse Point Set", "First Depth Map", and "First Point Cloud Data" are respectively the intermediate results of interpolation and the final dense point set.

[0237] Understandably, a surface is first generated by interpolation around sparse points, then a depth map is generated by camera projection, and finally each pixel is back-projected to obtain a dense point cloud.

[0238] A35: Random sampling consistency screening is performed on the first point cloud data to obtain the first principal plane point set;

[0239] It should be noted that the purpose of this step is to eliminate outliers such as window sills and air conditioners, and only retain the "main wall" points.

[0240] The “first principal plane point set” is the set of points that is identified as the sub-points of the wall.

[0241] Understandably, RANSAC repeatedly and randomly selects three points to form a plane, counts the number of points that fit together, and selects the plane with the most fitting points as the master bedroom wall, while retaining the corresponding points.

[0242] A36: Estimate the normal vector of the first principal plane point set to obtain the first normal vector, and calculate the angle between it and the preset reference normal vector to obtain the first tilt angle.

[0243] It should be noted that the purpose of this step is to calculate the spatial orientation of the main plane.

[0244] The "first normal vector" is the unit vector perpendicular to the plane.

[0245] It is understandable that by calculating the covariance matrix of the point set on the principal plane, the eigenvector corresponding to the smallest eigenvalue is the normal vector, which is then output after normalization.

[0246] Furthermore, the first point cloud data and the first sparse point cloud are globally optimized and registered to obtain optimized point cloud data;

[0247] The first principal plane point set was recalculated based on the optimized point cloud data;

[0248] The normal vectors of the recalculated principal plane point set are estimated to obtain the corrected normal vectors.

[0249] The corrected first tilt angle is calculated based on the corrected normal vector and the preset reference normal vector.

[0250] Step S203: Perform crack segmentation on the visible light video stream to obtain the first crack width;

[0251] It should be noted that the purpose of this step is to simultaneously provide the width of wall cracks without adding hardware, thus achieving "one machine for multiple measurements".

[0252] "Crack segmentation" refers to using AI to label crack pixels as 1 and the rest as 0;

[0253] "First Crack Binary Image" is a black and white crack image; "Skeleton Extraction" transforms wide cracks into single-pixel centerlines; "First Crack Pixel Length" is the pixel count of the centerline; "Preset Pixel Scale" is the millimeter / pixel value calibrated on-site.

[0254] In one feasible implementation, step S203 includes steps A41 to A44:

[0255] A41: Perform crack segmentation on the visible light video stream to obtain the first crack binary image;

[0256] It should be noted that visible light video streams refer to sequences of images of a building captured by a regular camera. Crack segmentation is an image processing technique that separates cracked areas from the background in an image.

[0257] This process marks pixels in the cracked area as white and pixels in the non-cracked area as black, thus generating the first crack binary map, which provides a basis for subsequent crack analysis.

[0258] A42: Extract the skeleton from the binary image of the first crack to obtain the center line of the first crack;

[0259] It should be noted that the skeleton extraction technique simplifies the crack region to a centerline with a width of only one pixel. This step aims to enable more accurate measurement of the crack length and also provides a more intuitive representation for crack visualization.

[0260] A43: Perform pixel statistics on the center line of the first crack to obtain the pixel length of the first crack;

[0261] It should be noted that this step quantifies the length of the crack by calculating the number of pixels along the center line, providing the necessary data support for subsequent conversion to actual physical dimensions.

[0262] A44: Convert the first crack pixel length into the first crack width according to the preset pixel scale.

[0263] It should be noted that the preset pixel scale is a conversion factor based on camera calibration, used to convert the pixel length in the image into the physical length in the real world, such as millimeters or centimeters, thereby quantifying the crack width.

[0264] Step S204: Perform temperature gradient analysis on the thermal infrared video stream to obtain the first hollow mask.

[0265] It should be noted that thermal infrared video streams are sequences of images captured by thermal infrared cameras, which can display the temperature distribution on the surface of an object. Temperature gradient analysis is a technique that identifies areas of significant temperature change in the image by analyzing the rate of temperature change; these areas typically correspond to locations of voids.

[0266] Based on the above analysis, the first hollow mask can be obtained, which is a binary image of the same size as the thermal infrared image, where white pixels represent hollow areas and black pixels represent non-hollow areas.

[0267] In one feasible implementation, step S204 includes steps A51 to A53:

[0268] A51: Binarize the first gradient map to obtain the first binary gradient map;

[0269] It should be noted that this step converts the pixel values ​​in the gradient map into binary form by setting a threshold for the temperature gradient, thereby highlighting areas with significant temperature changes.

[0270] A52: Perform connected component analysis on the first binary gradient graph to obtain the first set of connected regions;

[0271] It should be noted that this step can identify all interconnected regions in the image, with each connected region representing a potential hollow area.

[0272] A53: Perform area filtering on the first connected region set, retaining regions with an area greater than a preset area threshold, to obtain the first hollow mask.

[0273] It should be noted that by setting an area threshold, larger areas that are more likely to be actual hollow areas can be screened out, thereby improving the accuracy of detection.

[0274] This embodiment provides a visual monitoring method for the safety edges of building structures. Through sub-pixel displacement calculation, monocular 3D reconstruction, crack segmentation, and temperature gradient analysis, displacement, tilt, crack, and void parameters are simultaneously extracted from visible light and thermal infrared video streams. These steps work together to achieve real-time multi-parameter monitoring of building structural safety, improving monitoring accuracy and reliability. Specifically, pyramid layering and phase consistency feature extraction enhance the robustness of displacement calculation, while inter-frame optical flow estimation and sub-pixel displacement mapping improve the accuracy of the displacement field. Monocular 3D reconstruction and plane fitting are used to accurately calculate the tilt angle of the building. Crack segmentation and skeleton extraction enable quantitative analysis of crack width, while temperature gradient analysis and connected component processing effectively identify void regions. The comprehensive application of these techniques ensures comprehensive, real-time, and accurate monitoring of the building structural safety status.

[0275] Based on the first and second embodiments of this application, the content that is the same as or similar to the above embodiments in the third embodiment of this application can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S40 also includes steps S401 to S404:

[0276] Step S401: Read the preset time window length and slice the first feature data packet according to the preset time window length to obtain the first data slice;

[0277] It should be noted that the purpose of this step is to organize and segment the feature data from a time perspective so that subsequent analysis can be carried out within a specific time range, providing a basis for trend prediction.

[0278] The preset time window length is the length of the time period set according to the monitoring requirements, which is used to segment the data.

[0279] The first characteristic data packet is a data packet containing building structural safety monitoring data, such as displacement, tilt, crack width, and hollow masking.

[0280] The first data slice is the data segment obtained by slicing the feature data packet according to the preset time window length.

[0281] It should be understood that the first feature data packet is sliced ​​according to a preset time window length to obtain the first data slice. For example, if the preset time window length is 10 minutes, then the data for every 10 minutes will be divided into an independent data slice. This helps to decompose a continuous data stream into multiple time segments, facilitating subsequent time series analysis and processing.

[0282] Step S402: Perform temporal encoding on the first data slice to obtain the first temporal feature;

[0283] It should be noted that the purpose of this step is to convert the data slices into feature vectors containing time series information in order to capture the temporal characteristics of the data.

[0284] The first data slice is the data segment obtained by slicing the feature data packet according to the preset time window length.

[0285] Timing coding is used to capture the temporal characteristics and sequential relationships of data.

[0286] The first temporal feature is the feature vector obtained after temporal coding, which contains time series information.

[0287] Temporal encoding is performed on the first data slice to obtain the first temporal feature. This step uses a specific encoding algorithm, such as a one-dimensional convolutional neural network, to transform the information in the data slice into a feature vector containing temporal relationships. Temporal encoding enhances the temporal attributes of the data, enabling subsequent spatiotemporal analysis to fully utilize information in the time dimension.

[0288] Step S403: Input the first temporal feature into the spatiotemporal graph convolutional network to obtain the first prediction parameters;

[0289] It should be noted that the purpose of this step is to use spatiotemporal graph convolutional networks to analyze temporal features and predict future trends in building structural safety parameters.

[0290] The first temporal feature is the feature vector obtained after temporal coding, which contains time series information.

[0291] Spatiotemporal graph convolutional networks are a type of neural network architecture that can simultaneously process time-series data and spatial relationships, making them suitable for capturing spatiotemporal correlations.

[0292] The first prediction parameter is the prediction result output by the spatiotemporal graph convolutional network, which includes the predicted values ​​of the building structure safety parameters at future time points.

[0293] The first temporal feature is input into the spatiotemporal graph convolutional network to obtain the first prediction parameters. Through its unique architecture, the spatiotemporal graph convolutional network can learn temporal and spatial patterns in data, thereby predicting the future structural safety of buildings. Specifically, the graph convolutional layers in the network capture spatial relationships, while the temporal convolutional layers capture the dynamic changes of the time series.

[0294] Step S404: Add the first prediction parameter to the first feature data packet, and send the first feature data packet containing the first prediction parameter to the cloud platform so that the cloud platform can predict the safety trend of the building structure based on the first prediction parameter.

[0295] It should be noted that the purpose of this step is to feed the predicted parameters back to the cloud platform to achieve prediction and early warning of building structural safety trends.

[0296] The first prediction parameter is the prediction result output by the spatiotemporal graph convolutional network, which includes the predicted values ​​of the building structure safety parameters at future time points.

[0297] A cloud platform is a centralized computing and storage platform used to process and analyze data from edge devices.

[0298] Building structural safety trend prediction involves using predictive parameters to forecast the future safety status of a building structure, so as to take maintenance measures in advance.

[0299] The first prediction parameter is added to the first feature data packet, and the first feature data packet containing the first prediction parameter is sent to the cloud platform. After receiving the updated feature data packet, the cloud platform uses the prediction parameter in it to predict the safety trend of the building structure. This step combines monitoring data with the prediction model, enabling the system not only to monitor the safety status of the building structure in real time, but also to predict future trends, providing a scientific basis for preventive maintenance.

[0300] This embodiment provides a visual monitoring method for building structure safety edges. By reading a preset time window and slicing the feature data packet, detailed analysis of the data can be performed within a specific time range. Temporal coding further enhances the temporal attributes of the data, enabling the spatiotemporal graph convolutional network to effectively learn the spatiotemporal correlations in the data, thereby improving the accuracy and reliability of predictions. Finally, the prediction parameters are fed back to the cloud platform, achieving seamless integration of monitoring and prediction, and providing strong technical support for the safety management of building structures.

[0301] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the visual monitoring method for building structure safety edges of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0302] This application also provides a visual monitoring device for the safety edges of building structures; please refer to [reference needed]. Figure 5 The building structure safety edge visual monitoring device includes:

[0303] The acquisition module 10 is used to acquire multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data.

[0304] The parameter extraction module 20 is used to perform synchronous parameter extraction processing on multimodal data to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask.

[0305] The data encapsulation module 30 is used to encapsulate the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask to obtain a first feature data packet;

[0306] The monitoring module 40 is used to perform structured numerical compression on the first feature data packet to obtain a compressed vector, and send the compressed vector to the cloud platform so that the cloud platform can update the digital twin model in real time based on the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring.

[0307] The building structure safety edge visual monitoring device provided in this application, employing the building structure safety edge visual monitoring method in the above embodiments, can solve the technical problem of how to perform multi-parameter synchronous monitoring of building structures. Compared with the prior art, the beneficial effects of the building structure safety edge visual monitoring device provided in this application are the same as those of the building structure safety edge visual monitoring method provided in the above embodiments, and other technical features in the building structure safety edge visual monitoring device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0308] In one embodiment, the parameter extraction module 20 is further configured to perform sub-pixel displacement calculation on the visible light video stream to obtain a first displacement sequence;

[0309] The first tilt angle is obtained by performing monocular 3D reconstruction and plane fitting on the visible light video stream.

[0310] The visible light video stream is segmented into cracks to obtain the first crack width;

[0311] Temperature gradient analysis was performed on the thermal infrared video stream to obtain the first hollow mask.

[0312] In one embodiment, the parameter extraction module 20 is also used to perform pyramid layering processing on the visible light video stream to obtain a multi-level pyramid image;

[0313] Phase consistency features are extracted from the multi-level pyramid image to obtain the first phase feature map;

[0314] Inter-frame optical flow estimation is performed on the first phase feature map to obtain the first optical flow field;

[0315] The first optical flow field is scaled to obtain the first displacement sequence.

[0316] In one embodiment, the parameter extraction module 20 is further configured to perform a Fourier transform on the first phase feature map to obtain a first frequency domain map;

[0317] The phase difference is calculated on the first frequency domain map to obtain the first phase difference map;

[0318] Subpixel displacement mapping is performed on the first phase difference map to obtain the first optical flow field.

[0319] In one embodiment, the parameter extraction module 20 is further configured to extract keyframes from the visible light video stream to obtain a second video frame sequence;

[0320] Feature points are extracted and matched from the second video frame sequence to obtain the first matching point pair;

[0321] Motion recovery structure calculation is performed on the first matched point pair to obtain the first sparse point cloud;

[0322] The first sparse point cloud is reconstructed by densification to obtain the first point cloud data.

[0323] Random sampling consistency screening is performed on the first point cloud data to obtain the first principal plane point set;

[0324] The normal vector of the first principal plane point set is estimated to obtain the first normal vector, and the angle between the normal vector and the preset reference normal vector is calculated to obtain the first tilt angle.

[0325] In one embodiment, the parameter extraction module 20 is further configured to perform surface diffusion processing on the first sparse point cloud to obtain a first diffusion point set;

[0326] A depth map is generated from the first set of diffusion points to obtain the first depth map;

[0327] The first depth map is back-projected into the point cloud to obtain the first point cloud data.

[0328] In one embodiment, the parameter extraction module 20 is further used to perform global optimization registration of the first point cloud data and the first sparse point cloud to obtain optimized point cloud data;

[0329] The first principal plane point set was recalculated based on the optimized point cloud data;

[0330] The normal vectors of the recalculated principal plane point set are estimated to obtain the corrected normal vectors.

[0331] The corrected first tilt angle is calculated based on the corrected normal vector and the preset reference normal vector.

[0332] In one embodiment, the parameter extraction module 20 is further used to perform crack segmentation on the visible light video stream to obtain a first crack binary image;

[0333] The skeleton of the first crack is extracted from the binary image of the first crack to obtain the center line of the first crack;

[0334] The pixel length of the first crack is obtained by performing pixel statistics on the center line of the first crack;

[0335] The pixel length of the first crack is converted into the width of the first crack according to the preset pixel scale.

[0336] In one embodiment, the parameter extraction module 20 is further used to calculate the temperature gradient of the thermal infrared video stream to obtain a first gradient map;

[0337] The first gradient map is binarized to obtain the first binary gradient map.

[0338] Perform connected component analysis on the first binary gradient graph to obtain the first set of connected regions;

[0339] The area of ​​the first connected region set is filtered, and regions with an area greater than a preset area threshold are retained to obtain the first hollow mask.

[0340] In one embodiment, the monitoring module 40 is further configured to input the first feature data packet into the autoencoder network to obtain the first compression vector;

[0341] The first compressed vector is sent to the cloud platform via a wireless communication network, so that the cloud platform can reconstruct the first feature data packet through a decoder and perform monitoring and analysis.

[0342] Receive the warning level identifier returned by the cloud platform and write the warning level identifier to local storage.

[0343] In one embodiment, the monitoring module 40 is further configured to receive a second video frame sequence from the drone platform and use the second video frame sequence as supplementary input to the multimodal data;

[0344] The second video frame sequence is processed using the same procedure as the multimodal data to obtain the second displacement sequence and the second tilt angle.

[0345] By using precise time synchronization and global optimized registration, the second displacement sequence, the second tilt angle, the first displacement sequence, and the first tilt angle are fused into the first feature data packet to obtain the second feature data packet;

[0346] The second feature data packet is sent to the cloud platform so that the cloud platform can perform building structure safety monitoring based on the second feature data packet.

[0347] In one embodiment, the monitoring module 40 is further configured to read a preset time window length and slice the first feature data packet according to the preset time window length to obtain a first data slice;

[0348] The first data slice is temporally encoded to obtain the first temporal feature;

[0349] The first temporal feature is input into the spatiotemporal graph convolutional network to obtain the first prediction parameters;

[0350] The first prediction parameter is added to the first feature data packet, and the first feature data packet containing the first prediction parameter is sent to the cloud platform so that the cloud platform can predict the safety trend of the building structure based on the first prediction parameter.

[0351] This application provides a visual monitoring device for the safety edge of a building structure. The visual monitoring device for the safety edge of a building structure includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the visual monitoring method for the safety edge of a building structure as described in Embodiment 1 above.

[0352] The following is for reference. Figure 6This document illustrates a structural schematic diagram suitable for implementing a building structure safety edge visual monitoring device according to embodiments of this application. The building structure safety edge visual monitoring device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The illustrated building structure safety edge visual monitoring device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0353] like Figure 6 As shown, the building structure safety edge visual monitoring device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the building structure safety edge visual monitoring device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the building structure safety edge visual monitoring equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a building structure safety edge visual monitoring equipment with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0354] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is processed by processing device 1001, the functions defined in the methods of the embodiments disclosed in this application are performed.

[0355] The building structure safety edge visual monitoring device provided in this application, employing the building structure safety edge visual monitoring method in the above embodiments, can solve the technical problem of how to perform multi-parameter synchronous monitoring of building structures. Compared with the prior art, the beneficial effects of the building structure safety edge visual monitoring device provided in this application are the same as those of the building structure safety edge visual monitoring method provided in the above embodiments, and other technical features in this building structure safety edge visual monitoring device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0356] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0357] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0358] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to perform the building structure safety edge visual monitoring method in the above embodiments.

[0359] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be instructed to be used by or in connection with a system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0360] The aforementioned computer-readable storage medium may be included in the building structure safety edge visual monitoring device; or it may exist independently and not be assembled into the building structure safety edge visual monitoring device.

[0361] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the building structure safety edge visual monitoring device, the device performs the following actions: acquires multimodal data, including visible light video streams, thermal infrared video streams, and point cloud data; performs parameter synchronous extraction processing on the multimodal data to obtain a first displacement sequence, a first tilt angle, a first crack width, and a first void mask; encapsulates the first displacement sequence, the first tilt angle, the first crack width, and the first void mask to obtain a first feature data packet; and performs structured numerical compression on the first feature data packet to obtain a compression vector, and sends the compression vector to a cloud platform so that the cloud platform updates the digital twin model in real time based on the first feature data packet and triggers an early warning mechanism, thereby completing the building structure safety monitoring.

[0362] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0363] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually proceed substantially in parallel, and they may sometimes proceed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0364] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0365] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the above-described visual monitoring method for building structure safety edges, and can solve the technical problem of how to perform multi-parameter synchronous monitoring of building structures. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the visual monitoring method for building structure safety edges provided in the above embodiments, and will not be repeated here.

[0366] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described visual monitoring method for building structure safety edges.

[0367] The computer program product provided in this application can solve the technical problem of how to perform multi-parameter synchronous monitoring of building structures. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the building structure safety edge visual monitoring method provided in the above embodiments, and will not be repeated here.

[0368] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method of visual monitoring of the safety edge of a building structure, characterized in that, The method includes: Collect multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data; The multimodal data is subjected to parameter synchronous extraction processing to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask; The first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask are encapsulated to obtain a first feature data packet; The first feature data packet is subjected to structured numerical compression to obtain a compressed vector, and the compressed vector is sent to the cloud platform so that the cloud platform can update the digital twin model in real time according to the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring. The step of performing parameter synchronous extraction processing on the multimodal data to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask includes: Subpixel displacement calculations are performed on the visible light video stream to obtain a first displacement sequence; The visible light video stream is subjected to monocular 3D reconstruction and planar fitting to obtain the first tilt angle; The visible light video stream is segmented to obtain a first crack width; Temperature gradient analysis was performed on the thermal infrared video stream to obtain the first hollow mask; The step of performing sub-pixel displacement calculation on the multimodal data to obtain the first displacement sequence includes: The visible light video stream is subjected to pyramid layering processing to obtain a multi-level pyramid image; Phase consistency features are extracted from the multi-level pyramid image to obtain a first phase feature map; Inter-frame optical flow estimation is performed on the first phase feature map to obtain the first optical flow field; The first optical flow field is scaled to obtain the first displacement sequence; The step of performing inter-frame optical flow estimation on the first phase feature map to obtain the first optical flow field includes: Perform a Fourier transform on the first phase feature map to obtain the first frequency domain map; The phase difference is calculated on the first frequency domain map to obtain the first phase difference map; Subpixel displacement mapping is performed on the first phase difference map to obtain the first optical flow field; The step of performing structured numerical compression on the first feature data packet to obtain a compressed vector, and sending the compressed vector to the cloud platform, includes: The first feature data packet is input into the autoencoder network to obtain the first compression vector; The first compressed vector is sent to the cloud platform via a wireless communication network, so that the cloud platform can reconstruct the first feature data packet through a decoder and perform monitoring and analysis. Receive the warning level identifier returned by the cloud platform and write the warning level identifier to local storage; The method further includes: Receive a second video frame sequence from the drone platform and use the second video frame sequence as supplementary input to the multimodal data; The second video frame sequence is processed using the same procedure as the multimodal data to obtain the second displacement sequence and the second tilt angle. By using precise time synchronization and global optimized registration, the second displacement sequence, the second tilt angle, the first displacement sequence, and the first tilt angle are fused into the first feature data packet to obtain the second feature data packet. The second feature data packet is sent to the cloud platform so that the cloud platform can perform building structure safety monitoring based on the second feature data packet.

2. The method according to claim 1, characterized in that, The step of performing monocular 3D reconstruction and planar fitting on the visible light video stream to obtain the first tilt angle includes: Keyframes are extracted from the visible light video stream to obtain a second video frame sequence; Feature point extraction and matching are performed on the second video frame sequence to obtain the first matching point pair; Motion recovery structure calculation is performed on the first matched point pair to obtain the first sparse point cloud; The first sparse point cloud is reconstructed by densification to obtain the first point cloud data; Random sampling consistency screening is performed on the first point cloud data to obtain the first principal plane point set; The normal vector of the first principal plane point set is estimated to obtain the first normal vector, and the angle between the normal vector and the preset reference normal vector is calculated to obtain the first tilt angle.

3. The method according to claim 2, characterized in that, The step of performing density reconstruction on the first sparse point cloud to obtain the first point cloud data includes: The first sparse point cloud is subjected to surface diffusion processing to obtain the first diffusion point set; A depth map is generated from the first set of diffusion points to obtain the first depth map; Point cloud back projection is performed on the first depth map to obtain the first point cloud data.

4. The method according to claim 2, characterized in that, After the step of performing monocular 3D reconstruction and planar fitting on the visible light video stream to obtain the first tilt angle, the method further includes: The first point cloud data and the first sparse point cloud are globally optimized and registered to obtain optimized point cloud data. The first principal plane point set is recalculated based on the optimized point cloud data; The normal vectors of the recalculated principal plane point set are estimated to obtain the corrected normal vectors. The corrected first tilt angle is calculated based on the corrected normal vector and the preset reference normal vector.

5. The method according to claim 1, characterized in that, The step of performing intelligent crack identification on the multimodal data to obtain the first crack width includes: The visible light video stream is segmented to obtain a first binary image of the crack; The skeleton of the first crack is extracted from the binary image of the first crack to obtain the center line of the first crack. The pixel length of the first crack is obtained by performing pixel statistics on the center line of the first crack. The pixel length of the first crack is converted into the width of the first crack according to a preset pixel scale.

6. The method according to claim 1, characterized in that, The step of performing void detection processing on the multimodal data to obtain a first void mask includes: Temperature gradient calculation is performed on the thermal infrared video stream to obtain a first gradient map; The first gradient map is binarized to obtain a first binary gradient map. Perform connected component analysis on the first binary gradient graph to obtain the first set of connected regions; The area of ​​the first connected region set is filtered, and regions with an area greater than a preset area threshold are retained to obtain the first hollow mask.

7. The method according to claim 1, characterized in that, The method further includes: Read the preset time window length and slice the first feature data packet according to the preset time window length to obtain the first data slice; The first data slice is temporally encoded to obtain the first temporal feature; The first temporal feature is input into the spatiotemporal graph convolutional network to obtain the first prediction parameters; The first prediction parameter is added to the first feature data packet, and the first feature data packet containing the first prediction parameter is sent to the cloud platform so that the cloud platform can predict the building structure safety trend based on the first prediction parameter.

8. A visual monitoring device for the safety edge of a building structure, characterized in that, The device includes: The acquisition module is used to acquire multimodal data, including visible light video stream, thermal infrared video stream, and point cloud data. The parameter extraction module is used to perform synchronous parameter extraction processing on the multimodal data to obtain the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask. The parameter extraction module is also used to perform sub-pixel displacement calculation on the visible light video stream to obtain a first displacement sequence; The visible light video stream is subjected to monocular 3D reconstruction and planar fitting to obtain the first tilt angle; The visible light video stream is segmented to obtain a first crack width; Temperature gradient analysis was performed on the thermal infrared video stream to obtain the first hollow mask; The parameter extraction module is also used to perform pyramid layering processing on the visible light video stream to obtain a multi-level pyramid image; Phase consistency features are extracted from the multi-level pyramid image to obtain a first phase feature map; Inter-frame optical flow estimation is performed on the first phase feature map to obtain the first optical flow field; The first optical flow field is scaled to obtain the first displacement sequence; The parameter extraction module is also used to perform a Fourier transform on the first phase feature map to obtain a first frequency domain map; The phase difference is calculated on the first frequency domain map to obtain the first phase difference map; Subpixel displacement mapping is performed on the first phase difference map to obtain the first optical flow field; The data encapsulation module is used to encapsulate the first displacement sequence, the first tilt angle, the first crack width, and the first hollow mask to obtain a first feature data packet; The monitoring module is used to perform structured numerical compression on the first feature data packet to obtain a compressed vector, and send the compressed vector to the cloud platform so that the cloud platform can update the digital twin model in real time according to the first feature data packet and trigger an early warning mechanism to complete the building structure safety monitoring. The monitoring module is used to input the first feature data packet into the autoencoder network to obtain the first compression vector; The first compressed vector is sent to the cloud platform via a wireless communication network, so that the cloud platform can reconstruct the first feature data packet through a decoder and perform monitoring and analysis. Receive the warning level identifier returned by the cloud platform and write the warning level identifier to local storage; The monitoring module is used to receive a second video frame sequence from the UAV platform and use the second video frame sequence as supplementary input for multimodal data; The second video frame sequence is processed using the same procedure as the multimodal data to obtain the second displacement sequence and the second tilt angle. By using precise time synchronization and global optimized registration, the second displacement sequence, the second tilt angle, the first displacement sequence, and the first tilt angle are fused into the first feature data packet to obtain the second feature data packet. The second feature data packet is sent to the cloud platform so that the cloud platform can perform building structure safety monitoring based on the second feature data packet.

9. A visual monitoring device for the safety edge of a building structure, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual monitoring method for building structural safety edges as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the building structure safety edge visual monitoring method as described in any one of claims 1 to 7.

Images