A machine learning based building crack monitoring method and system

Abstract

This invention discloses a machine learning-based method and system for monitoring building cracks, belonging to the fields of building structural health monitoring and computer vision technology. The method acquires visible light and infrared thermal images of the building to be monitored during historical and current acquisition periods. It registers the multimodal images to obtain crack masks and temperature anomaly region masks. Based on the historical crack end locations, it determines the crack propagation prediction region and filters out thermal anomaly regions to be verified within this prediction region and its vicinity. The thermal anomaly regions to be verified are associated and stored with the corresponding crack object, crack end, prediction region, generation time, and verification status. In subsequent acquisition periods, time-delay consistency verification is performed by adding new crack regions. By utilizing the spatial and temporal verification relationship between infrared thermal anomalies and crack geometric propagation, false alarms caused by environmental thermal interference are reduced, improving the engineering reliability of crack activity assessment.

Description

Technical Field

[0001] This application belongs to the field of building structural health monitoring and computer vision technology, specifically relating to a method and system for monitoring building cracks based on machine learning. Background Technology

[0002] Building cracks are an important indicator for evaluating the safety and durability of building structures. With the development of image acquisition equipment and computer vision technology, crack recognition methods based on visible light images have been gradually applied to building facades, concrete structures, masonry structures, and other scenarios. These methods typically acquire images of the building surface using a camera, and then use image processing algorithms or deep learning segmentation models to identify crack areas, thereby obtaining geometric information such as crack length, width, area, and end location.

[0003] Existing crack monitoring methods primarily rely on visible light images to identify crack regions and further extract geometric information such as crack length, width, area, and end location. While this method can reflect the apparent morphology of a crack at a single acquisition moment, for long-term monitoring scenarios, it is difficult to reliably determine whether a crack is in a state of continuous expansion based solely on the geometric differences of cracks in a single frame or adjacent images.

[0004] Infrared thermal imaging images are also available to obtain information on the temperature distribution of building surfaces. However, temperature anomalies on building facades are easily affected by factors such as the angle of sunlight, shadows, wind speed, surface moisture content, material reflection, stains, hollow areas, or water seepage. Infrared thermal anomalies do not necessarily correspond to crack propagation. If temperature anomaly areas are directly used as the basis for judging crack activity or propagation, it is easy to misjudge environmental thermal interference as a risk of crack development.

[0005] Furthermore, in long-term monitoring scenarios at fixed locations, there may be pixel-level deviations between historical and current images caused by micro-vibrations of the monitoring support, thermal expansion and contraction, slight lens offset, or image registration errors. Directly comparing historical crack masks with current crack masks may misjudge registration deviations, crack boundary jitter, or segmentation noise as newly added crack areas, thereby affecting crack activity scores or early warning results.

[0006] Therefore, existing crack monitoring methods lack a processing mechanism that can combine image registration quality, new crack area determination, infrared thermal anomaly verification, and crack propagation morphology constraints. This makes the use of infrared thermal anomalies in crack activity assessment susceptible to interference from environmental factors, and the stability and engineering reliability of crack propagation risk assessment are insufficient.

[0007] To address the aforementioned issues, this application provides a machine learning-based method and system for monitoring building cracks. It associates areas of abnormal infrared temperature with crack propagation direction and subsequent newly added crack areas to reduce the interference of environmental factors on monitoring results. Furthermore, it combines crack propagation patterns to differentially fuse multimodal features, thereby improving the reliability of crack activity status assessment. Summary of the Invention

[0008] The purpose of this invention is to provide a building crack monitoring method and system based on machine learning, in order to solve the problems of existing crack monitoring methods, such as infrared thermal anomalies being easily affected by environmental interference, single-frame images being difficult to determine the crack activity state, registration errors easily leading to misjudgment of newly added cracks, and multimodal fusion lacking crack propagation morphology constraints.

[0009] Infrared thermal anomaly regions are used as candidate auxiliary signals in crack activity assessment. It should be noted that this application does not presuppose that infrared thermal anomalies necessarily result from crack propagation, nor does it consider temperature anomaly regions in a single acquisition cycle as the sole causal basis for crack propagation. In this application, infrared thermal anomaly regions are only used as candidate auxiliary signals, and their risk contribution is constrained by the spatial constraints of the crack propagation prediction region and determined after spatiotemporal consistency verification through newly added crack regions in subsequent acquisition cycles. Candidate auxiliary signals are constrained by the spatial constraints of the crack propagation prediction region and are verified by time-delay verification through newly added crack regions in subsequent acquisition cycles. When a temperature anomaly region satisfies the spatial correlation condition with the crack propagation prediction region and satisfies the positional matching relationship with newly added crack regions within the effective verification cycle, it is determined to meet the spatiotemporal consistency verification condition and is used as a positive correction factor for the risk score of the corresponding crack object. When a temperature anomaly region is not verified by newly added crack regions within the effective verification cycle, its contribution to the crack activity score is reduced or it is marked as a candidate region for environmental thermal interference.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] In a first aspect, the present invention provides a machine learning-based method for monitoring building cracks, comprising:

[0012] S1. Collect multimodal image data of the building to be monitored, which correspond to the historical collection period and the current collection period in the time dimension. The multimodal image data includes visible light images and infrared thermal imaging images.

[0013] S2. Perform image registration on the multimodal image data to ensure that the visible light image and infrared thermal imaging image of the historical acquisition period and the current acquisition period are in a unified spatial coordinate relationship, and obtain the registration reprojection error of the corresponding acquisition period.

[0014] S3. Obtain historical crack masks and current crack masks based on visible light images from historical acquisition cycles and current acquisition cycles, respectively. Obtain temperature anomaly area masks based on infrared thermal imaging images from historical acquisition cycles and current acquisition cycles.

[0015] S4. Determine crack change information based on historical crack masks and current crack masks. The crack change information includes newly added crack regions, as well as at least one of crack end changes, crack width changes, crack length changes, and crack branch changes.

[0016] S5. Using the crack end positions from one or more acquisition cycles that have been processed before the current acquisition cycle as historical end positions, determine the crack propagation prediction region based on the historical end positions; when the number of historical end positions is insufficient to form an end displacement sequence, generate an initial crack propagation prediction region based on the tangent direction of the crack skeleton in the vicinity of the crack end; determine the thermal anomaly region to be verified related to the crack propagation prediction position based on the spatial coverage relationship or spatial proximity combination relationship between the temperature anomaly region mask and the crack propagation prediction region, and in combination with the comparison results of the temperature anomaly statistics and the background statistical benchmark.

[0017] S6. Associate and store the thermal anomaly region to be verified with the corresponding crack object, the corresponding crack end, the crack propagation prediction region, the region location, the generation time, and the verification status. In subsequent acquisition cycles, determine the time-delay consistency result based on the positional matching relationship between the newly added crack region and the thermal anomaly region to be verified and the crack propagation prediction region.

[0018] S7. Input at least one of the following into a machine learning model: crack geometry temporal difference features, temperature temporal difference features, texture temporal difference features, state of the thermal anomaly region to be verified, time delay consistency results, and crack propagation mode; and output a crack activity score or crack development status through the machine learning model.

[0019] S8. Output graded early warning results based on the crack activity score or crack development status.

[0020] Furthermore, image registration includes cross-modal registration and temporal registration, and the corresponding acquisition period is determined to be suitable for thermal anomaly verification based on the registration reprojection error.

[0021] Furthermore, the newly added crack region is determined based on the current crack mask and the historical crack mask after dilation processing. The dilation processing is used to compensate for image registration errors and crack mask boundary jitter.

[0022] Furthermore, the crack propagation prediction region is determined based on the positional changes of the crack tip during historical acquisition cycles; when the number of historical acquisition cycles is insufficient, an initial crack propagation prediction region is generated based on the tangential direction of the crack skeleton in the vicinity of the crack tip.

[0023] Furthermore, the thermal anomaly region to be verified is determined based on the spatial correlation between the temperature anomaly region and the crack propagation prediction region, as well as the statistical conditions of the temperature anomaly; the time-delay consistency result is determined based on the positional matching relationship between the newly added crack region and the thermal anomaly region to be verified in subsequent acquisition cycles.

[0024] Furthermore, the crack propagation pattern is determined based on the crack length change rate, maximum width change, end displacement, and branch number change. This crack propagation pattern is used to adjust the basic activity score or as an input feature for a machine learning model.

[0025] Secondly, the present invention provides a machine learning-based building crack monitoring system for executing the aforementioned machine learning-based building crack monitoring method. The system includes:

[0026] The data acquisition layer is used to acquire visible light images and infrared thermal images of the building to be monitored during historical and current acquisition periods.

[0027] The edge processing layer, which is communicatively connected to the data acquisition layer, is used to perform image registration on multimodal image data, generate historical crack masks and current crack masks, generate temperature anomaly area masks, and extract newly added crack areas and crack change information.

[0028] The cloud-based analysis layer, which is connected in communication with the edge processing layer, is used to determine the crack propagation prediction area, determine and store the thermal anomaly area to be verified, determine the positional matching relationship between the newly added crack area and the thermal anomaly area to be verified, determine the crack propagation mode, and calculate or correct the crack activity score.

[0029] The results output layer is connected to the cloud analysis layer and is used to output the crack development status or graded early warning results based on the crack activity score, time lag consistency results and risk triggering type.

[0030] The cloud-based analysis layer includes an extended prediction unit, a region-to-be-verified management unit, a time-delay consistency verification unit, an extended pattern classification unit, and an activity score correction unit. The extended prediction unit determines the predicted crack extension region; the region-to-be-verified management unit determines and stores the thermal anomaly regions to be verified; the time-delay consistency verification unit determines the positional matching relationship between newly added crack regions and the thermal anomaly regions to be verified; the extended pattern classification unit determines the crack extension pattern; and the activity score correction unit calculates or corrects the crack activity score.

[0031] In summary, this application includes at least one of the following beneficial technical effects:

[0032] 1. Based on historical crack masks, current crack masks, and temperature anomaly region masks, this application first determines the crack propagation prediction region according to the crack end positions in the completed acquisition cycle. Then, combining the spatial coverage relationship or spatial proximity combination relationship between the temperature anomaly region and the crack propagation prediction region, as well as the comparison results of the temperature anomaly statistics and the background statistical benchmark, the thermal anomaly regions to be verified are screened. The above processing ensures that the infrared thermal anomaly region does not directly participate in risk assessment based on independent detection results, but is jointly constrained by the crack geometric evolution direction, prediction range, and local background temperature differences. This reduces the influence of non-crack propagation factors such as sunlight, shadows, material reflection, local stains, or surface wetness on crack activity assessment.

[0033] 2. This application associates and stores the thermal anomaly region to be verified with the corresponding crack object, the corresponding crack tip, the crack propagation prediction region, the generation time, and the verification status. Within the effective verification period, it uses the positional matching relationship between newly added crack regions in subsequent acquisition periods and the thermal anomaly region to be verified and the crack propagation prediction region to determine the time-delay consistency result. Through this associated storage and subsequent verification mechanism, temperature anomalies appearing in the current period will not be directly identified as crack propagation risks. Only when newly added crack regions in subsequent periods meet the prediction region constraints and spatial proximity conditions will they be used as positive correction factors for the risk score of the corresponding crack object. Thermal anomaly regions that fail verification will have their score contribution reduced or be marked as candidate regions for environmental thermal interference, thereby reducing false alarms caused by single-period thermal anomalies.

[0034] 3. When determining newly added crack regions, this application compares the current crack mask with the historical crack mask after dilation processing, and correlates the dilation radius with the current period's registration-reprojection error, the historical registration-reprojection error statistics, and the minimum dilation radius. When the registration-reprojection error exceeds the historical statistical confidence interval, the dilation radius is increased and the contribution weight of newly added crack regions in the corresponding acquisition period to the crack activity score is reduced. Therefore, the extraction of newly added crack regions no longer relies solely on the direct difference between the historical mask and the current mask, but can compensate for pixel-level deviations caused by bracket micro-vibration, slight lens shift, thermal expansion and contraction, or residual registration errors, reducing the possibility of crack boundary jitter being misjudged as newly added cracks.

[0035] 4. This application uses at least one of the following as inputs for crack activity scoring or crack development status judgment: time-delay consistency results, temporal differences in crack geometry, temporal differences in temperature, temporal differences in texture, the state of the thermal anomaly region to be verified, and crack propagation mode. The fusion weights of geometric, temperature, and texture features are adjusted according to different crack propagation modes such as end-extending, width-opening, and bifurcation-complex types. This process avoids evaluating cracks with fixed weights for different propagation morphologies, enabling the crack activity score to match the actual crack evolution, and improving the correspondence between the graded early warning results and the temporal changes in crack status. Attached Figure Description

[0036] Figure 1 This is an overall architecture diagram of the machine learning-based building crack monitoring system of this invention.

[0037] Figure 2 This is an overall flowchart of the machine learning-based building crack monitoring method of the present invention.

[0038] Figure 3 This is a flowchart of the crack change information and temporal difference feature extraction process of the present invention.

[0039] Figure 4 This is a flowchart of the process for determining the thermal anomaly region to be verified and verifying the consistency of time delay in this invention;

[0040] Figure 5 This is a diagram showing the crack risk level and the state transition of the thermal anomaly region to be verified in this invention. Detailed Implementation

[0041] The following is in conjunction with the appendix Figures 1 to 5 The technical solution of the present invention will be described below. The following embodiments are used to explain the technical concept and implementation process of the present invention, and do not limit the scope of protection of the present invention.

[0042] The present invention provides a machine learning-based method and system for monitoring building cracks, including data acquisition, image registration, crack detection and segmentation, extraction of temperature anomaly areas, crack propagation prediction, management of thermal anomaly areas to be verified, time-delay consistency verification, machine learning scoring, and early warning output.

[0043] This method determines the crack propagation prediction region based on the crack tip location in historical acquisition cycles, and then filters out the thermal anomaly regions to be verified within the crack propagation prediction region and its vicinity. In subsequent acquisition cycles, the newly added crack regions are used to perform location matching verification of the thermal anomaly regions to be verified, obtaining time-delay consistency results. The machine learning model outputs a crack activity score or crack development status based on at least one of the following: temporal differences in crack geometry, temporal differences in temperature, temporal differences in texture, the state of the thermal anomaly region to be verified, and the time-delay consistency results.

[0044] The data acquisition layer is used to acquire visible light images and infrared thermal imaging images from historical acquisition cycles and the current acquisition cycle; the edge processing layer is used to perform image registration, crack mask generation, temperature anomaly area mask generation, extraction of newly added crack areas, and extraction of crack change information; the cloud analysis layer is used to perform crack propagation prediction area determination, management of thermal anomaly areas to be verified, time-delay consistency verification, propagation pattern classification, and machine learning scoring; the result output layer is used to output graded early warning results based on crack activity score, crack development status, time-delay consistency results, and risk trigger type.

[0045] The extended prediction unit, the region management unit, the time-delay consistency verification unit, the extended pattern classification unit, and the activity score correction unit in the cloud analysis layer correspond to the following steps in the method: crack extension prediction region determination, thermal anomaly region determination and storage, location matching relationship judgment, crack extension pattern classification, and crack activity score calculation or correction.

[0046] In terms of monitoring timing, one or more acquisition cycles that have been processed and stored before the current acquisition cycle are considered historical acquisition cycles, and acquisition cycles obtained after the current acquisition cycle are considered subsequent acquisition cycles. The crack propagation prediction region is determined based on the crack end position in the historical acquisition cycles; the thermal anomaly region to be verified is generated in the current acquisition cycle based on the crack propagation prediction region and the temperature anomaly region mask, and is associated and stored with the corresponding crack object, the corresponding crack end, the region position, the generation time, and the verification status; the time-delay consistency result is determined in subsequent acquisition cycles based on the positional matching relationship between the newly added crack region, the thermal anomaly region to be verified, and the crack propagation prediction region.

[0047] When a thermal anomaly region to be verified is generated, it is not directly output as the verified crack propagation risk. In subsequent acquisition cycles, if a newly added crack region satisfies the positional matching relationship with the thermal anomaly region to be verified and the corresponding crack propagation prediction region, the thermal anomaly region to be verified is updated as a verified thermal anomaly region, and the risk contribution of the corresponding crack object is increased. If the positional matching relationship is not satisfied within the effective verification cycle, the contribution of the thermal anomaly region to be verified to the crack activity score is reduced, or it is marked as a candidate region for environmental thermal interference.

[0048] In the initial stage of system startup, when the number of historical acquisition cycles is insufficient to form an end displacement vector sequence, an initial crack propagation prediction region is generated based on the tangent direction of the crack skeleton in the neighborhood of the crack end. When the temporal differences in crack geometry are insufficient to determine the crack propagation pattern, a default fusion weight strategy is used to calculate the basic activity score, or the score result corresponding to the default fusion weight strategy is used as the input feature of the machine learning model. As the number of historical acquisition cycles increases, the system switches to a crack propagation prediction region generation method based on the end displacement vector sequence, and uses accumulated samples to update the machine learning model parameters or score weights.

[0049] In the data acquisition layer, visible light cameras are used to acquire visible light images of building surfaces, while infrared thermal imagers are used to acquire infrared thermal images that have a time-corresponding relationship with the visible light images. The resolution, temperature measurement accuracy, lens focal length, installation distance, and field of view of the visible light camera and infrared thermal imager can be selected based on the size of the crack to be monitored, the monitoring distance, and the on-site environmental conditions, as long as they meet the accuracy requirements for crack image segmentation, infrared temperature anomaly identification, and cross-modal registration. In actual deployment, the visible light camera and infrared thermal imager can be fixedly installed on a monitoring bracket, and the spatial mapping relationship between the two types of images can be obtained through on-site joint calibration, providing a basis for subsequent cross-modal registration and time-series analysis.

[0050] During on-site joint calibration, multiple calibration points or fixed reference points are selected in the near-end, far-end, edge regions, and crack-adjacent regions within the shared effective field of view, and the local registration and reprojection errors are calculated separately. When the shared effective field of view cannot cover the crack area to be monitored, or when the registration and reprojection error of any local region exceeds a preset local error threshold, the corresponding region or the corresponding acquisition period is marked as invalid for thermal anomaly verification. When the difference between the registration and reprojection errors of the near-end and far-end exceeds a preset difference threshold, a regional mapping model is used for registration, or the contribution of temperature anomaly information in that region to the crack activity score is reduced.

[0051] The data acquisition layer may also include a time synchronization module to ensure that visible light images and infrared thermal imaging images have a corresponding acquisition time relationship. The time synchronization module can be implemented using network time synchronization, device-triggered synchronization, or unified timestamp recording. For long-term monitoring scenarios with fixed camera positions, the module can also record the device installation location, shooting angle, and calibration parameters to facilitate image registration and registration error verification in subsequent acquisition cycles.

[0052] The edge processing layer can be implemented by an embedded computing unit, industrial computer, or server, and its hardware configuration is determined based on image resolution, acquisition frequency, and real-time requirements. The edge processing layer can deploy image processing modules and crack segmentation modules to output the masks, registration errors, and crack variation information required for subsequent scoring.

[0053] The cloud-based analytics layer is used to determine the predicted crack propagation area, manage the thermal anomaly area to be verified, verify time-delay consistency, classify propagation patterns, and correct crack activity scores. The results output layer can include mobile terminals, PC display interfaces, on-site alarm devices, or remote management platforms, used to output crack development status, risk trigger types, crack activity scores, and graded early warning results to maintenance personnel.

[0054] like Figure 2 As shown, the monitoring method of the present invention includes the following steps.

[0055] S1 collects multimodal image data of the building to be monitored during historical and current acquisition periods.

[0056] Multimodal image data includes visible light images and infrared thermal images. Visible light images are used to acquire the surface morphology of cracks, while infrared thermal images are used to acquire the temperature distribution on the building surface. The acquisition cycle can be determined based on the building type, crack risk level, site environment, and monitoring requirements. For areas with high risk or rapid changes, the acquisition cycle can be shortened; for stable areas, the acquisition cycle can be extended. The duration of a single acquisition cycle is denoted as Tc, and the total duration of the effective verification cycle formed by N consecutive subsequent acquisition cycles is denoted as Tv. Then:

[0057] Tv = N × Tc;

[0058] Where Tv represents the total duration of the effective verification cycle, N represents the number of subsequent acquisition cycles used for time-delay consistency verification, and Tc represents the time interval between two adjacent image acquisitions.

[0059] To avoid delayed early warnings due to excessively long data collection intervals, TV also meets the following requirements:

[0060] Tv≤min(Ta,Dlim / vh);

[0061] Where Ta represents the preset allowable warning time lag for the crack object to be monitored, Dlim represents the preset dangerous expansion amount, vh represents the historical crack expansion rate calculated based on the historical acquisition period, and min represents taking the smaller value. The historical crack expansion rate vh can be determined based on the ratio of the crack end displacement, crack length increase, or crack width increase to the corresponding time interval.

[0062] When Tv is greater than Ta, or when the crack is estimated to reach Dlim within Tv based on vh, shorten Tc or decrease N; when the crack is in a stable state and vh is lower than the preset stable rate threshold, increase Tc or N appropriately. This establishes a correspondence between the quantity constraints of subsequent acquisition cycles and the actual time length, crack propagation rate, and allowable early warning time lag in engineering.

[0063] S2 performs image registration on multimodal image data.

[0064] Image registration includes cross-modal registration and temporal registration. Cross-modal registration is used to achieve spatial alignment between visible light images and infrared thermal images acquired at the same time; temporal registration is used to achieve spatial alignment between images from historical acquisition cycles and images from the current acquisition cycle.

[0065] In long-term monitoring scenarios at fixed locations, time-series registration can prioritize on-site calibration parameters, fixed reference points, template matching, phase correlation, or affine registration methods to compensate for image translation, rotation, and scale changes caused by micro-vibrations, thermal expansion and contraction, or minor attitude shifts of the monitoring support. When there are insufficient fixed reference points, significant changes in equipment attitude, or movement of the acquisition platform, feature point matching and outlier removal can also be used to estimate the geometric transformation model. After registration, the reprojection error is calculated primarily based on fixed reference points, manually calibrated points, or stable crack feature points confirmed to have not expanded after continuous acquisition cycles. When the reprojection error exceeds a preset registration error threshold, the acquisition cycle is marked as an invalid cycle for thermal anomaly verification to avoid misjudgment of new cracks or misverification of thermal anomalies caused by registration errors.

[0066] Registration check points include at least one of fixed reference points, manually calibrated points, stable crack end points, crack skeleton bifurcation points, and stable crack skeleton sampling points. Stable crack end points and stable crack skeleton sampling points refer to crack feature points that have not been determined to have expanded or have not been used as current crack expansion ends in crack expansion prediction during continuous acquisition cycles. Active ends that have already participated in crack expansion prediction or new crack region determination in the current cycle are not used separately as check points for registration reprojection error. Let the i-th registration check point in the historical image be pi, and its coordinates after registration transformation H mapped to the current image coordinates be H(pi). Let the check point corresponding to pi in the current image be qi. Then the reprojection error ei of the i-th check point is:

[0067] ei = ||H(pi) - qi||2;

[0068] Calculate the root mean square reprojection error e for K registration test points:

[0069] e=sqrt((1 / K)×Σi=1..K ei²);

[0070] Where pi represents the i-th registration check point in the historical image, qi represents the registration check point corresponding to pi in the current image, H represents the spatial transformation relationship obtained by cross-modal registration or temporal registration, H(pi) represents the image coordinates of pi after mapping by H, ei represents the reprojection error of the i-th check point, K represents the number of check points involved in the error calculation, e represents the registration reprojection error of the corresponding acquisition period, ||2| represents the Euclidean distance, sqrt represents the square root operation, and Σ represents the summation operation.

[0071] When K is less than the preset number of test points, the registration reprojection error is calculated using fixed reference points or manual calibration points. When e is greater than the preset registration error threshold, the acquisition period is marked as an invalid period for thermal anomaly verification.

[0072] S3 generates crack masks and temperature anomaly region masks.

[0073] Crack masks can be obtained through image segmentation models or image processing algorithms. Image segmentation models can employ encoder-decoder structures, dilated convolutional structures, or other semantic segmentation networks suitable for segmenting elongated crack targets; image processing algorithms can employ edge detection, thresholding, morphological processing, or combinations thereof. The crack mask generation method is not limited to a specific network structure; both image segmentation models and image processing algorithms can be used to generate crack masks. The geometric transformation between the historical crack mask and the current crack mask is used for subsequent extraction of newly added crack regions, generation of crack expansion prediction regions, and feature input for machine learning models.

[0074] The temperature anomaly region mask can be determined based on the deviation of pixel temperature from the local background region in the infrared thermal image. The local background region can be determined based on crack areas, crack propagation prediction areas, or their adjacent areas. When calculating temperature anomaly regions, obvious saturated pixels, device defects, highly reflective areas, or occluded areas can be removed to reduce the impact of environmental noise on temperature anomaly identification.

[0075] In this embodiment, the current temperature anomaly region mask is mainly used to generate candidate temperature anomaly regions for the current acquisition cycle and further filter the thermal anomaly regions to be verified; the historical temperature anomaly region mask is mainly used to determine whether the temperature anomaly region is a newly emerging or persistent temperature anomaly and to calculate the temperature time-series difference characteristics. If the current temperature anomaly region is located in or near the crack propagation prediction region and meets the temperature anomaly statistical conditions relative to the background statistical benchmark, it can be identified as a thermal anomaly region to be verified; if a historical temperature anomaly region persists for multiple acquisition cycles but has not been verified by subsequently added crack regions, its contribution to the crack activity score is reduced or it is marked as a candidate region for environmental thermal interference. Thus, the current temperature anomaly region mask is used to discover the object to be verified, and the historical temperature anomaly region mask is used to determine the persistence, time-series changes, and environmental interference probability of the object.

[0076] S4 extracts crack change information and temporal difference features.

[0077] Crack variation information is extracted based on historical and current crack masks. This variation information includes at least one of the following: newly added crack regions, changes at crack ends, changes in crack width, changes in crack length, and changes in crack branching.

[0078] The newly added crack area is the remaining area after deducting the area covered by the historical crack mask after dilation processing from the current crack mask.

[0079] In the initial stage, before the number of historical registration reprojection error samples reaches the preset number, the minimum expansion radius or the preset initial expansion radius is used. The preset initial expansion radius can be the same as the minimum expansion radius, or it can be set to a value greater than the minimum expansion radius based on the prior registration stability of the monitoring scenario. For example, in a scenario with a fixed position and stable installation, the preset initial expansion radius can be the minimum expansion radius. In a scenario where the monitoring support is susceptible to wind vibration or temperature deformation, the preset initial expansion radius can be a value greater than the minimum expansion radius to provide greater tolerance for registration errors in the early stages of system startup.

[0080] Once the number of historical registration and reprojection error samples reaches a preset number, the mean, standard deviation, or quantiles of the historical registration and reprojection errors are statistically analyzed to form a statistical confidence interval. When the registration and reprojection error of the current period is within the statistical confidence interval, the minimum expansion radius is used; when the registration and reprojection error of the current period exceeds the statistical confidence interval, the expansion radius is expanded, and the contribution weight of newly added crack regions in this period to the crack activity score is reduced. Thus, the determination of newly added crack regions and registration quality form a closed-loop constraint, reducing the impact of low-quality registration periods on the crack activity score.

[0081] After obtaining the crack mask, texture features of the crack region or its neighborhood can be extracted from the visible light image. These texture features include at least one of crack edge sharpness, grayscale variation, local roughness, or grayscale co-occurrence matrix statistical features. Temperature features of the crack region, crack propagation prediction region, or its background contrast region can be extracted from the infrared thermal imaging image. These temperature features include at least one of temperature mean, temperature standard deviation, temperature gradient, percentage of temperature-abnormal pixels, or temperature anomalous statistics. These features are used for subsequent crack activity score calculation or score correction, and no specific feature extraction algorithm is limited.

[0082] S5, determine the crack propagation prediction area and the thermal anomaly area to be verified. Before determining the thermal anomaly area to be verified, determine the environmental interference of the current acquisition period based on the proportion of temperature anomaly area in the infrared thermal imaging image, the number of temperature anomaly connected regions, the acquisition time, ambient light information, and surface wetness information. When the proportion of temperature anomaly area is greater than a preset area proportion threshold, or when the acquisition time, ambient light information, and surface wetness information meet preset environmental interference conditions, mark the current acquisition period as an environmental interference verification restriction period. For temperature anomaly areas generated within the environmental interference verification restriction period, do not increase the crack risk level based on the temperature anomaly area, or reduce its contribution weight in the selection of thermal anomaly areas to be verified. The ambient light information can be obtained from at least one of the brightness statistics of the visible light image, the overall temperature distribution of the infrared thermal imaging image, the acquisition time, and the solar azimuth calculation results. The surface wetness information can be obtained from at least one of the proportion of reflective areas in the visible light image, the low-temperature connected regions in the infrared thermal imaging image, the on-site humidity record, or the rainfall record. The aforementioned ambient light and surface wetness information are used to determine whether there is large-scale environmental thermal interference in the current acquisition cycle, and are not limited to requiring the setting of independent hardware sensors.

[0083] Before extracting the crack ends, topological breakage processing is performed on the crack mask or crack skeleton. For the crack mask output by the crack segmentation model, it is first refined to obtain the crack skeleton, and then the skeleton endpoints and skeleton bifurcation points are detected. If the distance between two skeleton endpoints is less than a preset breakpoint distance threshold, and the angle between the skeleton tangent directions in the neighborhood of the two skeleton endpoints is less than a preset breakpoint angle threshold, and the image region between the two skeleton endpoints satisfies the crack continuity condition, then the two skeleton endpoints are determined to be breakpoints of the same crack object and connected.

[0084] Let the two endpoints of the skeleton to be judged be ai and aj, and their corresponding neighborhood skeleton tangent direction vectors be ti and tj, respectively. Then the endpoint distance dij and the direction angle θij are respectively:

[0085] dij = ||ai-aj||2;

[0086] θij=arccos(|ti·tj| / (||ti||2×||tj||2));

[0087] Where ai represents the image coordinates of the i-th skeleton endpoint, aj represents the image coordinates of the j-th skeleton endpoint, ti represents the skeleton tangent direction vector in the neighborhood of the i-th skeleton endpoint, tj represents the skeleton tangent direction vector in the neighborhood of the j-th skeleton endpoint, dij represents the Euclidean distance between two skeleton endpoints, θij represents the angle between the corresponding skeleton tangent directions of two skeleton endpoints, arccos represents the inverse cosine function, || represents the absolute value, · represents the vector dot product, and |||2 represents the Euclidean norm.

[0088] When dij is less than a preset breakpoint distance threshold and θij is less than a preset breakpoint angle threshold, it is further determined whether the image region between ai and aj satisfies the crack continuity condition. The crack continuity condition includes at least one of the following: grayscale difference is less than a preset grayscale difference threshold, texture direction consistency is greater than a preset texture consistency threshold, or temperature change does not belong to a region of strong environmental interference.

[0089] For short crack segments whose length is less than a preset short branch length threshold, whose area is less than a preset area threshold, or whose segments fail to appear stably in adjacent acquisition cycles, their corresponding endpoints are marked as pseudo-ends and are not included in the end displacement vector sequence calculation. After completing the breakpoint connection and pseudo-end removal, the end positions of the same crack end in multiple completed acquisition cycles are obtained, and the end displacement vector sequence is calculated.

[0090] The crack propagation prediction region is determined based on the positional changes of the crack tip during historical acquisition cycles. Specifically, the crack skeleton and crack tip are extracted from historical crack masks, and the tip positions of the same crack object in multiple acquisition cycles are correlated. The principal axis direction, length, and width of the crack propagation prediction region are determined based on the average direction, displacement amplitude, and directional dispersion of the tip displacement vector sequence.

[0091] For example, the crack propagation prediction region can be generated as follows: Let the image coordinates of the same crack end in consecutive completed acquisition cycles be P1, P2, ..., Pk, where k ≥ 2. Calculate the end displacement vector ΔPi = Pi - Pi-1, i = 2, ..., k between adjacent acquisition cycles. After removing displacement vectors whose displacement length is less than the registration reprojection error, the remaining displacement vectors are taken as the valid end displacement vectors.

[0092] If the number of effective end displacement vectors is not less than a preset number, the average direction of the effective end displacement vectors is calculated. Specifically, each effective end displacement vector is normalized to a unit vector, and then the unit vectors are summed and normalized again to obtain the principal axis direction of the crack propagation prediction region. If the sum of the unit vectors is less than a preset stability threshold, the end displacement direction is considered unstable, and the width of the crack propagation prediction region is increased. The end displacement amplitude can be taken as the average value of the lengths of the effective end displacement vectors, and the direction dispersion can be taken as the standard deviation or variance of the direction angles of the effective end displacement vectors.

[0093] The crack propagation prediction region can be set as a rectangular, fan-shaped, or capsule-shaped region extending forward along the principal axis, starting from the current crack tip. Taking a rectangular region as an example, let the current crack tip be P, the unit direction of the principal axis be u, and the unit direction perpendicular to u be n. Then the crack propagation prediction region R satisfies the following: the projected distance of any point X within the region relative to P satisfies 0 ≤ (XP)·u ≤ L, and the lateral distance satisfies |(XP)·n| ≤ W / 2. Where L is the length of the prediction region, and W is the width of the prediction region.

[0094] The predicted region length L can be determined based on the end displacement amplitude, registration reprojection error, and minimum predicted length; the predicted region width W can be determined based on the average crack pixel width, registration reprojection error, and directional dispersion. Specifically, L is a combination of the mean end displacement length, the standard deviation of the displacement length, the current period registration reprojection error, and the minimum predicted length; W is a combination of the average crack pixel width, the current period registration reprojection error, and the lateral offset corresponding to the directional dispersion. Therefore, when the end displacement direction is stable, the crack propagation predicted region is narrower; when the end displacement directional dispersion is large or the registration error is large, the crack propagation predicted region is correspondingly wider to cover possible propagation deviations.

[0095] The prediction region length L and prediction region width W are determined as follows:

[0096] L=max(Lmin,μd+σd+ke×e);

[0097] W=max(Wmin,wc+2e+kd×tan(σθ)×L);

[0098] Where L represents the length of the crack propagation prediction region along the principal axis, W represents the width of the crack propagation prediction region perpendicular to the principal axis, Lmin represents the preset minimum prediction length, Wmin represents the preset minimum prediction width, μd represents the average length of the effective end displacement vector, σd represents the standard deviation of the effective end displacement vector length, ke represents the registration error compensation coefficient, e represents the registration reprojection error of the current acquisition cycle, wc represents the average pixel width of the crack in the neighborhood of the current crack end, kd represents the direction dispersion compensation coefficient, σθ represents the standard deviation of the direction angle of the effective end displacement vector, tan represents the tangent function, and max represents taking the larger value.

[0099] When the end displacement direction is stable, σθ is small, and the predicted region width W is mainly determined by the average pixel width wc of the crack and the registration reprojection error e; when the end displacement direction dispersion is large, W increases with σθ to cover possible deviations in the expansion direction.

[0100] When the number of historical acquisition cycles is insufficient at the initial stage of system startup, or the number of effective end displacement vectors is insufficient to form an end displacement sequence, an initial crack propagation prediction region is generated based on the tangent direction of the crack skeleton within the crack end neighborhood. The tangent direction can be obtained through least-squares straight-line fitting of skeleton points within the crack end neighborhood, or through the direction of the line connecting adjacent skeleton points within the end neighborhood. At this time, the length and width of the prediction region use preset initial values ​​and are updated after accumulating end displacement vectors in subsequent acquisition cycles.

[0101] After obtaining the crack propagation prediction region, candidate temperature anomaly regions are determined based on the distribution of historical or current temperature anomaly region masks within the crack propagation prediction region and its vicinity. The spatial relationship between the candidate temperature anomaly region and the crack propagation prediction region includes spatial coverage and spatial proximity. Spatial coverage refers to the overlap ratio between the candidate temperature anomaly region and the crack propagation prediction region being greater than a preset overlap ratio threshold; spatial proximity refers to the centroid distance between the candidate temperature anomaly region and the crack propagation prediction region being less than a preset centroid distance threshold, and the minimum distance between them being less than a preset minimum distance threshold. A candidate temperature anomaly region is identified as a thermal anomaly region to be verified when it satisfies either the spatial coverage relationship or the spatial proximity relationship, and its temperature anomaly statistics are higher than the background statistical benchmark. This avoids misidentifying environmental temperature anomalies as thermal anomaly regions to be verified simply because a single pixel falls into the prediction region or because a single distance indicator is small.

[0102] Temperature anomaly statistics are used to characterize the degree of temperature deviation of candidate temperature anomaly regions relative to the background region. The candidate temperature anomaly region is denoted as A, and the background sample set is denoted as B. The background sample set B can be jointly composed of the first background control region and the second background control region on both sides of the crack propagation prediction region. The first background control region and the second background control region have similar areas and distances from the crack tip to the crack propagation prediction region, and avoid identified crack areas, highly reflective areas, obstructed areas, and equipment defects.

[0103] The temperature anomaly statistics for candidate temperature anomaly region A can be at least one of the following: the absolute value of the difference between the average temperature of the candidate temperature anomaly region and the average temperature of the background sample set; the deviation of the highest or lowest temperature of the candidate temperature anomaly region from the average temperature of the background sample set; the proportion of pixels in the candidate temperature anomaly region that meet the temperature anomaly conditions; and the mean temperature gradient in the candidate temperature anomaly region. To avoid missed detections due to different directions of temperature rise and fall anomalies, this implementation can use the absolute value of the temperature difference between the candidate temperature anomaly region and the background sample set as the initial screening statistic for temperature anomalies, while retaining the sign of the temperature difference, the direction of temperature change, and the number of duration periods as subsequent scoring features. Specifically, let the set of temperature pixels in candidate temperature anomaly region A be TA, and the set of temperature pixels in background sample set B be TB, then the temperature difference δT, the absolute temperature difference Tabs, and the temperature difference sign sT are respectively:

[0104] δT = mean(TA) - mean(TB);

[0105] Tabs = |δT|;

[0106] sT = sign(δT);

[0107] Where TA represents the set of temperature pixels within the candidate temperature anomaly region A, TB represents the set of temperature pixels within the background sample set B, mean represents the average value, δT represents the average temperature difference between the candidate temperature anomaly region and the background sample set, Tabs represents the absolute value of the temperature difference, sT represents the sign of the temperature difference, and sign represents the sign function. When sT is positive, it indicates that the candidate temperature anomaly region is warmer than the background sample set; when sT is negative, it indicates that the candidate temperature anomaly region is cooler than the background sample set.

[0108] Tabs is used to determine whether a candidate temperature anomaly region is included in the set of thermal anomalies to be verified; δT, sT, and the number of duration cycles of the temperature anomaly are used as inputs to the machine learning model or preset scoring rules to distinguish between heating anomalies, cooling anomalies, and persistent environmental thermal disturbances.

[0109] Background statistical benchmarks are used to determine whether the temperature deviation of candidate temperature anomaly regions relative to the background sample set reaches an abnormal level. Let σB be the standard deviation of the temperature pixel set TB in the background sample set B, and MADB be the median absolute deviation. Then:

[0110] σB = std(TB);

[0111] MADB=median(|TB-median(TB)|);

[0112] τT=λ×σB, or τT=λ×MADB;

[0113] Where σB represents the temperature standard deviation of the background sample set, std represents the standard deviation calculation, MADB represents the absolute deviation of the median of the background sample set, median represents the median calculation, τT represents the temperature deviation threshold, and λ represents the threshold adjustment coefficient.

[0114] When the absolute value of the temperature difference (Tabs) between candidate temperature anomaly region A and background sample set B is greater than the temperature deviation threshold (τT), and the proportion of temperature anomaly pixels in candidate temperature anomaly region A is greater than a preset proportion threshold, the candidate temperature anomaly region is considered to meet the statistical conditions for temperature anomalies. The initial value of λ can be 2.0, and the initial value of the temperature anomaly pixel proportion threshold can be 10%. After obtaining historical samples for this monitoring scenario, λ and the temperature anomaly pixel proportion threshold are updated using historical sample calibration.

[0115] Candidate temperature anomaly regions must simultaneously satisfy both spatial constraints and statistical conditions for temperature anomalies to be identified as thermal anomaly regions to be verified. Spatial constraints include spatial coverage relationships or spatial proximity relationships. When a candidate temperature anomaly region has sufficient area overlap with the crack propagation prediction region, it can be considered to satisfy the spatial constraints; when the candidate temperature anomaly region does not form sufficient area overlap, it must simultaneously satisfy both the conditions of close centroid distance and close minimum distance to be considered to satisfy the spatial proximity relationship. This avoids identifying an environmental temperature anomaly region as a thermal anomaly region to be verified solely because a single pixel falls within the crack propagation prediction region or because a single distance index is small.

[0116] Infrared thermal anomaly regions are used as candidate auxiliary signals in crack activity assessment. Candidate temperature anomaly regions must simultaneously meet both spatial constraints and temperature anomaly statistical conditions to be identified as thermal anomaly regions to be verified. These regions also require location matching verification through newly added crack regions in subsequent acquisition cycles. Temperature anomaly regions that fail time-delay verification have their score contribution reduced or are marked as candidate regions for environmental thermal interference.

[0117] S6 performs time-delay consistency verification.

[0118] After the thermal anomaly region to be verified is generated, it is associated and stored with the corresponding crack object, the corresponding crack end, the crack propagation prediction area, the region location, the generation time, and the verification status. In subsequent acquisition cycles, the positional matching relationship is determined based on the spatial relationship between the newly added crack region, the thermal anomaly region to be verified, and the crack propagation prediction area. When the newly added crack region meets the prediction area constraint and the spatial proximity condition with the thermal anomaly region to be verified, it is determined that the newly added crack region and the thermal anomaly region to be verified satisfy the positional matching relationship.

[0119] The predicted region constraints include: the newly added crack region must at least partially fall within the crack propagation predicted region at the corresponding crack tip, or the overlap ratio between the newly added crack region and the crack propagation predicted region must be greater than a preset overlap ratio threshold. Spatial proximity conditions include: the crossover ratio between the newly added crack region and the thermal anomaly region to be verified must be greater than a preset crossover ratio threshold, or the centroid distance between the two must be less than a preset centroid distance threshold, or the minimum distance between the two must be less than a preset minimum distance threshold.

[0120] The directional consistency correction condition is not a mandatory condition for all scenarios, but rather a condition to enhance the confidence of the location matching relationship. The directional consistency correction condition includes: the angle between the direction of the newly added crack region relative to the corresponding historical crack end and the principal axis direction of the crack propagation prediction region is less than a preset angle threshold. This angle can be calculated using the angle between the direction of the line connecting the centroid of the newly added crack region and the historical crack end, and the principal axis direction of the crack propagation prediction region; it can also be calculated using the direction of the line connecting the skeleton end of the newly added crack region and the historical crack end. When the prediction region constraint and spatial proximity condition are met, the location matching is determined to be successful; when the directional consistency correction condition is also met, the positive correction magnitude of this location matching on the crack activity score is increased.

[0121] If the thermal anomaly region to be verified is verified in subsequent acquisition cycles, a first correction amount is added to the crack activity score; if the directional consistency correction condition is also met, a second correction amount is further added; if the thermal anomaly region to be verified is not verified within the effective verification cycle, its contribution to the crack activity score is reduced, or it is marked as a candidate region for environmental thermal interference. The first and second correction amounts can be determined through historical sample calibration, and the first correction amount is preferably greater than the second correction amount.

[0122] If the location matching relationship is satisfied within M consecutive subsequent acquisition cycles, the thermal anomaly region to be verified is updated to a verified thermal anomaly region, and the crack propagation risk verified by time delay is output or the risk level of the corresponding crack object is increased; if the location matching relationship is not satisfied within N consecutive subsequent acquisition cycles, the contribution of the thermal anomaly region to be verified to the crack activity score is reduced, and it is marked as a candidate region for environmental thermal interference.

[0123] Temporal consistency verification is used to determine whether the spatial and temporal consistency conditions are met between the thermal anomaly region to be verified, the predicted crack propagation region, and the subsequently added crack region. It is not used to prove a direct physical causal relationship between the temperature anomaly region and crack propagation. After the thermal anomaly region to be verified is reviewed by the subsequently added crack region, it is only used as an auxiliary correction factor for the crack activity score. When it fails to pass the review within the effective verification period, its score contribution is reduced or it is marked as a candidate region for environmental thermal interference.

[0124] In one specific implementation, the corrected score S1c after time-delay consistency verification is calculated as follows:

[0125] S1c=clip(S0+ηv×Iv+ηd×Id-ηh×Ih-ηq×Iq, 0, 1);

[0126] Wherein, S1c represents the crack activity score after time-delay consistency verification correction, S0 represents the basic activity score, ηv represents the positive correction coefficient when the thermal anomaly region to be verified is verified by the subsequently added crack region, Iv represents the verification indicator, Iv is 1 when the thermal anomaly region to be verified meets the position matching relationship within the effective verification period, otherwise it is 0; ηd represents the direction consistency correction coefficient, Id represents the direction consistency indicator, Id is 1 when the angle between the direction of the newly added crack region relative to the corresponding historical crack end and the main axis direction of the crack propagation prediction region is less than the preset angle threshold, otherwise it is 0; ηh represents the negative correction coefficient of the environmental thermal interference candidate state, Ih represents the environmental thermal interference candidate indicator, Ih is 1 when the thermal anomaly region to be verified is not verified within the effective verification period, otherwise it is 0; ηq represents the negative correction coefficient of the registration quality decline, Iq represents the registration quality invalid or low quality indicator, Iq is 1 when the corresponding acquisition period is marked as the thermal anomaly verification invalid period or the environmental interference verification restricted period, otherwise it is 0; clip indicates that the score is restricted to the range of 0 to 1.

[0127] In the above formula, ηv, ηd, ηh and ηq are determined by historical sample calibration and satisfy ηv is greater than ηd, so that the impact of position matching verification on the score is greater than the direction consistency enhancement condition.

[0128] S7 performs crack propagation pattern classification and machine learning scoring.

[0129] The crack propagation mode is determined according to a preset priority based on the crack length change rate, maximum width change, end displacement, and branch number change. The preset priority is a regular judgment order, specifically: bifurcation composite type takes priority, end extension type takes second, width expansion type takes third, and default fusion strategy is the last resort.

[0130] In one specific implementation, the crack length change rate is the ratio of the crack skeleton length in the current acquisition cycle to the crack skeleton length in the historical acquisition cycle; the maximum width change is the difference between the maximum crack width in the current acquisition cycle and the maximum crack width in the historical acquisition cycle; the end displacement is the distance between the current crack end position and the historical crack end position; and the branch number change is the difference between the current crack skeleton branch number and the historical crack skeleton branch number.

[0131] First, determine whether the conditions for a bifurcation composite type are met. When the change in the number of branches is greater than or equal to 1, or when two or more new skeleton branches are formed in the neighborhood of the same crack end, and the angle between the extension directions of different new skeleton branches is greater than a preset angle threshold, or the length or end displacement of the corresponding new skeleton branch is greater than a preset displacement threshold, the crack propagation mode is determined to be a bifurcation composite type. The angle between the extension directions of different new skeleton branches refers to the angle between the main extension directions of different new skeleton branches, starting from the neighborhood of the same crack end; the preset angle threshold can be 30° to 60°, and the initial value can be 45°; the preset displacement threshold can be determined based on the registration reprojection error and the pixel physical size, and initially can be a pixel displacement greater than the current period's registration reprojection error. The angle in the above bifurcation determination is used to characterize the degree of directional dispersion between different new skeleton branches, which is different from the directional consistency angle between the new crack region and the main axis direction of the crack propagation prediction region in the time-delay consistency verification.

[0132] If the bifurcation composite condition is not met, then it is determined whether the end-extension condition is met. When the end displacement is greater than a preset end displacement threshold, or the crack length change rate is greater than a preset length change rate threshold, the crack propagation mode is determined to be end-extension type. The preset end displacement threshold can be determined based on the registration reprojection error and pixel physical size, and the initial value of the preset length change rate threshold can be 5%.

[0133] If the conditions for bifurcation composite type and end extension type are not met, then it is determined whether the condition for width opening type is met. When the maximum width change exceeds the preset width change threshold, the crack propagation mode is determined to be width opening type. The preset width change threshold can be determined based on the crack width measurement accuracy, registration reprojection error, and pixel physical size.

[0134] If none of the above conditions are met, or if the temporal differences in crack geometry are insufficient to stably distinguish crack propagation patterns, a default fusion weighting strategy is adopted. The default fusion weighting strategy can use equal weighting of geometric features, temperature features, and texture features, or it can use an initial fusion weighting that prioritizes geometric features and supplements them with temperature and texture features.

[0135] In one specific implementation, the machine learning scoring process includes feature normalization, basic activity score calculation, model input construction, and score output.

[0136] First, the temporal differences in crack geometry, temperature, and texture are normalized. The temporal differences in crack geometry include at least one of the following: crack length change rate, maximum width change, end displacement, and branch number change. The temporal differences in temperature include at least one of the following: temperature anomaly statistics, temperature anomaly pixel ratio, and temperature gradient change. The temporal differences in texture include at least one of the following: crack edge grayscale change, local roughness change, or grayscale co-occurrence matrix statistics.

[0137] In one implementation, a basic activity score S0 is first calculated based on geometric feature scores, temperature feature scores, and texture feature scores.

[0138] S0 = α × G + β × T + γ × E;

[0139] Where G is the normalized geometric feature score, T is the normalized temperature feature score, E is the normalized texture feature score, α, β, and γ are the fusion weights of geometric features, temperature features, and texture features, respectively, and α+β+γ=1.

[0140] For end-extending cracks, α, β, and γ can be taken as 0.5, 0.3, and 0.2, respectively; for width-opening cracks, α, β, and γ can be taken as 0.4, 0.2, and 0.4, respectively; for bifurcated composite cracks, α, β, and γ can be taken as 0.3, 0.4, and 0.3, respectively; when the crack propagation mode cannot be stably determined, α, β, and γ can be taken as 0.45, 0.3, and 0.25, respectively. The above fusion weights can be updated through historical sample calibration.

[0141] The machine learning model is used to output a crack activity score S or a crack development status based on at least one of the following: a basic activity score S0, temporal differences in crack geometry, temporal differences in temperature, temporal differences in texture, the state of the thermal anomaly region to be verified, time-delay consistency results, and crack propagation patterns. The machine learning model can employ a regression model, a classification model, an ensemble learning model, or a neural network model, specifically a logistic regression model or a linear regression model. The input feature vector is denoted as x, x = [x1, x2, ..., xm], where x1 to xm represent at least one of the following: normalized crack length change rate, maximum width change, end displacement, branch number change, temperature anomaly statistics, temperature difference sign, temperature anomaly duration cycles, texture change, the state of the thermal anomaly region to be verified, time-delay consistency results, crack propagation pattern encoding, and registration reprojection error.

[0142] When using a logistic regression model to output the crack activity score S, the calculation formula is as follows:

[0143] S=σ(w0+Σj=1..m wj×xj);

[0144] σ(z) = 1 / (1 + exp(-z));

[0145] Where S represents the crack activity score output by the machine learning model, and S is in the interval between 0 and 1; σ represents the Sigmoid function; z represents the linear weighted result; w0 represents the bias term; wj represents the model weight corresponding to the j-th input feature xj; xj represents the j-th input feature; m represents the number of input features; Σ represents the summation operation; and exp represents the exponential function with the natural constant e as the base.

[0146] Let Q be the number of training samples, Sq be the model output corresponding to the q-th training sample, and Yq be the true value of the crack activity. Then the model training loss function Loss is:

[0147] Loss=(1 / Q)×Σq=1..Q(Sq-Yq)²+ρ×Σj=1..m wj²;

[0148] Where Loss represents the training loss, Q represents the number of training samples, Sq represents the model output score of the q-th training sample, Yq represents the ground truth value of the crack activity of the q-th training sample, ρ represents the regularization coefficient, and wj represents the weight of the j-th model. w0 and wj are determined by minimizing Loss.

[0149] When a classification model is used to output the crack development status, the crack development status includes stable state, general development state and rapid development state. The training labels of the classification model are determined by the crack activity ground value Y or the on-site verification results. The loss function can be cross-entropy loss.

[0150] When a thermal anomaly region to be verified is verified by a newly added crack region in a subsequent acquisition cycle, the machine learning model improves the crack activity score or risk level of the corresponding crack object; when a thermal anomaly region to be verified is not verified by a newly added crack region within an effective verification cycle, the machine learning model reduces the contribution of the thermal anomaly region to be verified to the crack activity score, or identifies its corresponding state as a candidate state of environmental thermal interference; when the corresponding acquisition cycle is marked as an invalid thermal anomaly verification cycle or an environmental interference verification restriction cycle, the risk level is not improved based on the temperature anomaly region within that acquisition cycle.

[0151] The training samples for the machine learning model include visible light images, infrared thermal imaging images, crack masks, temperature anomaly region masks, newly added crack regions, the status of thermal anomaly regions to be verified, and the corresponding crack activity ground truth values ​​from historical and subsequent acquisition cycles. The crack activity ground truth values ​​are calculated based on at least one of the following: crack length growth rate, maximum width growth, end displacement, area change rate, branch number change, and displacement change measured by on-site crack gauges or displacement meters.

[0152] The true value of crack activity Y is calculated as follows:

[0153] Y=clip(aL×Ln+aW×Wn+aD×Dn+aA×An+aB×Bn+aM×Mn, 0, 1);

[0154] Where Y represents the true value of crack activity, Ln represents the normalized crack length growth rate, Wn represents the normalized maximum width growth, Dn represents the normalized end displacement, An represents the normalized area change rate, Bn represents the normalized branch number change, and Mn represents the normalized field crack gauge or displacement gauge displacement change; aL, aW, aD, aA, aB, and aM represent the weights corresponding to the above normalized features, and aL+aW+aD+aA+aB+aM=1; clip indicates that the calculation results are restricted to the interval between 0 and 1.

[0155] For any original change Z, its normalized value Zn can be determined as follows:

[0156] Zn = min(Z / Zref, 1);

[0157] Where Z represents the original change to be normalized, Zref represents the reference hazard change corresponding to the original change, Zn represents the normalized change, and min represents taking the smaller value. For example, the reference hazard change corresponding to the crack length growth rate is the length growth rate reference value, the reference hazard change corresponding to the maximum width growth is the width growth reference value, and the reference hazard change corresponding to the end displacement is the end displacement reference value. The above reference hazard changes can be determined based on engineering specifications, field monitoring experience, or historical sample calibration.

[0158] When on-site crack gauge or displacement gauge data is missing, Mn is set to 0, or aM is proportionally allocated to aL, aW, aD, aA, and aB. When an image feature is invalid due to occlusion, invalid registration, or insufficient segmentation quality, the weight corresponding to that feature is not included in the current truth value calculation, and its weight is proportionally allocated to the remaining valid features.

[0159] The crack activity score S is limited to the range of 0 to 1. When the crack activity score S is lower than the first activity threshold, a stable state is output; when the crack activity score S is greater than or equal to the first activity threshold but lower than the second activity threshold, a general development state is output; when the crack activity score S is greater than or equal to the second activity threshold, a rapid development state or a higher-level warning is output. The first and second activity thresholds are determined through historical sample calibration; in the initial stage, the first activity threshold can be 0.3 and the second activity threshold can be 0.6. After the number of historical samples reaches a preset number, the crack activity score distribution of historical samples and the actual crack expansion results are updated. When the number of newly added review samples reaches the preset model update number, or when the deviation between the machine learning model output and the on-site review results exceeds the preset drift threshold, the machine learning model is periodically retrained and incrementally updated using the newly added review samples, or only the score weights and judgment thresholds are updated. Before the model update, the newly added review samples are divided into training samples and validation samples; after the model update, the false detection rate, false negative rate, and average warning delay are calculated on the validation samples. The updated model will only be used in subsequent data collection cycles if it meets the preset false positive rate threshold and false negative rate threshold; otherwise, the previous model will continue to be used or only the scoring threshold will be updated.

[0160] S8 outputs the crack development status or graded early warning results.

[0161] Based on the crack activity score, time-delay consistency results, and risk trigger type, the system outputs crack development status or graded early warning results. Crack development status can include stable, general, and rapid development states. Risk trigger types can include crack geometric propagation risk, thermal anomaly verification risk, time-delay verified crack propagation risk, and environmental thermal interference candidate states. The results output layer can send the score, status, risk trigger type, and corresponding image area to mobile terminals, PC display interfaces, or remote management platforms for maintenance personnel to view and handle.

[0162] To train the machine learning model, training samples can be constructed, including multi-period visible light images, infrared thermal imaging images, crack masks, temperature anomaly area masks, newly added crack areas, the status of thermal anomaly areas to be verified, and verification results. The ground truth value of crack activity is calculated based on at least one of the following: crack length growth rate, maximum width growth, end displacement, area change rate, branch number change, and displacement change collected by on-site crack gauges or displacement gauges. Samples with obstructions, rainwater, shadows, strong reflections, equipment malfunctions, or construction disturbances are discarded as invalid samples.

[0163] λ, temperature anomaly pixel ratio threshold, preset angle threshold, crossover ratio threshold, centroid distance threshold, minimum distance threshold, end displacement threshold, length change rate threshold, width change threshold, first activity threshold, second activity threshold, M and N can all be determined by historical sample calibration.

[0164] In one calibration embodiment, historical samples that have already undergone verification are selected as calibration samples. Calibration samples include samples where crack propagation was confirmed in subsequent acquisition cycles, and environmental temperature anomaly samples where crack propagation did not occur in subsequent verification cycles. Each set of calibration samples includes visible light images, infrared thermal imaging images, crack masks, temperature anomaly area masks, newly added crack areas in subsequent acquisition cycles, and verification results.

[0165] During calibration, candidate parameter combinations are generated using grid search, stepwise search, or Bayesian optimization. The candidate parameter combination is denoted as θ, and θ includes at least one of λ, temperature anomaly pixel percentage threshold, cross-union ratio threshold, centroid distance threshold, minimum distance threshold, end displacement threshold, length change rate threshold, width change threshold, first activity threshold, second activity threshold, M, and N.

[0166] For each candidate parameter combination θ, historical sample backtesting is performed to obtain the false positive rate (FPR)(θ), the false negative rate (FNR)(θ), and the average warning delay (Delay)(θ). Under the constraint that FPR(θ) ≤ FPRmax, the parameter combination that minimizes the objective function J(θ) is selected.

[0167] J(θ)=ω1×FNR(θ)+ω2×Delay(θ)+ω3×||θ-θ0||2;

[0168] Where θ represents the candidate parameter combination, θ0 represents the currently used parameter combination, FPR(θ) represents the false positive rate corresponding to parameter combination θ, FNR(θ) represents the false negative rate corresponding to parameter combination θ, Delay(θ) represents the average warning delay corresponding to parameter combination θ, FPRmax represents the maximum false positive rate allowed by the project, J(θ) represents the calibration objective function value corresponding to parameter combination θ, ω1, ω2 and ω3 represent the weights of false negative rate, average warning delay and parameter change magnitude, respectively, and ||θ-θ0||2 represents the Euclidean distance between the candidate parameter combination and the current parameter combination.

[0169] When multiple parameter combinations all satisfy FPR(θ)≤FPRmax, the parameter combination with smaller J(θ) and shorter average warning delay is preferred. Once the number of newly calibrated samples reaches the preset update quantity, the above calibration process is re-executed. During the initial system deployment phase, λ can be set to 2.0, the threshold for the proportion of temperature-abnormal pixels can be set to 10%, M can be set to 2, N can be set to 3, the first activity threshold can be set to 0.3, and the second activity threshold can be set to 0.6. After the number of historical samples reaches the preset quantity, the above parameters are updated based on the backtracking calculation results of the newly added samples.

[0170] The end displacement threshold, length change rate threshold, and width change threshold are determined based on the registration reprojection error, pixel physical size, and on-site crack gauge or displacement gauge data. Among them, the end displacement threshold is greater than the end position jitter caused by the registration error, and the length change rate threshold and width change threshold are greater than the changes corresponding to the segmentation error and pixel quantization error.

[0171] like Figure 1 As shown, the data acquisition layer outputs visible light images, infrared thermal imaging images, acquisition time, and calibration parameters; after receiving the above data, the edge processing layer outputs crack masks, temperature anomaly area masks, registration reprojection errors, and crack change information; the cloud analysis layer determines the crack propagation prediction area and the thermal anomaly area to be verified based on the data output by the edge processing layer, and performs time-delay consistency verification through the newly added crack areas in subsequent acquisition cycles; the result output layer outputs crack activity scores, crack development status, risk triggering types, and corresponding image areas.

[0172] like Figure 2As shown, the method flow executes in the following order: data acquisition, image registration, mask generation, crack change information extraction, crack propagation prediction region determination, thermal anomaly region determination to be verified, time-delay consistency verification, crack propagation pattern classification, activity score correction, and early warning output. Specifically, the current acquisition cycle is used to generate the thermal anomaly region to be verified, and subsequent acquisition cycles are used to verify whether this thermal anomaly region matches the newly added crack region.

[0173] like Figure 3 As shown, the crack variation information includes changes in crack length, width, end position, and number of branches. This crack variation information is used to determine newly added crack areas, crack propagation prediction areas, and crack propagation patterns, and serves as the geometric feature input for activity scoring.

[0174] like Figure 4 As shown, the crack propagation prediction region is generated based on the crack tip. Temperature anomaly regions must satisfy a spatial coverage relationship or spatial proximity relationship with the crack propagation prediction region, and also meet temperature anomaly statistical conditions, to be identified as thermal anomaly regions to be verified. After the thermal anomaly regions to be verified are generated, they are stored in association with the corresponding crack object, crack tip, prediction region, and generation time.

[0175] like Figure 5 As shown, the thermal anomaly region to be verified has three states: pending verification, verified, and candidate for environmental thermal interference. In the pending verification state, if a newly added crack region within M consecutive subsequent acquisition cycles satisfies a positional matching relationship with the thermal anomaly region to be verified, it transitions to the verified state; if no positional matching relationship is satisfied within N consecutive subsequent acquisition cycles, it transitions to the candidate for environmental thermal interference state. The verified state is used to improve the risk contribution of the corresponding crack object, while the candidate for environmental thermal interference state is used to reduce or eliminate the positive contribution of the temperature anomaly region to the risk score.

[0176] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0177] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for monitoring building cracks, characterized in that, include: S1. Collect multimodal image data of the building to be monitored in the historical acquisition period and the current acquisition period, wherein the multimodal image data includes visible light images and infrared thermal imaging images; S2. Perform image registration on the multimodal image data to ensure that the visible light images and infrared thermal imaging images in the historical acquisition period and the current acquisition period are in a unified spatial coordinate relationship; S3. Obtain historical crack masks and current crack masks based on the registered visible light images, and obtain temperature anomaly region masks based on the registered infrared thermal imaging images. S4. Determine the newly added crack region based on the historical crack mask and the current crack mask, and determine the crack propagation prediction region based on the crack end position in at least one acquisition cycle that has been processed before the current acquisition cycle. S5. The temperature anomaly region in the temperature anomaly region mask is used as a candidate auxiliary signal. Based on the spatial correlation between the candidate auxiliary signal and the crack propagation prediction region, the thermal anomaly region to be verified is determined. The thermal anomaly region to be verified is associated with and stored with the corresponding crack object, the corresponding crack end, the crack propagation prediction region, the generation time, and the verification status. S6. In subsequent acquisition cycles within the effective verification period, determine the time-delay consistency result based on the positional matching relationship between the newly added crack area, the thermal anomaly area to be verified, and the crack propagation prediction area. S7. Input the time-delay consistency results and at least one of the following: crack geometry time-series difference characteristics, temperature time-series difference characteristics, texture time-series difference characteristics, state of the thermal anomaly region to be verified, and crack propagation mode into the machine learning model. Output the crack activity score or crack development status through the machine learning model, and output the crack propagation risk or graded early warning result based on the crack activity score or crack development status.

2. The building crack monitoring method according to claim 1, characterized in that, The image registration includes cross-modal registration and temporal registration; The cross-modal registration is used to achieve spatial alignment between visible light images and infrared thermal imaging images within the same acquisition cycle. The temporal registration is used to achieve spatial alignment between historical acquisition period images and current acquisition period images; After image registration is completed, the registration reprojection error of the corresponding acquisition period is obtained. When the registration reprojection error is greater than the preset registration error threshold, the corresponding acquisition period is marked as an invalid period for thermal anomaly verification. The registration reprojection error is calculated based on at least one of the following verification points: a fixed reference point, a stable crack end point that has been confirmed not to have expanded, a crack skeleton bifurcation point, and a stable sampling point of the crack skeleton.

3. The building crack monitoring method according to claim 2, characterized in that, The newly added crack area is the remaining area in the current crack mask after deducting the area covered by the historical crack mask after expansion processing. The expansion radius of the expansion process is determined based on the current periodic registration and reprojection error, the statistical results of historical registration and reprojection errors, and the minimum expansion radius. When the number of historical registration reprojection error samples does not reach the preset number, the minimum expansion radius or the preset initial expansion radius is used. Once the number of historical registration and reprojection error samples reaches a preset number, if the current period's registration and reprojection error is within the statistical confidence interval of the historical registration and reprojection error, then the minimum expansion radius is adopted; if the current period's registration and reprojection error exceeds the statistical confidence interval, then the expansion radius is expanded, and the contribution weight of newly added crack regions in the corresponding acquisition period to the crack activity score is reduced.

4. The building crack monitoring method according to claim 1, characterized in that, Determining the crack propagation prediction region includes: The crack skeleton is extracted based on the historical crack mask. After breaking point connection repair and pseudo-end removal of the crack skeleton, the crack ends are extracted. Obtain the end position of the same crack tip in multiple completed acquisition cycles, and calculate the end displacement vector sequence; Based on the average direction, displacement amplitude, and directional dispersion of the end displacement vector sequence, a crack propagation prediction region is generated in front of the crack end. When the number of completed acquisition cycles is insufficient to form an end displacement vector sequence, an initial crack propagation prediction region is generated based on the tangent direction of the crack skeleton in the neighborhood of the crack end.

5. The method for monitoring building cracks according to claim 1, characterized in that, The spatial relationships include spatial coverage relationships or spatial proximity combination relationships; The spatial coverage relationship includes a ratio of overlapping area between temperature anomaly region and crack propagation prediction region that is greater than a preset overlap ratio threshold. The spatial proximity combination relationship includes the fact that the centroid distance between the temperature anomaly region and the crack propagation prediction region is less than a preset centroid distance threshold, and the minimum distance between the two is less than a preset minimum distance threshold. When a temperature anomaly region satisfies the spatial coverage relationship or the spatial proximity combination relationship, and the temperature anomaly statistic of the temperature anomaly region is higher than the background statistical benchmark, the corresponding temperature anomaly region is determined as a thermal anomaly region to be verified.

6. The method for monitoring building cracks according to claim 5, characterized in that, A first background control region and a second background control region are determined on both sides of the crack propagation prediction region, and a background sample set is formed by the first background control region and the second background control region; The background statistical benchmark includes a temperature deviation threshold determined based on the standard deviation or median absolute deviation of the background sample set; When the temperature anomaly statistical condition adopts the absolute value of temperature difference condition, the temperature deviation threshold is λ times the standard deviation of the background sample set, or λ times the absolute deviation of the number of digits in the background sample set. When the absolute value of the temperature difference between the candidate temperature anomaly region and the background sample set is greater than the temperature deviation threshold, and the proportion of temperature anomaly pixels in the candidate temperature anomaly region is greater than the preset proportion threshold, the candidate temperature anomaly region is determined to meet the temperature anomaly statistical conditions, and the sign of the temperature difference between the candidate temperature anomaly region and the background sample set, the direction of temperature change, and the number of duration cycles are retained as input features for crack activity scoring or crack development status judgment.

7. The building crack monitoring method according to claim 1, characterized in that, The location matching relationship includes prediction region constraints and spatial proximity conditions; The prediction region constraint includes that the newly added crack region at least partially falls within the crack propagation prediction region at the corresponding crack end, or the overlap ratio between the newly added crack region and the crack propagation prediction region is greater than a preset overlap ratio threshold. The spatial proximity conditions include the crossover ratio between the newly added crack region and the thermal anomaly region to be verified being greater than a preset crossover ratio threshold, or the centroid distance between the two being less than a preset centroid distance threshold, or the minimum distance between the two being less than a preset minimum distance threshold. When the predicted region constraint and spatial proximity condition are met, it is determined that the newly added crack region and the thermal anomaly region to be verified satisfy the position matching relationship. The total duration Tv of the effective verification cycle is determined by the number of subsequent acquisition cycles N and the duration of a single acquisition cycle Tc, satisfying Tv=N×Tc, and Tv is not greater than the allowable warning delay Ta, or not greater than the time limit Dlim / vh determined by the preset dangerous expansion amount Dlim and the crack historical expansion rate vh.

8. The method for monitoring building cracks according to claim 7, characterized in that, The position matching relationship also includes a direction consistency correction condition; The direction consistency correction condition includes that the angle between the direction of the newly added crack region relative to the corresponding historical crack end and the main axis direction of the crack propagation prediction region is less than a preset angle threshold. When the predicted region constraint and spatial proximity condition are met, the position matching relationship is determined to be valid; when the direction consistency correction condition is met simultaneously, the confidence of the position matching relationship is increased, or the positive correction magnitude of the position matching relationship to the crack activity score is increased.

9. The method for monitoring building cracks according to claim 1, characterized in that, The crack propagation mode is determined based on the temporal differences in crack geometry, and the fusion weights of geometric features, temperature features, and texture features are adjusted according to the crack propagation mode. The crack propagation mode includes at least one of the following: end-extending type, width-opening type, and bifurcation composite type; When the change in the number of branches meets the bifurcation determination condition, or when two or more new skeleton branches are formed in the neighborhood of the same crack end, and the angle between the extension directions of the different new skeleton branches is greater than the preset angle threshold, or the length or end displacement of the corresponding new skeleton branch is greater than the preset displacement threshold, it is determined to be a bifurcation composite type. When the bifurcation compound type determination condition is not met, but the end displacement or length change rate meets the end extension determination condition, it is determined to be an end extension type. When the conditions for determining the bifurcation compound type and the end extension type are not met, but the maximum width change meets the condition for determining the width opening type, it is determined to be the width opening type. When the temporal differences in crack geometry are insufficient to stably distinguish crack propagation patterns, the default fusion weight strategy is used to calculate the basic activity score, or the score result corresponding to the default fusion weight strategy is used as the input feature of the machine learning model.

10. A building crack monitoring system based on machine learning, characterized in that, For performing the building crack monitoring method according to any one of claims 1 to 9, the building crack monitoring system comprises: The data acquisition layer is used to acquire visible light images and infrared thermal images of the building to be monitored during historical and current acquisition periods. The edge processing layer, which is communicatively connected to the data acquisition layer, is used to perform image registration on multimodal image data, generate historical crack masks and current crack masks, generate temperature anomaly region masks, and extract newly added crack regions. The cloud-based analysis layer, which is connected in communication with the edge processing layer, is used to determine the crack propagation prediction area, determine and store the thermal anomaly area to be verified, determine the positional matching relationship between the newly added crack area and the thermal anomaly area to be verified, determine the time-delay consistency result, and calculate or correct the crack activity score. The results output layer is connected to the cloud analysis layer and is used to output the crack development status or graded early warning results based on the crack activity score, time lag consistency results and risk triggering type. The cloud-based analysis layer includes an extended prediction unit, a region management unit to be verified, a time-delay consistency verification unit, an extended pattern classification unit, and an activity score correction unit.

Images