Curtain wall structural adhesive operation and maintenance safety monitoring system and method based on multispectral-visual morphology bimodal fusion

Abstract

The embodiment of the invention provides a curtain wall structural adhesive operation and maintenance safety monitoring system and method based on multispectral-visual morphology bimodal fusion, and belongs to the technical field of building curtain wall safety monitoring. The system comprises an optical detection module, a visual detection module, an environment sensor assembly and a control and fusion decision module. The method comprises the following steps: acquiring multi-band reflection spectrum data and a surface topography image of the structural adhesive in parallel; performing temperature compensation and illumination normalization processing on the spectral data based on the real-time environment data; extracting optical features and visual morphology features, and calculating respective aging levels and confidence coefficients; and dynamically calculating a fusion weight according to the bimodal confidence, carrying out weighted fusion on the aging levels of the two modals, and outputting a comprehensive aging evaluation result. According to the method, the detection stability is improved through environment adaptive compensation, and comprehensive, accurate and reliable evaluation of the structural adhesive from internal chemical aging to external physical damage is realized by using an intelligent fusion mechanism based on confidence.

Description

Technical Field

[0001] This application relates to the field of building curtain wall safety monitoring technology, and in particular to a curtain wall structural adhesive operation and maintenance safety monitoring system and method based on multispectral-visual morphology dual-modal fusion. Background Technology

[0002] Structural adhesives are crucial bonding and sealing materials in modern building curtain wall systems, and their durability directly affects the overall safety and lifespan of the curtain wall. Prolonged exposure to harsh environments such as ultraviolet radiation, temperature and humidity cycles, and acid rain can cause chemical degradation and physical deterioration of the structural adhesive, leading to decreased adhesion, loss of elasticity, and potentially serious safety accidents such as glass panel detachment. Therefore, regular and effective safety monitoring of existing curtain wall structural adhesives is essential.

[0003] However, the operation and maintenance inspection of existing building curtain walls faces many unique challenges: First, there is usually a lack of initial state baseline data, making it difficult to establish accurate damage evolution models; second, the on-site environment is complex and variable (such as light and temperature fluctuations), which places extremely high demands on the stability and robustness of the inspection technology; third, the aging of structural adhesives is a complex process involving changes in internal chemical structure and damage to external physical morphology, which cannot be fully characterized by a single inspection method.

[0004] Currently, common non-destructive testing methods have the following limitations: 1) Spectroscopic detection methods (such as infrared and Raman spectroscopy): These methods typically target specific wavelengths or chemical bonds and can only reflect a single aging mechanism, failing to comprehensively capture the complex aging process of structural adhesives involving multiple parallel mechanisms. Furthermore, these methods are extremely sensitive to changes in ambient temperature and light, lacking effective on-site compensation methods, resulting in poor comparability of test results.

[0005] 2) Visual inspection method: Relies on visible light imaging, can only identify surface defects (such as cracks and peeling) that have developed to the macro scale, and cannot detect early internal chemical structure deterioration, resulting in a lag in early warning.

[0006] 3) Simple information fusion: Even when studies attempt to combine multiple technologies, they often employ simple averaging or superposition with fixed weights, lacking intelligent decision-making mechanisms. When detection results from different modalities contradict each other, they cannot effectively identify and process the discrepancies, resulting in a high false positive rate.

[0007] Therefore, there is an urgent need for an intelligent operation and maintenance safety monitoring technology and system for curtain wall structural adhesives that can adapt to complex site environments, integrate internal and external status information, and achieve early warning and accurate assessment. Summary of the Invention

[0008] The main objective of this application is to propose a curtain wall structural adhesive operation and maintenance safety monitoring system and method based on multispectral-visual morphology dual-modal fusion. Through broadband multispectral analysis, visual morphology detection, environmental adaptive compensation, and confidence-based intelligent fusion decision-making, a comprehensive, accurate, and real-time assessment of the aging state of the structural adhesive can be achieved.

[0009] To achieve the above objectives, one aspect of this application proposes a curtain wall structural adhesive operation and maintenance safety monitoring system based on multispectral-visual morphology dual-modal fusion, comprising: The optical detection module, including a broadband light source, optical components, a multi-band filter wheel, a photodetector, and an analog-to-digital converter, is used to acquire multi-band reflectance spectral data of structural adhesive samples. The visual inspection module includes an illumination source and an industrial camera, used to acquire surface morphology images of the structural adhesive sample; Environmental sensor components are used to acquire real-time temperature and illuminance data of the detection environment; The control and fusion decision module is communicatively connected to the optical detection module, the visual detection module, and the environmental sensor assembly, respectively. The control and fusion decision module is configured as follows: Based on the data from the environmental sensor components, the multi-band reflectance spectral data is subjected to temperature compensation and illumination normalization processing to eliminate environmental interference. Optical features are extracted from the processed multi-band reflectance spectral data, and visual morphological features are extracted from the surface morphology image. Based on the optical features and the visual morphology features, the optical modal aging level and its confidence level, and the visual modal aging level and its confidence level are calculated respectively. Based on the optical modality confidence and the visual modality confidence, the fusion weights of the two modalities are dynamically calculated; The optical modal aging level and the visual modal aging level are weighted and fused according to the fusion weights to output a comprehensive aging level evaluation result.

[0010] In some embodiments, the multi-band filter wheel includes at least three filters of different bands, covering a spectral range of 320 nm to 1100 nm.

[0011] In some embodiments, the control and fusion decision module includes: An environmental compensation unit is used to perform the temperature compensation and illumination normalization processes. The optical feature extraction unit is used to extract broadband spectral distribution curve features and characteristic peak parameters related to chemical bond changes from the compensated spectral data. The visual feature extraction unit is used to extract at least one morphological feature parameter from the surface morphology image, including crack density, surface roughness, and peeling area ratio, using a multi-scale morphological gradient and adaptive region growing algorithm.

[0012] In some embodiments, the optical feature extraction unit is configured to use partial least squares to reduce the dimensionality of high-dimensional spectral features, and input the reduced features into a support vector machine classification model to obtain the optical modal aging level and classification probability.

[0013] In some embodiments, the control and fusion decision module is configured to dynamically calculate the fusion weights according to the following formula:

[0014]

[0015] in, and These are the fusion weights for the optical and visual modalities, respectively. and These are the calculated confidence scores for the optical and visual modalities, respectively.

[0016] In some embodiments, the optical modal confidence and visual modality confidence It is calculated based on at least one of the following factors: signal-to-noise ratio, classification probability, and feature consistency of the corresponding modality.

[0017] In some embodiments, the control and fusion decision module further includes a conflict detection unit, configured as follows: Calculate the difference between the optical modal aging level and the visual modal aging level; When the difference value exceeds the first preset threshold, a historical data cross-validation or secondary inspection process is triggered.

[0018] To achieve the above objectives, another aspect of this application proposes a working method based on the aforementioned monitoring system, comprising the following steps: Parallel optical and visual inspection: The optical inspection module acquires multi-band reflectance spectral data of the structural adhesive sample, while the visual inspection module acquires surface morphology images of the sample. Environmental data acquisition: Real-time acquisition of ambient temperature and illuminance through environmental sensor components; Environmental adaptive compensation: Based on the ambient temperature and illuminance, the multi-band reflectance spectral data are subjected to temperature compensation and illuminance normalization processing; Feature extraction: Optical features are extracted from the compensated spectral data, and visual morphological features are extracted from the surface morphology image; Confidence assessment: Based on the optical features and visual morphology features, calculate the optical modal aging level and its confidence level, and the visual modal aging level and its confidence level, respectively. Dynamic fusion decision: The fusion weight is dynamically calculated based on the optical modality confidence and the visual modality confidence, and the aging levels of the two modalities are weighted and fused using the weight to output the comprehensive aging level.

[0019] In some embodiments, the step of extracting visual shape features includes: The surface topography image is processed using a multi-scale morphological gradient algorithm to obtain a multi-scale gradient map, which is then fused. The fused gradient map is segmented based on a local adaptive threshold to obtain candidate crack pixels; An adaptive region growing algorithm is used to grow and verify the candidate crack pixels in order to quantify the crack density.

[0020] In some embodiments, a conflict resolution step is also included: Compare the optical modal aging level with the visual modal aging level; If the difference between the two exceeds the preset range, then at least one of the following operations will be performed depending on the degree of difference: enable cross-validation using historical detection data, trigger the system to perform a second detection, or generate a prompt message requiring manual review.

[0021] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0022] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0023] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0024] Compared with the prior art, this application has the following beneficial effects: 1) Comprehensiveness: By using dual-modal parallel detection of optics (internal chemistry) and vision (external morphology), the aging state characterization of structural adhesives from microscopic molecular structure changes to macroscopic physical damage is realized.

[0025] 2) Accuracy: An environmental adaptive compensation mechanism is introduced to effectively suppress the impact of temperature and light fluctuations on spectral detection, and improve the consistency and comparability of detection results under different times and environments.

[0026] 3) Intelligence: The system adopts a confidence-based dynamic fusion weight algorithm instead of fixed weights, which enables the system to automatically adjust the contribution of each modality according to the specific data quality of each detection and to activate the error correction mechanism when the results conflict, which significantly improves the reliability and robustness of decision-making.

[0027] 4) Practicality: The system is non-destructive and in-situ testing, easy to operate, and is particularly suitable for large-scale periodic inspections and safety assessments of existing building curtain walls, and has significant engineering application value. Attached Figure Description

[0028] Figure 1 This is a structural block diagram of the curtain wall structural adhesive operation and maintenance safety monitoring system based on multispectral-visual morphology dual-modal fusion in the embodiments of this application.

[0029] Figure 2 This is a schematic diagram of the overall structure of a curtain wall structural adhesive operation and maintenance safety monitoring system according to an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of the multispectral SVM algorithm (PLS-SVM fusion algorithm) provided in the embodiments of this application.

[0031] Figure 4 This is a schematic diagram of the visual shape algorithm (multi-scale morphological gradient and adaptive region growing algorithm) provided in the embodiments of this application.

[0032] Explanation of the labels in the diagram: 1-Broadband light source, 2-Industrial camera, 3-Lighting source and environmental sensor assembly, 4-Aperture, 5-Lens group, 6-Multi-band filter wheel, 7-Large target photodetector, 8-High-precision analog-to-digital converter (ADC), 9-Control and fusion decision module. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0035] As a key connecting material in modern building curtain wall systems, the durability of glass structural adhesives directly determines the service life and safety of the building's external envelope. Long-term exposure to environmental factors such as ultraviolet radiation, acid rain, and temperature fluctuations can cause irreversible aging and degradation of glass structural adhesives, leading to decreased bonding performance and reduced mechanical strength. In severe cases, this can result in safety accidents such as glass curtain wall detachment.

[0036] The inspection of existing building curtain walls faces unique challenges: (1) lack of original design parameters and initial state benchmark data, making it difficult to establish an accurate damage evolution model; (2) existing inspection technologies are not robust to environmental interference (light changes, temperature fluctuations), resulting in a deviation of more than 20% in the test results of the same sample at different times; (3) a single inspection method is difficult to fully characterize the complex aging process of structural adhesives.

[0037] Traditional destructive testing requires disassembly and sampling, which is time-consuming and costly, and cannot meet the needs of large-scale real-time evaluation. Existing non-destructive testing technologies have the following limitations: (1) Single-band spectral detection methods: such as infrared spectroscopy, which only relies on the ratio of absorption peaks at a specific wavenumber, and monochromatic Raman spectroscopy, which only focuses on the vibration of specific chemical bonds. These methods can only reflect changes in a single chemical mechanism and cannot fully characterize the complex aging process of glass glue through multiple mechanisms. Broadband multispectral detection can cover a wider wavelength range. By analyzing the overall spectral distribution curve and multiple characteristic peaks, it can simultaneously capture multiple aging mechanisms such as siloxane bond breakage, reduced crosslinking density, and yellowing. (2) Single visual detection method: It can only analyze macroscopic defects on the surface and cannot detect internal chemical structure deterioration and early aging signs, resulting in detection lag. (3) Poor environmental adaptability: Existing spectral detection methods are sensitive to changes in environmental temperature and light conditions and lack effective environmental compensation mechanisms, resulting in poor comparability of detection results at different times or under different weather conditions. (4) Simple information fusion method: Even if existing technologies combine multiple detection methods, they are only simple superposition of independent detection results or fixed weight average, lacking intelligent fusion mechanism. When different detection results contradict each other, they cannot be effectively handled, resulting in a high misjudgment rate.

[0038] In view of this, this application proposes a structural adhesive monitoring system and method based on multispectral-visual morphology dual-modal fusion. Through broadband multispectral analysis, environmental adaptive compensation, dual-modal parallel detection, and intelligent fusion decision-making, it achieves a comprehensive, accurate, and real-time assessment of the aging state of glass adhesive. Specifically, the core innovations of this application include: Innovation Point 1: Synergistic Analysis of Broadband Multispectral Distribution and Characteristic Peaks Unlike traditional single-band or narrow-band spectral detection, this application employs a broadband multispectral detection method, covering a wide band from visible to near-infrared. It extracts spectral features at different wavelengths using a multi-band filter wheel. This method not only analyzes the overall spectral distribution curve morphology but also simultaneously extracts the position, intensity, and peak width parameters of multiple characteristic peaks. This broadband multispectral information can simultaneously reflect multiple chemical mechanism changes during the aging process of structural adhesives, including siloxane bond breaking (corresponding to absorption / reflection changes at specific wavelengths), decreased crosslinking density (leading to changes in the overall spectral curve morphology), and yellowing reactions (causing characteristic peak shifts in the visible light band). Compared to single-band methods, it provides a more comprehensive and accurate assessment of the aging state.

[0039] Innovation Point Two: Adaptive Environmental Compensation Mechanism In response to the complex and ever-changing nature of existing building site environments, this application innovatively introduces environmental sensor components and an environmental compensation unit to achieve environmentally adaptive spectral detection. Specifically, it includes: (1) Temperature compensation algorithm: establishing a temperature-spectral response relationship model, correcting the reflectivity data of each band based on the real-time measured ambient temperature, and eliminating ambient temperature differences ( (1) The influence of the range) on the spectral characteristics; (2) Adaptive correction under illumination conditions: The ambient light interference is eliminated by dark current differential technology, and the data of the illumination sensor is normalized to ensure that the comparability of the detection results under different illumination conditions (indoor, outdoor, cloudy, sunny) is improved to more than 95%. This adaptive compensation mechanism enables the system to maintain stable detection performance under various environmental conditions, which is something that traditional spectral detection methods cannot achieve.

[0040] Innovation Point 3: Dual-modal parallel detection mechanism This application designs a dual-modal detection architecture in which an optical inspection module and a visual inspection module operate in parallel. The optical inspection module characterizes the aging of the structural adhesive from the perspective of internal chemical composition through multi-band spectral analysis, and can detect early chemical structural changes (such as molecular chain breakage and reduced cross-linking degree), providing early warning capabilities. The visual inspection module characterizes the aging of the structural adhesive from the perspective of external surface morphology through an industrial camera, and can detect macroscopic physical damage (such as cracks, peeling, and increased surface roughness). The two modalities detect the structural adhesive from different angles and depths, complementing each other: the optical modality captures early internal aging, while the visual modality confirms the degree of external damage, jointly achieving a comprehensive assessment of the aging state of the structural adhesive.

[0041] Innovation Point 4: Confidence-Based Intelligent Dual-Modal Fusion Decision Making Unlike simple fixed-weight fusion, this application establishes a confidence-based intelligent fusion framework. First, the confidence evaluation unit calculates the detection confidence of the optical and visual modalities based on multiple factors such as signal-to-noise ratio, classification probability, and feature consistency. Then, the adaptive fusion unit dynamically calculates the fusion weights based on the confidence of the two modalities. and And calculate the weighted and integrated overall aging level. This dynamic weighting mechanism automatically adjusts the contribution of the two modalities based on different detection conditions: increasing the weight of the optical modality when its confidence level is high (e.g., good lighting conditions, high signal-to-noise ratio), and increasing the weight of the visual modality when its confidence level is high (e.g., clear surface features). Furthermore, the conflict detection unit can identify inconsistencies between the two modal results and trigger secondary checks or manual verification to avoid misjudgments. This intelligent fusion mechanism significantly improves the accuracy and reliability of the detection results.

[0042] like Figure 1 As shown in the figure, this application provides a curtain wall structural adhesive operation and maintenance safety monitoring system based on multispectral-visual morphology dual-modal fusion. Its main hardware consists of an optical detection module, a visual detection module, environmental sensor components, and a control and fusion decision module. The following is a detailed description... Figure 2 Explanation: (1) System hardware composition and workflow The optical path of the optical detection module begins with a broadband light source 1 (such as a halogen tungsten lamp or LED array). The continuous broadband beam emitted by this light source is focused and shaped by the lens group 5 after the aperture is controlled by the aperture 4, forming a uniform light spot on the surface of the structural adhesive sample to be tested. The beam then enters a multi-band filter wheel 6, which is driven by a motor and can sequentially switch at least three different wavelengths (covering 320-1100nm) of filters to achieve beam splitting. The reflected light from the sample surface is received by a large-target photodetector 7 (spectral response range 320-1100nm) and converted into an analog electrical signal. This signal is then converted into a digital signal by a high-precision ADC 8 and sent to the control and fusion decision module 9.

[0043] The illumination source of the vision inspection module (integrated in component 3) is an LED ring light, which provides uniform illumination to the sample surface. The industrial camera 2 vertically captures high-resolution topographic images of the sample surface and transmits them to the control and fusion decision module 9.

[0044] The environmental sensor assembly (integrated in assembly 3) includes a high-precision temperature sensor and an illuminance sensor, which monitor environmental parameters in real time and send them to the control and fusion decision module 9.

[0045] The control and fusion decision module 9 serves as the core of the system. It employs a high-performance embedded processor and is responsible for coordinating the operation of various hardware components, processing data, executing algorithms, and outputting the final results.

[0046] (2) Detailed description of the optical detection module The optical detection in this embodiment employs a broadband multispectral strategy. The multi-band filter wheel 6 covers the visible and near-infrared bands. The visible light band (~400-700nm) is sensitive to aging products such as yellowing and oxidation of the structural adhesive, while the near-infrared band (~700-1100nm) shows a significant response to changes in molecular chain structure and crosslinking density. By sequentially measuring the reflectance in each band, a broadband reflectance spectrum curve of the sample can be constructed.

[0047] Module 9 controls the rotating wheel 6 to switch filters and performs dark current acquisition (light source 1 is turned off) before each band measurement, eliminating substrate noise through signal differential. The acquired multidimensional spectral data vectors form the basis of the sample's optical characteristics.

[0048] (3) Detailed description of the visual inspection module The visual inspection module focuses on extracting surface physical defects. Its core algorithm flow is as follows: Figure 4 As shown, it mainly includes: 1) Multi-scale morphological gradient calculation: morphological gradient calculation (dilation map minus erosion map) is performed on the original image using circular structuring elements with different radii (e.g., 1, 3, 5, 7 pixels) to capture fine and coarse cracks respectively.

[0049] 2) Multi-scale gradient fusion: Weighted fusion of gradient maps at various scales, with an emphasis on preserving large-scale (i.e. more significant) crack features.

[0050] 3) Local adaptive threshold segmentation: Calculate the mean and standard deviation of the local neighborhood of each pixel in the fused gradient map, dynamically set the segmentation threshold to adapt to the contrast changes in different regions of the image, and obtain candidate crack pixels.

[0051] 4) Adaptive region growth: Using candidate pixels as seeds, region growth is performed based on the similarity between pixel grayscale and the growth region. The growth threshold is adaptively adjusted according to the region contrast to connect broken crack segments.

[0052] 5) Morphological feature verification and quantification: Calculate morphological parameters such as aspect ratio, curvature, and continuity for the grown region, and filter out pseudo-cracks. Finally, quantify and output morphological feature parameters such as crack density, average width, maximum length, surface roughness, and spalling area ratio.

[0053] (4) Detailed description of environmental compensation mechanism To achieve stable on-site monitoring, the system integrates an environmental adaptive compensation unit. This unit corrects the reflectance of each wavelength band based on real-time temperature data using a pre-calibrated temperature-spectral response model (such as a polynomial correction model). Simultaneously, it performs illumination normalization processing on the spectral data by combining dark current differential and real-time illuminance data. This mechanism effectively eliminates... The influence of temperature difference and daily light changes improves the comparability of test results under different environments to over 95%.

[0054] (5) Detailed description of dual-modal signal fusion and decision-making process The fusion decision-making process embodies the intelligence of this invention, and its core process is as follows: Feature Extraction and Classification: The environmentally compensated spectral data is fed into the optical feature extraction unit to extract broadband spectral curve morphological parameters (such as curvature and area) and the position, intensity, and peak width parameters of multiple feature peaks. These high-dimensional features are first subjected to partial least squares (PLS) dimensionality reduction (extracting 3-5 latent variables most relevant to aging), and then input into a support vector machine (SVM) classifier to obtain the optical modal aging level. and its classification probability (See process) Figure 3 The morphological feature parameters output by the visual feature extraction unit are directly input into another SVM classifier to obtain the visual modality aging level. and its probability .

[0055] Confidence assessment: The confidence assessment unit calculates the confidence score (C) for each mode by integrating multiple factors. For optical modes, The spectral signal-to-noise ratio (SNR) and classification probability ( The confidence level of the visual modality is calculated by weighting the consistency of the variation trends of the characteristic peak parameters. The calculation is similar, taking into account image sharpness, classification probability, and consistency of morphological features.

[0056] Dynamic weighted fusion: The adaptive fusion unit dynamically allocates weights based on confidence level.

[0057]

[0058] The final overall aging rating is:

[0059] This mechanism ensures that the system relies more on optical results when the lighting is good and the spectral data quality is high, while it relies more on visual results when surface defects are obvious and the image is clear.

[0060] Collision Detection and Resolution: Calculation of Collision Detection Unit If the difference is ≤1, the fusion result is output normally; if the difference is 2, historical data cross-validation is triggered, and the past records of the sample are queried to assist in the judgment; if the difference is ≥3, it is considered that the conflict is serious, and the secondary detection process is automatically triggered. If the conflict still exists, an "uncertain" flag is output and a manual review is prompted, which greatly avoids misjudgment.

[0061] Finally, the control and fusion decision module 9 generates a complete test report containing the overall level, results of each modality, fusion weights, environmental parameters, and conflict explanations, which is stored and uploaded for further analysis and decision-making.

[0062] The following is a further supplementary description of the solutions in the embodiments of this application, in conjunction with specific implementation methods.

[0063] The working process of the optical detection module is as follows: Figure 2 As shown, a broadband light source 1 is used as the detection light source. The broadband continuous beam emitted by the light source 1 is transmitted forward and controlled by the aperture 4. The beam size and spot uniformity can be controlled by adjusting the aperture diameter. After passing through the aperture 4, the beam enters the lens group 5. Specifically, this lens group consists of a high-transmittance quartz lens or a coated glass lens, with a total transmittance ≥95% and an adjustable focal length. The function of the lens group 5 is to focus the beam and improve the beam quality, ensuring the formation of a uniform spot with a diameter of 10-30 mm on the sample surface.

[0064] The beam, shaped by lens group 5, enters the multi-band filter wheel 6. This wheel contains at least three filters of different wavelengths, covering a wide spectral range of 320-1100 nm (including visible and near-infrared bands). This broadband multispectral design is one of the core innovations of this invention: unlike traditional single-band or narrow-band detection, broadband multispectral can simultaneously capture multiple chemical mechanisms in the aging process of structural adhesives. Specifically, the visible light band (400-700 nm) can detect yellowing reactions, and the near-infrared band (700-1100 nm) can reflect changes in crosslinking density. The wheel 6 is driven by a motor via an electrical control line by the control and fusion decision module 9, realizing automatic switching of the filters. The beam of light selected by the wheel 6 is tilted and irradiated onto the surface of the curtain wall structural adhesive sample placed on the sample detection device.

[0065] The sample surface diffusely reflects incident light, and the reflected light contains spectral characteristics of both the sample surface and its interior. Because the aging of the glass structural adhesive leads to changes in its chemical composition and molecular structure (such as siloxane bond breakage, reduced crosslinking density, and yellowing), different wavelengths of light exhibit varying sensitivities to these changes. Therefore, the spectral characteristics of the reflected light change accordingly with the degree of sample aging. By analyzing the overall morphology of the broadband spectral distribution curve, macroscopic information about the structural adhesive's aging can be obtained; by extracting the position, intensity, and peak width parameters of characteristic peaks, the degree of change in specific chemical bonds can be precisely quantified. The combination of these two methods achieves a comprehensive characterization of the aging state.

[0066] The light signal reflected from the sample is received by a large-area photodetector 7. Exemplarily, the large-area photodetector 7 employs a silicon photodiode with an effective photosensitive area ≥ 50 mm² and a spectral response range of 320-1100 nm, matching the wavelength range of the filter wheel, enabling high-sensitivity detection of broadband multispectral distributions. The light signal is converted into an electrical signal within the large-area photodetector 7, with the current amplitude proportional to the incident light intensity. Because samples at different aging stages exhibit varying reflectivities for different wavelengths of light, the intensity of the electrical signal output by the large-area photodetector 7 will also show characteristic differences.

[0067] The analog electrical signal output by the large-area photodetector 7 is transmitted to the high-precision analog-to-digital converter 8 via electrical cables. Exemplarily, the high-precision analog-to-digital converter 8 employs a 24-bit resolution and automatically adjusts the gain according to the intensity of the optical signal to obtain the optimal signal-to-noise ratio. The high-precision analog-to-digital converter 8 incorporates a digital low-pass filter, which can effectively suppress 50Hz / 60Hz power frequency interference and high-frequency noise. The digital signal converted by the high-precision analog-to-digital converter 8 is transmitted to the control and fusion decision module 9 via a data bus (SPI or I2C interface).

[0068] During multi-band detection, the control and fusion decision module 9 sends switching commands to the multi-band filter wheel 6 via control lines, sequentially switching to filters of different bands. After each band switch, the system waits 50-100ms to ensure mechanical vibration stability before the high-precision analog-to-digital converter 8 begins acquiring the reflected light signal in that band. To improve the signal-to-noise ratio, the control and fusion decision module 9 also controls the switching of the broadband light source 1. Before each band measurement, the light source is first turned off to measure the dark current, then turned on to measure the signal light, and the influence of dark current is eliminated through differential calculation. This dark current differential technique is an important component of the environmental adaptive compensation mechanism, effectively eliminating interference from ambient light. After completing the measurements of all bands, the control and fusion decision module 9 obtains a set of multi-dimensional spectral data vectors, constituting a complete characterization of the optical aging features of the sample.

[0069] The working process of the visual inspection module is as follows: Figure 2 As shown, the visual inspection module and the optical inspection module work in parallel to inspect the surface morphology of the sample. This dual-modal parallel inspection architecture is a key innovation of this invention: the optical inspection module characterizes aging from the perspective of internal chemical composition, capturing early changes in molecular structure; the visual inspection module characterizes aging from the perspective of external surface morphology, confirming macroscopic physical damage. The two complement each other, jointly achieving a comprehensive assessment of the aging state of the structural adhesive.

[0070] In one embodiment, the illumination source is an LED ring light source, which is installed above the sample detection device to provide uniform and stable illumination, avoiding strong reflections and shadows. Under the illumination source, the morphological features of the sample surface (including defects such as cracks, peeling, and changes in surface roughness) are clearly displayed.

[0071] Industrial camera 2 captures images of the sample surface vertically from above. Specifically, industrial camera 2 employs a CMOS or CCD image sensor with a resolution ≥5MP, capable of capturing micron-level surface details. By adjusting the lens focal length and working distance, images with different field of view sizes and resolutions can be obtained as needed, meeting different requirements for macroscopic defect observation and microscopic detail detection. Image data captured by industrial camera 2 is transmitted to control and fusion decision module 9 via a digital interface (GigE or USB 3.0). Module 9 receives and processes the image data in real time. Module 9 sends a shooting trigger command to industrial camera 2 via a control signal line, ensuring time synchronization between image acquisition and spectral acquisition.

[0072] The environmental sensor components operate as follows: The system is equipped with temperature and illuminance sensors to monitor environmental parameters at the detection site in real time. The temperature sensor has a measurement accuracy of ±0.5℃, and the illuminance sensor has a measurement range of 0-100,000 Lux. These environmental parameters are transmitted in real time to the environmental compensation unit of the control and fusion decision module 9. The environmental compensation unit establishes a temperature-spectral response relationship model, which can perform linear or nonlinear correction on the reflectance data of each band based on the real-time measured ambient temperature, eliminating the influence of ambient temperature difference (within ±20℃ range) on spectral characteristics. Simultaneously, the illuminance sensor data is used for illumination normalization processing, combined with dark current differential technology, to ensure the comparability of detection results under different illumination conditions (indoor, outdoor, cloudy, sunny). This adaptive environmental compensation mechanism is a significant innovation of this invention, enabling the system to maintain stable detection performance under various environmental conditions.

[0073] The implementation process of dual-modal signal fusion: The control and fusion decision module 9 is the core processing unit of the system. In one embodiment, the control and fusion decision module 9 adopts a high-performance embedded processing platform (ARM Cortex-A series or equivalent processor) and supports the Linux real-time operating system. The control and fusion decision module 9 includes multiple functional sub-modules: main control unit, environmental compensation unit, optical feature extraction unit, visual feature extraction unit, machine learning classifier, confidence evaluation unit, adaptive fusion unit, collision detection unit, and data storage unit.

[0074] Data streams from the high-precision analog-to-digital converter 8, industrial camera 2, and environmental sensors converge within the control and fusion decision module 9. The optical modal data stream contains multi-band reflectivity data (typically 5-10 bands), with a data update frequency of approximately 1-2 Hz, depending on filter switching and sampling speed. The visual modal data stream contains high-resolution image data (5 MP, approximately 2-3 MB / frame). The environmental data stream contains real-time temperature and illuminance data. All three data streams are temporarily stored in the input buffer of the control and fusion decision module 9 and timestamped to ensure time synchronization.

[0075] The environmental compensation unit first processes the optical modal data. Based on temperature sensor data, it applies a temperature-spectral response model to correct the reflectivity of each band. This model is established through prior calibration experiments and typically uses a polynomial fitting approach. ,in: The corrected reflectivity. For measured values, For wavelength, For temperature, For temperature deviation, and The calibration coefficients are then used. Next, based on the illuminance sensor data and dark current differential results, the reflectance is normalized to eliminate the influence of ambient light variations. The multi-band reflectance data after environmental compensation exhibits better consistency and comparability.

[0076] The optical feature extraction unit extracts two types of features from the compensated multi-band reflectance data: 1) broadband spectral distribution curve features, including overall shape parameters of the curve (such as curvature, slope, area, etc.), which reflect macroscopic information about the aging of the structural adhesive; 2) characteristic peak parameters, including peak position (wavelength), peak intensity (reflectance or absorbance), and peak width (FWHM), which precisely quantifies the degree of change in specific chemical bonds. For example, the breaking of siloxane bonds leads to a decrease in the intensity of a certain characteristic peak, a decrease in crosslinking density leads to an increase in peak width, and yellowing reactions cause the peak position to shift towards longer wavelengths. By extracting characteristic peak parameters from multiple bands, a multi-dimensional feature vector is constructed to comprehensively characterize the optical aging features of the structural adhesive.

[0077] The visual feature extraction unit—based on a multi-scale morphological gradient and adaptive region growing crack extraction algorithm—processes images from industrial camera 2, including preprocessing steps such as noise reduction and contrast enhancement. Then, it uses a multi-scale morphological gradient algorithm to extract crack features, such as… Figure 4 As shown. The algorithm includes the following key steps: 1) Multi-scale morphological gradient calculation: Morphological gradients are calculated using circular structuring elements SE_r with radii r = 1, 3, 5, and 7 pixels, respectively. The morphological gradient is defined as the difference between the dilated image and the eroded image. Where I(x,y) is the original image, Dilate is the morphological dilation operation, and Erode is the morphological erosion operation. Small scale (r=1,3) can capture fine cracks, while large scale (r=5,7) can capture coarse cracks.

[0078] 2) Multi-scale fusion: The gradient maps at four scales are fused according to their weights. Specifically, weighting coefficients , , , Larger scales have higher weight because coarse cracks are the main safety hazard.

[0079] 3) Local adaptive threshold segmentation: Unlike the global dual threshold strategy of the Canny operator, this embodiment uses a local adaptive threshold: .in This represents the average gradient within a local window (e.g., 25×25 pixels). For local standard deviation, The adaptive coefficient is (1.5~3.0). Candidate crack pixel set: .

[0080] 4) Adaptive region growing: for candidate sets Each pixel in the image is used as a seed point for region growing. The growing criterion is based on a similarity metric: .when At that time, add pixel (x, y) to the region. Growth threshold. Adaptive adjustment based on regional characteristics: High-contrast areas Low contrast areas .

[0081] 5) Morphological feature verification: Verify the morphological features of each growth region and eliminate false cracks. Valid crack criteria: Aspect ratio AR>5 (slender shape), curvature Bend>1.05 (moderate curvature is allowed), continuity Cont<0.1 (good continuity).

[0082] 6) Crack feature quantification: The final extracted morphological feature parameters include: ① Crack density ② Average crack width ③ Maximum crack length ④ Surface roughness; ⑤ Peeling area ratio. Compared with the traditional Canny edge detection algorithm, the algorithm of this invention has significant improvements in crack detection rate (+16%), false detection rate (-9%), and contour integrity (+27%).

[0083] The machine learning classifier uses a PLS-SVM fusion algorithm for the optical modality and an SVM algorithm for the visual modality. The machine learning classifier classifies and identifies the optical feature parameters and the visual feature parameters respectively.

[0084] For optical modes, a PLS-SVM (Partial Least Squares-Support Vector Machine) fusion algorithm is used, such as... Figure 3 As shown, it includes two stages: Phase 1: PLS Dimensionality Reduction. A high-dimensional spectral feature vector X (dimension m = 15~20) is input into the PLS model, extracting k = 3~5 latent variables T = [t1, t2, ..., t_k]. The PLS algorithm iteratively calculates: weight vector w_i = X_i^T·Y_i / ||X_i^T·Y_i||, score vector t_i = X_i·w_i, loading vectors p_i and c_i, and updates the residual X_{i+1} = X_i - t_i·p_i^T. The iterative process extracts principal components while maximizing the covariance with the aging level Y, effectively handling the collinearity problem of spectral data. The number of latent variables k is determined through cross-validation, selecting the k value that minimizes the prediction error RMSECV.

[0085] Advantages of PLS ​​dimensionality reduction: (1) It compresses high-dimensional features (15~20 dimensions) into a low-dimensional space (3~5 dimensions), reducing the risk of overfitting; (2) The extracted latent variables retain the information most relevant to the aging level; (3) It has a filtering effect and suppresses spectral noise; (4) The weight vector w_i can identify the spectral features most sensitive to aging.

[0086] Phase 2: SVM Classification. The dimensionality-reduced latent variable T is input into the SVM classification model. The SVM uses a radial basis function (RBF) kernel: K(T_i, T_j) = exp(-γ·||T_i - T_j|| 2 The parameter γ is optimized through grid search and cross-validation. Multi-class classification employs a One-vs-One (OvO) strategy, training 10 binary classifiers, and the aging level G_optical is determined by voting. The classification probability P_optical is calculated using Platt scaling: P(G=i|T) = 1 / (1 + exp(A_i·f_i(T) + B_i)).

[0087] Advantages of PLS-SVM compared to SVM alone: ​​In small sample sizes (10-20 samples per level), PLS-SVM achieves a classification accuracy of 92-95%, while SVM alone only achieves 85-88%, an improvement of approximately 8 percentage points. For intermediate levels (levels 1-3), the recognition accuracy improves by an average of 10%. This is because PLS dimensionality reduction effectively alleviates the overfitting problem caused by high-dimensional, small sample sizes, extracting latent variables highly correlated with aging levels, allowing SVM to establish a more robust classification boundary in a low-dimensional space.

[0088] For the visual modality, visual feature vectors (crack density, average width, maximum length, surface roughness, peeling area ratio, etc.) are input into the SVM classification model, which outputs the visual aging level G_visual and the classification probability P_visual. Since the visual features have low dimensionality (5-10 dimensions), PLS dimensionality reduction is not required, and good results can be obtained directly using SVM.

[0089] The confidence assessment unit calculates the detection confidence of each of the two modalities based on multiple factors, which is the foundation of the intelligent fusion decision-making in this invention. For the optical modality, the calculation of the confidence C_optical comprehensively considers three factors: 1) Signal-to-noise ratio (SNR_optical), quantified by the ratio of signal strength to noise strength; a higher SNR indicates better data quality; 2) Classification probability P_optical, the probability value output by the SVM model; a higher probability indicates a more reliable classification result; 3) Feature consistency Consistency_optical, evaluated by whether the changing trends of multiple feature peak parameters are consistent; higher consistency indicates a more reliable result. The specific calculation formula is as follows: Here, w1, w2, and w3 are weight coefficients, satisfying w1 + w2 + w3 = 1. Similarly, the confidence score C_visual for the visual modality is calculated. The confidence score ranges from 0 to 1; a higher value indicates a more reliable detection result for that modality.

[0090] The adaptive fusion unit is the core of the intelligent fusion decision-making in this embodiment. Unlike traditional fixed-weight fusion (such as simple averaging or fixed-ratio weighting), this embodiment dynamically calculates the fusion weights based on the confidence levels of the two modalities. The specific algorithm is as follows: Thus, when the confidence level of the optical modality is high (e.g., good lighting conditions, high signal-to-noise ratio, clear characteristic peaks), w_optical automatically increases, and the detection results of the optical modality dominate in the fusion; when the confidence level of the visual modality is high (e.g., clear surface features, good image quality), w_visual automatically increases, and the detection results of the visual modality dominate. Then, the overall aging level of the weighted fusion is calculated: This dynamic weighting mechanism can automatically adjust the contribution of the two modes according to different detection conditions, significantly improving the accuracy and robustness of the fusion results.

[0091] The conflict detection unit is used to identify contradictory results between two modalities and to take appropriate strategies. First, the difference between the two modal results is calculated: Then, different processing strategies are adopted according to the degree of difference: (1) When ΔG≤1, it is considered that the results of the two modes are basically consistent, and the fusion result G_fusion is output normally; (2) When ΔG=2, it is considered that there is a moderate degree of conflict, triggering cross-validation of historical data, querying the past detection records of the sample, analyzing whether the historical trend supports the result of a certain mode, and if the historical trend is clear, adjusting the weight to favor that mode; (3) When ΔG≥3, it is considered that there is a serious conflict, triggering a second check, and repeating a complete detection process. If the conflict continues, an uncertain result is output ( The system generates a manual review marker to prompt operators for manual confirmation. This conflict detection and handling mechanism effectively avoids misjudgments and improves system reliability. Finally, the control and fusion decision module 9 packages comprehensive information, including aging level, confidence score, optical modal results (including optical aging level, optical confidence, spectral distribution parameters, and characteristic peak parameters), visual modal results (including visual aging level, visual confidence, and morphological characteristic parameters), fusion weights, environmental parameters, and conflict descriptions, into a detection report. The report is stored in a structured data format (such as JSON or XML) in the data storage unit and simultaneously output to a host computer or cloud server via a communication interface (Ethernet, WiFi, USB, or RS485) for further data analysis, report generation, or remote monitoring. The data storage unit uses embedded memory (such as eMMC or SD card) to record complete data from each detection, supporting historical data queries and trend analysis, providing data support for historical cross-validation of conflict detection.

[0092] Through the above implementation methods, this application realizes the operation and maintenance safety monitoring of curtain wall structural adhesive based on four major innovations: (1) comprehensively characterize the multiple aging mechanisms of structural adhesive through broadband multispectral distribution and characteristic peak synergistic analysis; (2) achieve stable detection under complex environmental conditions through adaptive environmental compensation mechanism; (3) comprehensively evaluate the aging state from the perspectives of internal composition and external morphology through dual-modal parallel detection; and (4) significantly improve the accuracy and reliability of detection results through confidence-based intelligent fusion decision-making.

[0093] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A curtain wall structural adhesive operation and maintenance safety monitoring system based on multispectral-visual morphology dual-modal fusion, characterized in that, include: The optical detection module, including a broadband light source, optical components, a multi-band filter wheel, a photodetector, and an analog-to-digital converter, is used to acquire multi-band reflectance spectral data of structural adhesive samples. The visual inspection module includes an illumination source and an industrial camera, used to acquire surface morphology images of the structural adhesive sample; Environmental sensor components are used to acquire real-time temperature and illuminance data of the detection environment; The control and fusion decision module is communicatively connected to the optical detection module, the visual detection module, and the environmental sensor assembly, respectively. The control and fusion decision module is configured as follows: Based on the data from the environmental sensor components, the multi-band reflectance spectral data is subjected to temperature compensation and illumination normalization processing to eliminate environmental interference. Optical features are extracted from the processed multi-band reflectance spectral data, and visual morphological features are extracted from the surface morphology image. Based on the optical features and the visual morphology features, the optical modal aging level and its confidence level, and the visual modal aging level and its confidence level are calculated respectively. Based on the optical modality confidence and the visual modality confidence, the fusion weights of the two modalities are dynamically calculated; The optical modal aging level and the visual modal aging level are weighted and fused according to the fusion weights to output a comprehensive aging level evaluation result.

2. The monitoring system according to claim 1, characterized in that, The multi-band filter wheel contains at least three filters in different bands, covering a spectral range of 320 nm to 1100 nm.

3. The monitoring system according to claim 1, characterized in that, The control and fusion decision module includes: An environmental compensation unit is used to perform the temperature compensation and illumination normalization processes. The optical feature extraction unit is used to extract broadband spectral distribution curve features and characteristic peak parameters related to chemical bond changes from the compensated spectral data. The visual feature extraction unit is used to extract at least one morphological feature parameter from the surface morphology image, including crack density, surface roughness, and peeling area ratio, using a multi-scale morphological gradient and adaptive region growing algorithm.

4. The monitoring system according to claim 3, characterized in that, The optical feature extraction unit is configured to use partial least squares to reduce the dimensionality of high-dimensional spectral features, and input the reduced features into a support vector machine classification model to obtain the optical modal aging level and classification probability.

5. The monitoring system according to claim 1, characterized in that, The control and fusion decision module is configured to dynamically calculate the fusion weights according to the following formula: in, and These are the fusion weights for the optical and visual modalities, respectively. and These are the calculated confidence scores for the optical and visual modalities, respectively.

6. The monitoring system according to claim 5, characterized in that, The optical modal confidence and visual modality confidence It is calculated based on at least one of the following factors: signal-to-noise ratio, classification probability, and feature consistency of the corresponding modality.

7. The monitoring system according to claim 1, characterized in that, The control and fusion decision module further includes a conflict detection unit, configured as follows: Calculate the difference between the optical modal aging level and the visual modal aging level; When the difference value exceeds the first preset threshold, a historical data cross-validation or secondary inspection process is triggered.

8. A method of operating the monitoring system according to any one of claims 1 to 7, characterized in that, Includes the following steps: Parallel optical and visual inspection: The optical inspection module acquires multi-band reflectance spectral data of the structural adhesive sample, while the visual inspection module acquires surface morphology images of the sample. Environmental data acquisition: Real-time acquisition of ambient temperature and illuminance through environmental sensor components; Environmental adaptive compensation: Based on the ambient temperature and illuminance, the multi-band reflectance spectral data are subjected to temperature compensation and illuminance normalization processing; Feature extraction: Optical features are extracted from the compensated spectral data, and visual morphological features are extracted from the surface morphology image; Confidence assessment: Based on the optical features and visual morphology features, calculate the optical modal aging level and its confidence level, and the visual modal aging level and its confidence level, respectively. Dynamic fusion decision: The fusion weight is dynamically calculated based on the optical modality confidence and the visual modality confidence, and the aging levels of the two modalities are weighted and fused using the weight to output the comprehensive aging level.

9. The working method according to claim 8, characterized in that, The steps for extracting visual shape features include: The surface topography image is processed using a multi-scale morphological gradient algorithm to obtain a multi-scale gradient map, which is then fused. The fused gradient map is segmented based on a local adaptive threshold to obtain candidate crack pixels; An adaptive region growing algorithm is used to grow and verify the candidate crack pixels in order to quantify the crack density.

10. The working method according to claim 8, characterized in that, It also includes conflict resolution steps: Compare the optical modal aging level with the visual modal aging level; If the difference between the two exceeds the preset range, then at least one of the following operations will be performed depending on the degree of difference: enable cross-validation using historical detection data, trigger the system to perform a second detection, or generate a prompt message requiring manual review.

Images