A multi-camera infrastructure and LLM / VLM-based multi-agent system with long-term anomaly detection function and automatic reporting for building health monitoring.

Abstract

The invention relates to a structural health monitoring system that enables the acquisition of simultaneous image data from structures via single or multiple industrial cameras, the generation of contactless physical displacement time series from these images using an optical flow-based method, the identification of anomaly indicators by monitoring the displacement time series over the long term using unsupervised deep learning, the detection of structural damage zones and relative damage severity from field images using VLM-based agents, the cross-validation and data fusion of displacement and anomaly data with visual damage findings within an LLM / VLM-based multimodal agent architecture, and the automatic generation of explainable technical reports containing the risk level, current condition, and recommended interventions of the structure based on the integrated findings obtained.

Description

1 TARIFF MULTIPLE CAMERA INFRASTRUCTURE AND LLM / VLM-BASED MULTIPLE AGENTS INSTRUCTIONS. LONG-TERM ANOMALY DETECTION FOR HEALTH MONITORING PURPOSES FUNCTIONAL AND AUTOMATIC REPORTING SYSTEM TECHNICAL FIELD The invention enables real-time imaging of structures via single or multiple industrial cameras. obtaining image data and extracting these images using optical flow-based methods. generating time series of contactless physical displacement, displacement Anomaly 10 is detected by monitoring time series over the long term using unsupervised deep learning. Identifying indicators, determining structural damage zones and relative areas from field images. Detection of damage severity, displacement, and other parameters through VLM-based agents. Anomaly data and visual damage findings in LLM / VLM-based multimodal agent architecture it undergoes cross-validation and data fusion, and the results obtained Based on integrated findings, the risk level of the structure, its current condition, and the proposed 15 Automatic generation of explainable technical reports including interventions It relates to a building health monitoring system that provides PREVIOUS TECHNIQUE In the field of building health monitoring, 20 are used to determine the behavior of buildings. sensor networks using contact sensors such as accelerometers, LVDTs, and strain gauges It is known that these systems are created. Measurements obtained from structures in these systems. The behavior of the structure is determined by subjecting the data to modal or statistical analyses. Evaluation is provided. However, contact sensors are placed on the structure. installation, creation of necessary cabling and infrastructure and 25 of the said infrastructure Long-term maintenance is required, especially in historical buildings. The limited nature of physical intervention facilitates the implementation of contact sensor infrastructures. It can make things more difficult. Laser displacement for contactless measurement in the known state of the art. sensors, digital image correlation and camera-based vibration or displacement 30 Measurement methods can be used. In camera-based methods, optical flow and properties can be measured. images are captured using methods such as tracking and marker-based tracing. Tracking the movements of designated points or areas and their locations It is possible to obtain the change values. However, the camera 2 where displacement inference based on the experimental model is performed at the single experiment or prototype scale. in applications, within a structure similar to a sensor network of numerous cameras an infrastructure suitable for simultaneous operation and long-term field monitoring The availability of these products may remain limited. In camera-based measurements, lighting changes, camera movements, 5 Deviations from the installation and environmental effects affect the measurement results. It can have an impact. Changes due to drift and jitter in long-term measurements. can accumulate and the reliability of the displacement data obtained It can have an impact. In addition, continuous video recording can result in a high amount of... data is generated and that data represents the behavior of the structure in a meaningful way. the need to convert it into time series and evaluate it over a long period It is emerging. Unsupervised deep learning methods on time series data in the literature. It is also known that anomaly detection can be performed using autoencoder-based methods. learning the normal behavior of the system through approaches and the learned 15 Deviations from behavior identified as indicators of anomaly It is possible. However, obtaining only numerical anomaly scores is not sufficient for detection. In order for the observed change to be correlated with an actual damage area on the structure, This may necessitate the need for expert evaluation. Visual 20 for identifying signs of structural damage from field images. Analytical methods can also be used. However, visual damage findings, anomalies obtained from long-term displacement time series are the same as physical anomalies. It is important to evaluate it within its context. Displacement or anomaly Determining which structural region the data corresponds to, both numerically and visually. It enables the establishment of a relationship between the findings. 25 In current systems, numerical analysis results and visual damage findings are separated. separate evaluation and transformation of the obtained data into an actionable outcome. This can lead to the need for expert opinion at this stage. Different data Checking the consistency of findings obtained from various sources, identifying possible inconsistencies. determination, assessment of the general condition and risk level of the structure and 30 Single or multiple cameras to present the results in a clear and explainable technical report. its infrastructure operates within a sensor network-like architecture, analyzing images and physical data. displacement time series are generated, the generated time series are long damage findings obtained from field images were analyzed in a time-bound manner. 3 the alteration and anomaly data are matched within the same physical context and all words The subject is cross-validation and risk assessment of the findings using multimodal artificial intelligence agents. an integrated building health monitoring system, which is subject to evaluation, as mentioned above It was developed through invention with the aim of remedying the aforementioned shortcomings. THE PURPOSE OF THE INVENTION One aim of the invention is to enable the non-contact monitoring of the behavior of structures. the camera-based measurement infrastructure that provides multiple cameras with channel logic scalable, like a sensor network, that can be managed and collect data simultaneously. The aim is to ensure that it is carried out within a structure. 10 Another purpose of the invention is to improve video quality by industrial cameras. pixels of specific points or regions of interest on the structure being monitored in the images Identifying positional changes at the level and defining those changes. Converting to physical displacement values ​​using a scaling factor The aim is to provide the physical displacement time series of movements on the structure. The aim is to obtain it in this way. Another objective of the invention is to analyze displacement obtained from a multi-camera infrastructure. long-term evaluation of time series and deep learning-based The aim is to enable the learning of the normal behavior of the structure using this approach. Within this scope, deviations from normal behavior are rated as anomaly scores or 20 Identifying potential dysregulation indicators and long-term behavior The aim is to monitor the changes. Another objective of the invention is to analyze visual data from site images of the structure. evaluation by artificial intelligence agents and on the images signs of structural damage found, the areas where such damage is located, and 25 The aim is to determine their relative intensities. Another objective of the invention is to physically position target tracking points on the structure. anomalies obtained from time series analysis by enabling the identification of their counterparts The visual damage findings obtained from field images and indicators are of the same structure. The aim is to enable its association through the region. Target tracking within the scope of the invention 30 point, image-based on the structure via industrial cameras (1) via the tracked and optical flow-based displacement calculation module (8) It refers to the physical point for which a displacement time series is created. These target tracking points are located without a physical sensor on the structure. 4 Because they generate measurement data, they function as camera-based virtual sensors. in the scope of multimodal agent-based artificial intelligence architecture (9) and related flowchart The term 'sensor' used refers to a camera-based virtual sensor, not a separate physical sensor element. This refers to the target tracking points that function as sensors. Another aim of the invention is to identify anomalies 5 obtained from numerical analysis. Damage findings obtained from visual analysis together with indicators evaluation, checking the consistency between the findings, and The aim is to identify potential discrepancies. This allows for the analysis of data obtained from different sources. creating a multi-evidence assessment using the evidence gathered together is intended. 10 Another purpose of the invention is time series analysis, visual analysis, and target tracking. obtained from point-to-building area matching and cross-validation processes the evaluation of the results within a multimodal agent-based artificial intelligence architecture and the overall condition of the structure, its usability, risk level, and the level of intervention required. The aim is to ensure that the relevant results are formed. 15 Another aim of the invention is to provide explanatory results based on the analyses performed. Automatic generation of visualizations, risk maps, and final technical reports. The aim is to ensure that the measurement, analysis, interpretation, and decision-making stages are completed. Instead of being carried out as independent processes, a modular and integrated approach It is intended to be realized within the architectural framework. 20 Another objective of the invention is to improve camera network management by addressing displacement and anomalies. data, visual damage findings, multi-agent outputs, and final reports all in one package. The aim is to enable monitoring through the user interface. In this context, the system structure... suitable for long-term health monitoring procedures and operator The aim is to create a platform that can be followed by. 25 LIST OF FIGURES Figure 1. Industrial cameras (1), camera of the building health monitoring system subject to the invention. lenses (2), panoramic tripod heads (3), profile clamps (4), tripods (5), digital This is a general perspective view showing the video recorder (6) and the steel profile pipe (7). 30 Figure 2. Industrial camera (1), camera lens (2) and panoramic tripod head (3) This is the first perspective view where the two parts are shown separated. Figure 3. Industrial camera (1), camera lens (2) and panoramic tripod head (3) This is the second perspective view where the two parts are shown separated. Figure 4. Perspective view of the digital video recorder (6). Figure 5. Perspective view of the profile clamp (4). Figure 6. Perspective view of the tripod (5). Figure 7. Multi-camera arrangement of the structural health monitoring system, subject to the invention, on the carrier. General perspective shown together with its structure and digital video recorder (6) 5 It is the appearance. Figure 8. Reference by optical flow-based displacement calculation module (8). pixel coordinates of the video frame at the time step related to the video frame comparison, pixel displacement determination and scaling factor Flowchart 10 shows the steps involved in converting the process into physical displacement. It is a diagram. Figure 9. Time series analysis, field image analysis, measurement point–structure area. multimodal system including matching, cross-validation and risk assessment modules This is the flowchart of the agent-based artificial intelligence architecture (9). The corresponding numbers in the figures are: 1. Industrial camera 2. Camera lens 3. Panoramic tripod head 4. Profile clamp 20 5. Tripod 6. Digital video recorder 7. Steel profile pipe 8. Optical flow-based displacement calculation module 9. Multimodal agent-based artificial intelligence architecture 25 DETAILED DESCRIPTION OF THE INVENTION The invention describes a building health monitoring system that obtains image data from the building. industrial cameras (1) that enable the use of these cameras (1) camera lenses (2) used, cameras (1) in line with the structure to be monitored 30 Panoramic tripod heads (3) that enable the positioning of the system's carrier profile clamps (4), tripods (5) and steel profile used in the creation of the structure pipe (7), video data obtained from cameras (1) simultaneously Digital video recorder (6) that enables recording and storage of recorded video. 6 Optical flow that enables the extraction of displacement values ​​from video data. the displacement calculation module based on (8) and numerical and visual data together multimodal agent-based artificial intelligence architecture that enables evaluation (9) It includes a system containing software. The element defined as the system here is the software. It refers to any electronic device capable of being driven. 5 Industrial cameras (1) enable high-speed image acquisition and on the structure Determining movements invisible to the naked eye through image data. Industrial cameras (1) are provided together with compatible camera lenses (2). Focus and brightness adjustments are made manually via the camera lenses (2). It is carried out as follows. 10 Each industrial camera (1) can be screwed onto the panoramic tripod head (3). It is mounted. The panoramic tripod head (3) is mounted on the structure to be monitored by the relevant camera (1). It ensures that the area is directed and aligned to the desired location. Panoramic tripod heads (3) are mounted on a high rigidity steel profile pipe (7) It is fixed. Thanks to this structure, multiple industrial cameras (1) can be monitored. During this process, it is ensured that it is positioned on the supporting structure that is aligned with it. Steel profile pipe (7) can be attached to adjustable tripods (5) via profile clamps (4) It is connected. This structure restricts the movement of the system and the camera. This ensures that the structural system is less affected by ground irregularities. Tripods (5) can carry the steel profile pipe (7) and connect it to the ground. It enables its creation. Each of the industrial cameras (1) records digital video via cables. The digital video recorder (6) is connected to the device (6). The digital video recorder (6) receives video from the cameras (1). It enables the simultaneous recording and storage of digital data. The video recorder (6) is powered by electrical energy using an adapter. Digital 25 The video recorder (6) is also connected to a computer via cable. Video images obtained from the cameras (1) can be accessed through the device's application. It can be viewed simultaneously and the technical settings of the application in question. This can be done through this method. The recorded videos can be used in later analyses. The digital video recording device (6) is transferred to the computer for use. 30 In the structure where multiple industrial cameras (1) are used together cameras (1) can be managed with channel logic and image data simultaneously This is achieved. Thus, the camera-based measurement infrastructure provides independent, individual measurements. Instead of individual display units, it operates as a multi-camera network. Multi-camera 7 Images obtained from the infrastructure show the displacement of different measurement points. It is used in the creation of information. The system described in the invention is a sub-software unit of optical flow-based displacement. video obtained using calculation module (8), industrial cameras (1) 5. Non-contact determination of the dynamic behavior of structures from their images. This method provides the ability to point to the same physical location in different video frames. Determining the corresponding pixel coordinates and the occurrences between the coordinates It is based on converting the incoming change into a physical displacement value. As part of the optical flow-based displacement calculation module (8), firstly A reference video frame is selected from the video recording of the structure being monitored. The 10 in question... reference video frame, for comparisons to be made during viewing time It is based on. Video frames found later in the video recording are used as a basis. The subject is analyzed by comparing it to a reference video frame. The structure being observed. Pixel coordinates are assigned to predefined regions of interest. Pixel coordinates (xₚ,₁, 15) corresponding to the same physical point in the reference video frame. In the video frame at time step i, examined with yₚ,₁), the physical point in question is identified. The corresponding pixel coordinates (xₚ,ᵢ, yₚ,ᵢ) are determined using the optical flow method. This is calculated. Thus, the reference video frame of the same physical point is compared with the i-th time frame. The positional change between video frames at each step is determined. Pixel coordinates of the reference video frame and video at time step i. The difference between the pixel coordinates of the square determines the pixel-level location. The displacement value is being calculated. This displacement is at the pixel level; (Xₚ,ᵢ, Yₚ,ᵢ) = (xₚ,ᵢ, yₚ,ᵢ) − (xₚ,₁, yₚ,₁) It is expressed by the equality. Here (Xₚ,ᵢ, Yₚ,ᵢ) is the reference point of the relevant physical point. It represents the pixel-level displacement at the i-th time step according to the video frame. This process involves different time points evaluated in the video recording. By applying this method, displacement values ​​are obtained at the pixel level. Displacement values ​​calculated at the pixel level are physical displacements. Scaling factor (SPF) defined within the system in converting values It is used. The scaling factor (SF) is the size of the pixels in the image plane. 8 It expresses the physical equivalent in the real world and is measured in millimeters / pixel or similar units. It is defined in units. Accordingly, the physical displacement at time step i. value; (Xᵢ, Yᵢ) = ÖF × (Xₚ,ᵢ, Yₚ,ᵢ) 5 It is calculated using the equality. Here, (Xᵢ, Yᵢ) is the i-th point of the relevant physical point. It represents the physical displacement value in a time step. The process in question... The physical location of the structure at the relevant measurement point is determined by performing this for different time steps. The change values ​​are generated as a time series. The resulting value is 10. Change time series are used to monitor the dynamic behavior of a structure, and to identify anomalies. by detection and multimodal agent-based artificial intelligence architecture (9) to be used in the analyses to be carried out in the next stages of the system is being transferred. Multimodal agent-based artificial intelligence architecture (9), time series data, field 15 their locations according to their tasks, along with images and target tracking point information. joint evaluation through specialized artificial intelligence agents It provides. The central conducting orchestrator within the architecture, different It takes data from the sources and determines the type of data and the analysis to be performed accordingly. It redirects to the relevant modules. 20 In the first module of the multimodal agent-based artificial intelligence architecture (9), the structure The resulting displacement or similar time series data is analyzed. An AI agent structured for time series analysis, target tracking. a deep learning-based approach that considers the relationships between points It uses. The artificial intelligence agent in question learns the normal behavior of the structure. 25 and deviations from learned normal behavior as indicators of anomaly. This assessment determines potential irregularities in the structure. or anomaly scores that may be indicative of damage are generated. Time series in the analysis, the displacement time series of neighboring measurement points These relationships are evaluated within the framework of the Relation-AE approach, and these relationships are 30 Evolutionary autoencoders are used in the analysis. Through evolutionary autoencoders... The relational structure that represents normal behavior is identified, and the components that arise from this structure... Deviations are generated as anomaly scores. 9 Location tracking created from a network of multiple cameras as part of long-term behavioral monitoring. Transformation time series using an unsupervised deep learning approach Normal behavior is being evaluated through an autoencoder-based approach. a model is being created and consists of the aforementioned normal behavioral model Deviations are identified. Thus, without the need for labeled damage data, 5 anomalies of behavioral changes that occur during long-term monitoring These indicators are used to ensure they are identified. In the second module of the multimodal agent-based artificial intelligence architecture (9), the structure belongs to field images are visually analyzed by an artificial intelligence agent. is being evaluated. The agent in question can be configured based on VLM. 10 The AI ​​agent identifies signs of structural damage in field images. It determines, identifies the area where the damage is located, and assesses the relative severity of the damage. This process creates visual information about the physical condition of the structure. Findings are being produced. Target 15 in the third module of the multimodal agent-based artificial intelligence architecture (9) This ensures the determination of the locations of the monitoring points on the structure. The AI ​​agent working for this purpose captured images of the placement of target tracking points. or evaluating descriptive information and determining which of those points are part of the structure It determines that it is related to the region. Thus, what is determined in time series analysis 20 Matching can be done between visual findings. In the fourth module of the multimodal agent-based artificial intelligence architecture (9), the first, Cross-validation and data fusion of data obtained from the second and third modules. It is subjected to processing. The fourth module includes an LLM-based agent, Relation-AE analysis summaries created in the first module and 25 created in the second module VLM-based visual findings are evaluated together. Thus, time series data... relational anomalies identified through and the corresponding anomalies Visual findings are correlated and visual evidence regarding anomalies is provided. Anomaly indicators, visual damage findings, and target tracking points are being created. The matching information between the building area and the building zone is included together in the module in question. 30 is being evaluated. The artificial intelligence agent processes information from different data sources. evaluating their consistency, as determined through time series data. to determine whether anomalies are supported by visual findings and It identifies potential discrepancies between different data sources. Thus a multi-evidence assessment that uses both quantitative and visual findings is being created. Cross-modal agent-based artificial intelligence architecture (9) fifth module Integrated findings generated from the verification process are used for risk assessment and decision-making. It is being analyzed for the purpose of production. The LLM-based artificial intelligence system working in this module is 5. The intelligence agent assesses the overall condition of the structure, its usability, and the required level of intervention. It evaluates and creates a structural risk assessment. As a result of the analysis... This can be explained in terms of the current condition of the structure, the level of risk, and the proposed interventions. and traceable outputs are generated. The LLM-based executive orchestrator derives 10 from displacement and anomaly analyses. Damage area and relative severity obtained from numerical outputs and visual analyses. It directs the information between the relevant modules and integrates the data in question. This allows for the evaluation of both numerical and visual evidence. evaluating the relationship between them, identifying consistencies and contradictions. It is possible to mark them. 15 Automatic damage assessment based on evaluations performed by the system. Damage assessment reports and automatic status reports can be generated. Automatic damage assessment. the report regarding the identified damage area, relative severity of damage and visual evidence It includes information. The automated status report, on the other hand, includes monitoring results and identified anomalies. These reports include the incidents, risk levels, and recommended actions. These reports are from 20 It can be created in a structured format that includes tables, text, and explanatory reasoning. and can be converted to PDF or similar file output. The results of the analyses performed by the system are presented with explanatory visuals and risk assessments. The data is automatically generated in the form of maps and final technical reports. Camera Network information, physical displacement time series, anomaly indicators, visual 25 Damage findings, multimodal agent analyses, and generated reports are all available through a single user interface. It can be monitored through this system. With this structure, the process from image acquisition to physical location is monitored. from long-term anomaly analysis to visual damage to the creation of replacement value from identification, cross-validation of different evidence, to risk assessment and final The processes leading up to report generation are integrated within the system in 30 steps. is being carried out.

Claims

11 REQUESTS 1. A building health monitoring system for contactless monitoring of structures. Its characteristic is; - Multiple 5s that enable the acquisition of image data belonging to the monitored structure. industrial camera (1), - image obtained by at least one industrial camera (1) a system that can run software to process its data, - within the system in question, a reference video is selected from the image data. Selecting a square represents 10 predefined regions of interest on the monitored structure. Determining pixel coordinates, pixels in the reference video frame pixel of the video frame in the time step related to its coordinates calculating the coordinates of the pixel in question using the optical flow method Determining the difference between the coordinates, the resulting pixel-level location Change values ​​using a defined scaling factor for physical location 15 converting into replacement values ​​and the physical displacement in question to create displacement time series from the values structured optical flow-based displacement calculation module (8), - time series data, field images, and location data of measurement points. to collect the information and perform the relevant analysis according to the data type and analysis needs. 20 central orchestrator directs the modules, obtained from the structure the displacement time series the relationships between the measurement points to analyze using a deep learning-based approach that takes into account, to learn the normal behavior of the structure and from that normal behavior The deviations that occur are referred to as anomaly indicators and anomaly scores. 25 The first module aims to determine structural features by analyzing field images. signs of damage, areas of damage, and relative severity of the damage. The second module is designed to determine the information regarding the placement of measurement points. by analyzing the images or descriptive information of each measurement point to determine which region of the structure it corresponds to and time series 30 The anomalies obtained from the analysis and the visual findings are the same as the physical The third module, anomaly, is designed to ensure matching within the context. indicators, visual damage findings, and measurement point-structure area. evaluating matching information together, from different data sources 12 Analyzing the consistency of information, identifying issues in time series data. whether the detected anomalies are supported by visual findings and whether different to identify potential discrepancies between data sources and the data in question fourth to perform cross-validation and data fusion on it. The module analyzes the integrated findings obtained from the cross-validation process. 5 by assessing the overall condition of the building, its usability, and any necessary interventions. to determine the level, to conduct a structural risk assessment and the structure regarding the current situation, risk level and proposed interventions Module five and location to generate explainable and traceable outputs. Numerical outputs obtained from change and anomaly analyses and visual 10 Damage area and relative severity information obtained from the analyses are transferred to the relevant modules. to route between them, the data in question in an integrated manner to enable evaluation of the differences between numerical and visual evidence to identify and evaluate the consistencies and contradictions based on the assessment results. 15 a multimodal agent containing an LLM-based executive orchestrator to provide It is characterized by having an artificial intelligence architecture based on (9). 25

Images