Monitoring of environmental and structural conditions

Abstract

A system and method for monitoring environmental and structural conditions provides a sensor attached to a structure and a camera that automatically acquires and transmits images in response to data obtained from the sensor. [Solution] Another embodiment of the system and method according to the present invention includes an inertial measuring unit and a camera. One embodiment of the system and method according to the present invention employs a programmable controller that receives data signals from an inclinometer, accelerometer, magnetometer, vibration sensor, torsion sensor, wind sensor, temperature sensor, humidity sensor, methane sensor, airborne particle sensor and / or impact force sensor, and based on the data signals, automatically determines whether a digital image (e.g., a video clip) is needed, and if it is determined that one is needed, automatically activates a camera near the sensor and sends an alert to a remote receiver along with the sensor data and camera image.

Description

Detailed Description of the Invention

[0001] 〔Background and Summary〕 This application generally relates to monitoring environmental and structural conditions, and more particularly to monitoring environmental and structural conditions including inertial measurement units and cameras.

[0002] It is known to monitor vegetation contact with power lines and wildfires near power lines using cameras. Such a system is described in U.S. Patent Publication No. 2021 / 0073692, titled "Method and System for Utility Infrastructure Condition Monitoring, Detection and Response". U.S. Patent Publication No. 2021 / 0073692 was invented by Chinmoy Saha, Jeffrey Pickles, and Kourosh Jafari Khouzani, is owned by the same assignee as this application, and is incorporated herein by reference. Although this conventional system represents an important advance in the art, further improvements are desired to remotely detect other aspects of structures that are often difficult to access.

[0003] Conventional devices use accelerometers installed on power lines to detect their movement. This is disclosed in U.S. Patent Publication 2014 / 0136140 by Chan et al., titled "Systems and Methods for Detecting Overhead Line Motion," filed on 15 May 2014. U.S. Patent Publication 2014 / 0136140 is incorporated herein by reference. However, this conventional system only uses a camera to acquire and map the position of nearby objects. Attaching accelerometers and associated hardware directly to movable power lines has the disadvantage of adding weight and further oscillating momentum to the power lines. Thus, Chan's device has a very limited range of sensing targets and does not teach any automatic camera control or automatic image manipulation based on sensing data.

[0004] The present invention provides a system and method for monitoring environmental and structural conditions, comprising a sensor attached to a structure and a camera that automatically acquires and transmits images in response to data obtained from the sensor. Another embodiment of the system and method according to the present invention includes an inertial measuring unit and a camera. One embodiment of the system and method according to the present invention employs a programmable controller that receives data signals from an inclinometer, accelerometer, magnetometer, vibration sensor, torsion sensor, wind sensor, temperature sensor, humidity sensor, methane sensor, airborne particle sensor and / or impact force sensor, and automatically determines whether a digital image (e.g., a video clip) is needed based on the data signal, and if it is determined that one is needed, it automatically activates a camera near the sensor and sends an alert to a remote receiver along with the sensor data and camera image. A further embodiment employs a remote central controller and / or software commands to receive sensor data from structures such as high-voltage transmission tower infrastructure structures, transmission line support infrastructure structures, wind turbine infrastructure structures, bridge infrastructure structures, road infrastructure structures, rail infrastructure structures, liquid or gas storage tank structures, building structures, ship structures, and aircraft structures. Another embodiment of the system and method according to the present invention detects and senses undesirable abnormal characteristics of a structure in real time compared to previously detected and sensed normal characteristics of the structure, automatically determines whether a potentially hazardous condition exists on or near the structure, then automatically acquires and transmits images (e.g., video clips) from a camera on the structure or a camera near the structure, and sends a warning alert to the user along with the sensed abnormal data and associated images. In a further embodiment, the system employs sensors and cameras to determine whether undesirable corrosion is present on a structure, and if so, transmits sensor data and associated images to a remote user.

[0005] This system offers advantages over conventional devices. For example, it automatically senses and monitors real-time movement characteristics of hard-to-access mechanical structures or their surrounding areas, and transmits sensor data, along with camera images of the sensed area, to one or more remote control centers or users. This advantageously improves the safety, maintenance, and operation of structures before failure or more serious hazards occur. Furthermore, the system and method according to this application provide a precautionary measure to avoid wildfires, structural collapse, damage caused by surface irregularities, and / or reduce structural stress, strain, bending, twisting, scoliosis, leaning, tremors, flutter, aeroelastic buffeting, galloping, and other failure modes or collapses by advantageously sensing and determining potential hazards and problems in or near the structure being sensed and alerting the user. The system and method relating to this application enable the taking of preventative measures to repair structures after earthquakes, mudflows, liquefaction, subsidence, erosion, rock landslides, corrosion, impact forces, wind, floods, fires, etc., by advantageously sensing and determining potential hazards and problems in or near a structure to be sensed and issuing alerts to the user. The system is also advantageous in that it integrates multiple sensors and camera images with at least one electrical system and at least one controller, receives sensing data, performs automatic determination, and reports information quickly and proactively to a control center and remote users. Additional advantages and features of the system will become apparent from the following description and the attached claims, which will be referenced in conjunction with the attached drawings.

[0006] [Brief explanation of the drawing] Figure 1 is a side elevation view of the environmental and structural condition monitoring system according to the present invention.

[0007] Figure 2 is an electrical diagram of the IMU of this system.

[0008] Figure 3 is an electrical diagram related to the camera in this system.

[0009] Figure 4 is a schematic diagram of the IMU of this system used on a power transmission tower.

[0010] Figure 5 is a schematic diagram of the camera used in this system on power line poles and power line towers.

[0011] Figure 6 is a perspective view of the system used on roads and bridges.

[0012] Figure 7 is a perspective view of this system used on railway tracks.

[0013] Figure 8 is a perspective view of this system used on a wind turbine.

[0014] Figure 9 is a perspective view of the system used on a fluid storage tank.

[0015] Figure 10 is a schematic diagram of this system used on buildings and communication towers.

[0016] Figure 11 is a perspective view of the system used on a ship.

[0017] Figure 12 is a perspective view of this system as it is used on an aircraft.

[0018] Figure 13 is a partially exploded and partially fractured perspective view showing a camera and anemometer of an alternative embodiment in this system.

[0019] Figures 14 to 16 are software flow diagrams related to this system.

[0020] Figure 17 shows a camera image displayed on the graphical user interface of this system, which targets surface corrosion on power transmission towers.

[0021] Figure 18 is a camera image displayed on a graphical user interface related to the present system, targeting surface corrosion on the metal riser of a transmission line support pillar.

[0022] Figure 19A is a camera image displayed on a graphical user interface related to the present system, targeting surface corrosion on the base of a utility pole.

[0023] Figure 19B shows a camera image displayed on a graphical user interface related to the present system, targeting surface corrosion on the cross arm of a transmission line support pillar.

[0024] Figure 20 shows a sensor position map displayed on a graphical user interface related to the present system.

[0025] Figure 21 shows a sensor list displayed on a graphical user interface related to the present system.

[0026] Figure 22 is a camera image matched with camera image data displayed on a graphical user interface related to the present system.

[0027] Figures 23 to 25 show inclinometer graphs of IMU data displayed on a graphical user interface related to the present system.

[0028] Figure 26 shows a camera image and related GUI displaying ground erosion and subsidence in the vicinity of a structure adopting the present system.

[0029] Figure 27 shows a camera image and related GUI displaying lateral bending deformation of a structure adopting the present system.

[0030] Figure 28 shows a camera image and related GUI displaying a rock slide colliding with a structure adopting the present system.

[0031] Figure 29 is a schematic diagram of the methane sensor used in conjunction with this system.

[0032] [Detailed explanation] A preferred embodiment of the environmental and structural condition monitoring system 31 shown in Figures 1 to 5 includes an inertial measurement unit ("IMU") 33 and at least one camera 35. A solar panel 37 is connected to and supplies electricity to the IMU 33 and camera 35 by wires of an electrical circuit 39. The IMU 33, camera 35, and solar panel 37 are mounted by one or more brackets 41 to a stationary structure such as a utility pole 43 or a high-voltage distribution tower 44. Both have one or more cross arms 45 extending laterally to which power lines 47 are attached.

[0033] In the configuration shown in Figure 1, the camera 35 is separated from and spaced apart from the housing 49 of the IMU 33. An electric actuator operates the gimbal 51, which moves the camera in the pan and tilt directions. The camera also includes an electromagnetically operated zoom lens or two or more neighboring cameras with different preset focal lengths. At least one programmable microprocessor controller 53 is located within the IMU housing 49 and automatically controls the operation of the gimbal and associated actuators, causing the camera 35 to automatically acquire (capture) digital images and / or digital video of the target area. The controller 53 may be located on-site at the structure being monitored or may be cloud-based.

[0034] In the alternative configuration shown in Figure 13, multiple cameras 61 and 63 are each fixedly coupled to the IMU housing 49. Each camera includes a lens cover 65, multiple stacked lenses 67, an image sensor 69, an optional DTI, an image processor 73, and an optional image stabilizer. Each camera is preset to a different focal length and / or field of view. This embodiment also includes a photovoltaic panel 37, an electrical circuit 39, a programmable controller 53, an antenna 103 inside the housing 49, and a battery 107, all directly mounted to the housing 49. An anemometer (i.e., wind sensor) 98 is coupled to the outside of the housing 49.

[0035] Referring to Figures 2 and 3, the components of the IMU 33 include, in addition to the controller 53, one or more sensors 101, an antenna 103, a router 105, a battery 107, a battery fuse / circuit breaker 109, a solar power fuse / circuit breaker 111, a USB-C port 113, an Ethernet port 115, a LAN connection 117, a 12-volt junction 119, and an electrical grounding junction 121. All of these are located within a self-contained housing 49. These IMU components are electrically connected to each other by an electrical circuit 39, and are also electrically connected to the solar power panel 37 and the camera 35. In an alternative configuration, the IMU sensor 101 may be located separately from the controller 53 and the antenna 103: for example, multiple sensors may be mounted on the upper part of a tower or column structure, with a single controller and antenna connected to those sensors located at the base of the structure.

[0036] In one configuration, the camera 35 shares the same IMU antenna 103, router 105, battery 107, fuse / circuit breaker 109 and fuse / circuit breaker 111, junction 119 and junction 121, in addition to the shared controller 53 and solar panel 37. However, in the alternative configuration shown in Figure 3, the camera 35 has its own dedicated electrical circuit 123, which includes the antenna 125, router 127, battery 129, battery fuse / circuit breaker 131, solar fuse / circuit breaker 133, data port 135, PoE port 137, LAN connection 139, 12-volt junction 141, electrical ground junction 143, PoE injector 145, and solar panel 147.

[0037] The IMU sensor 101 includes an inclinometer and optionally, but preferably, an accelerometer and a magnetometer. Each IMU detects acceleration, angular velocity, 3-axis angle, and magnetic field. An exemplary inclinometer, accelerometer, and magnetometer unit is available from Wit Motion Shenzhen Co. as model HWT905 (RS485), but other sensors may be used.

[0038] Additionally or alternatively, the IMU sensor may include dedicated vibration sensors, torsion sensors, anemometers (wind speed sensors), thermometers (temperature sensors), hygrometers (humidity sensors), methane sensors, airborne particulate sensors, and / or impact force sensors. However, tiltmeters, accelerometers, and / or magnetometers can detect many of these behaviors (phenomena), including vibration, torsion, and impact force. An exemplary weather observation unit is available from Hinovision Solutions LLC (dba Linovision USA) as model IOT-S300WS7 or model IOT-S300WS8, but other sensors may be used. This exemplary weather sensor senses temperature, humidity, atmospheric pressure, wind speed, and wind direction, and optionally senses light and precipitation.

[0039] By using particle counting sensors to monitor suspended particles and transmitting this data to a controller, the controller can automatically determine whether an undesirable or abnormal amount has been detected. If so, the controller will have (one or more) cameras acquire images of the potential source of the particles and send the matched data and images to the end user. Sources may include wildfires and associated smoke, industrial fires, smog and pollution, and volcanic ash. (One or more) wind direction and speed sensors provide real-time data to the controller, and its internal software predicts particle movement patterns and concentrations. Examples of affected end users include farmers downwind, hospitals and elderly care facilities, city crisis management personnel, and airports.

[0040] As shown in Figures 4 and 5, the IMU housing 49 and its bracket 41 are preferably mounted on the side of the structural leg of the tower 44, on the side of the utility pole 43, or on a cross arm 45 attached thereto. The mounting position of the IMU sensor is determined by structural analysis of the specific infrastructure structure, taking into account various parameters such as the type of host structure, major failure modes, stiffness, geographical location, topography, climatic conditions, natural frequency, and environmental conditions. Furthermore, the IMU can optionally be mounted on opposite sides of an elongated structure. This allows the controller to automatically determine whether undesirable torsional or lateral deformation is occurring in the monitored beam by comparing IMU data sensed from these opposite locations.

[0041] The camera unit 35 is installed either on the same host infrastructure structure to which (one or more) IMUs 33 are mounted, or at a different nearby location, and is connected to the same cloud system. An example of a nearby location is a configuration in which the camera 35 is mounted on a separate, dedicated, stationary pole or building structure next to the power transmission tower 44 or utility pole 43 to which the IMU 101 is mounted. Installing (one or more) cameras in a nearby location while still being at a distance from them has the advantage of providing a view of the part of the structure to which the IMU is mounted. This view may be partially obstructed if the cameras are not installed in this manner, when they are mounted on the IMU housing, or when they are placed inside the IMU housing. As an alternative configuration, one camera is directly mounted on the IMU housing, and at least a second camera is installed in a nearby location while still at a distance from the IMU, resulting in a configuration in which different views are obtained. It is also envisioned that one or more cameras may acquire and transmit images of the surrounding landscape or vegetation (e.g., trees 102) near the columns, towers, or power lines attached thereto. This camera view of the surrounding area is ideally suited to indicating the reason for anomalies in the IMU sensor output data and to showing environmental behavior near or adjacent to the structure (e.g., mudflows, impacts on the tower 44 by rolling rocks 160 (see Figure 28), vegetation impacts, floods, erosion and settlement of the ground 162 near the foundation 227 of the tower 44 or other structure, forest fires and associated smoke, etc.). In contrast, a camera view of the structure directly is ideally suited to showing torsional, bending, and / or lateral deformation 164 of the tower 44 or other structure (see Figure 27). A camera view of the structure directly is also well suited to showing surface corrosion, deformation, direct impacts and contacts, damage, etc., with respect to the structure being monitored.

[0042] In summary, this system automatically detects and identifies normal and abnormal conditions such as motion, deterioration, and collapse, and sends alerts. The IMU sensor and pan-tilt-zoom camera are connected via a wired or wireless network. Furthermore, the IMU's magnetometer, accelerometer, and inclinometer detect the heading, angle, and acceleration of the structural portion to which they are attached, acquiring sensing data. The associated controller and its software then analyze this sensing data using an optimized set of algorithms to detect mechanical failure modes (e.g., bending, torsion, scoliosis, leaning, shaking, flutter, aeroelastic buffeting, galloping, other similar phenomena, or collapse) individually or in combination with other sensor data. As a result, the controller automatically triggers and activates one or more nearby cameras, focusing on specific target locations on the structure or specific target locations in the vicinity of the structure, and uses computer vision software and associated algorithms to verify one or more malfunction modes detected by the IMU. One or more cameras verify the environment, such as the ground or vegetation in the vicinity of the structure, for potential causes of malfunction modes, such as corrosion, subsidence, erosion, mudflows, rock landslides, floods, wildfires, wildfire smoke, or contact with foreign objects or impact forces. Similarly, if one or more cameras detect any anomalies, the controller and its software automatically acquire the corresponding IMU data and verify the visual detection results. The IMU and cameras may operate in conjunction or, to a lesser degree, independently.

[0043] End users receive these AI-driven alerts on their mobile and / or desktop devices via the internet or other communication connections. These alerts are used to help users make decisions regarding corrective actions or to automatically schedule maintenance or repair work.

[0044] Camera images and / or video are streamed to remote cloud storage 161 and processed using computer vision software. As a result, the IMU or the programmable controller 163 of the central control station can automatically detect objects of interest in the host structure or objects of interest in the vicinity of the host structure, and conditions related to the structural state. A matching algorithm for specific detections is executed by the IMU or the programmable controller of the central control station, and the conditions detected by the IMU are matched with the conditions detected by the camera's computer vision. The programmable controller of the IMU or the programmable controller of the central control station automatically sends an alert regarding the sensed conditions. This alert, along with the corresponding IMU sensor data and (one or more) camera images, and an automatically determined confidence level, is sent to a portable, handheld personal digital assistant display 165 for remote users. Such remote users include maintenance and repair technicians, and / or emergency response personnel such as fire department personnel.

[0045] Figure 6 shows IMUs 33 and (one or more) cameras 35 coupled to road structures 171 and bridge structures 173. For road 171, one or more IMUs 33 are mounted on the side edge of the road, and one or more associated cameras 35 are each mounted on a nearby but spaced-away upright pole 174, or on an elevated spanning beam 179, which can also hold traffic signs. For bridge 173, one or more IMUs 33 are mounted on a pylon 177, crossbeam 179, suspension cable 181, road surface, or subsupport structure, while the cameras 35 may be coupled to any of these same structural components, or to nearby but spaced-away upright poles. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing, or it may be located at a single location on the structure and used for all sensors and cameras.

[0046] Figure 7 shows IMUs 33 and (one or more) cameras 35 coupled to the railway track 191 and associated railway bridge structures. One or more IMUs 33 are mounted on the side of each rail 191, on the upright beams 193, crossbeams 195, or on the sub-support structure 197. The cameras 35 may be coupled to any of these same structural components, or to nearby upright columns at a distance. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing, or it may be located at a single position on the structure and used for all sensors and cameras.

[0047] Figure 8 shows an IMU 33 and (one or more) cameras 35 coupled to a wind turbine structure 201. The wind turbine structure 201 has a generator 203, rotating blades 205, and an upright, stationary tower 207, which are made of metallic and / or composite materials. One or more IMUs 33 are coupled to the side of the tower 207, the blades 205, or the housing of the generator 203. The cameras 35 may be coupled to any of these same structural components, or to an upright column located nearby but at a distance. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing, or it may be located at a single location on the structure and used for all sensors and cameras.

[0048] Figure 9 shows an IMU 33 and (one or more) cameras 35 coupled to a metal, stationary fluid storage tank structure 211, such as a fuel storage tank or chemical storage tank. One or more IMUs 33 are coupled to the side of the tank 211 or to brackets extending therefrom. The cameras 35, on the other hand, may be coupled to any of these same structural components, or to a nearby, albeit spaced-out, upright column. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing, or it may be located at a single position on the structure and used for all sensors and cameras.

[0049] Figure 10 shows an IMU 33 and (one or more) cameras 35 coupled to a stationary building structure 221, such as a high-rise office building, residential building, or industrial building. One or more IMUs 33 are mounted on a steel girder 223, a communication tower or communication mast 225, or a foundation 227. The cameras 35 may be coupled to any of these same structural components, or to a nearby upright column at a distance. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing, or it may be located at a single location on the structure and used for all sensors and cameras.

[0050] The IMU and controller are configured to detect and determine whether there are potentially hazardous environmental impacts on a bridge by, for example, automatically comparing real-time sensing data values ​​related to vibration or movement due to undesirable abnormal winds via extreme harmonic frequencies of the bridge with previously detected or predetermined thresholds for normal bridge harmonic frequencies. In another example, the IMU and controller can detect and determine whether there are potentially hazardous environmental impacts on a bridge by, for example, automatically comparing real-time sensing data values ​​related to vibration or movement due to water flow from undesirable abnormal floods, impacts from ships, or impacts from rocks or driftwood on the base of a pylon with previously detected or predetermined thresholds for normal base and pylon vibration or movement. In yet another example, the IMU and controller may detect and determine whether there are potentially hazardous environmental impacts on a road or rail by, for example, automatically comparing real-time sensing data values ​​related to vibrations from rocks or vehicles on the road surface or rail due to undesirable abnormal impacts with previously detected or predetermined thresholds for normal road or rail vibration. In yet another example, the IMU and controller can sense and determine whether there are potentially hazardous environmental impacts on a road or rail by automatically comparing real-time sensing data values ​​for torsion or bending of a road, bridge, or rail due to, for example, undesirable abnormal temperatures, with previously sensed or predetermined thresholds for normal road, bridge, or rail movement. In yet another example, the IMU and controller can be used to sense and determine whether there are potentially hazardous environmental impacts on a wind turbine tower, communication tower, tank, or girder or foundation of a building by automatically comparing real-time sensing data values ​​for vibration, torsion, or bending of a structure due to, for example, undesirable abnormal earthquake or ground subsidence, with previously sensed or predetermined thresholds for normal structure movement.

[0051] If the controller determines that real-time sensing data values ​​exceed a nominal safety threshold, the controller automatically activates one or more cameras to acquire and transmit one or more images of the road, bridge, or rail and / or surrounding area. These images are then matched and annotated as part of an alert or warning message (for example, by using a frame superimposed on the camera image to outline a target area of ​​interest, such as impact location, bending location, or fracture location). These matched and annotated IMU data and camera images, as well as the alert message, are received by (a) a programmable controller in a remote central control station, which displays the data and images as a graphical user interface on a display monitor connected to the central controller, and / or by one or more remote portable personal digital assistant displays, such as mobile phones.

[0052] Figure 11 shows an IMU 33 and (one or more) cameras 35 coupled to a vessel 241. One or more IMUs 33 are coupled to the mast or uprights 243 at the bow, the outer surface of the bridge wing 245, the outer and upper surfaces of the vessel's superstructure 247, and / or other internal or external locations on the vessel. Cameras 35, on the other hand, may be coupled to any of these same structural components. The IMUs 33 and cameras 35 are connected to a programmable controller 53. The programmable controller 53 may be located inside each IMU housing or in a single location in the vessel's control room and may be used for all sensors and cameras. Each IMU and its associated controller automatically detects and determines, after excluding nominal safe vibration, motion, and shock values, whether the vessel is experiencing abnormal and undesirable vibration, motion, or shock, as perceived in real time, such as due to undesirable movement of cargo such as containers, excessive waves on the hull, wind vibrations, or collisions with docks or floating objects. If the IMU and controller determine that such a potentially dangerous condition has occurred, the controller automatically activates the camera to acquire images of the target area of ​​the vessel detected by the IMU, annotates these images, matches them with the detection data, and then transmits these images / data to the central controller 53 located in the vessel's control room and / or remotely.

[0053] Referring to Figure 12, the IMU 33 and camera 35 are coupled to the front of the upright tail fin 251 on an aircraft, for example, plane 253. The IMU 33 and camera 35 can also be mounted on the upper or side of the fuselage 255 of plane 253 without compromising the aerodynamic properties of those surfaces. By this arrangement, if the IMU and the programmable controller 53 installed in the cockpit determine that abnormal and undesirable vibration, motion, or shock characteristics have been detected, the pilot and / or ground maintenance personnel can acquire data from the IMU sensors on the ground or in flight, and also visually inspect the aircraft surface, which is normally difficult to access. Each IMU and its associated controller automatically detects and determines, after excluding nominal safe vibration, motion, and shock values, whether abnormal and undesirable vibration, motion, or shock is occurring on the aircraft, such as due to undesirable cargo movement, excessive turbulence, bird strikes, jet turbine vibrations, or collisions with the passenger walkway, as perceived in real time. If the IMU and controller determine that such a potentially dangerous condition has occurred, the controller automatically activates the camera to acquire images of the target area of ​​the aircraft detected by the IMU, annotates these images, matches them with the detected data, and then transmits these images / data to the central controller 53 located in the cockpit and / or remotely.

[0054] Referring to Figures 14A, 14B, 15, and 16, one aspect of the system and method according to the present invention employs a programmable controller that receives data signals from an inclinometer, accelerometer, magnetometer, vibration sensor, torsion sensor, wind sensor, temperature sensor, humidity sensor, methane sensor, suspended particle sensor, and / or impact force sensor, and automatically determines whether a digital image is required based on the data signal, and if it is determined to be required, automatically activates one or more cameras in the vicinity of (one or more) sensors to acquire images (annotated by the controller's software), and then sends an alert to a remote receiver along with the sensor data and the annotated camera images. A further embodiment employs a remote central controller and / or software commands that receive sensor data from structures such as high-voltage transmission tower infrastructure structures, transmission line support infrastructure structures, wind turbine infrastructure structures, bridge infrastructure structures, road infrastructure structures, rail infrastructure structures, liquid or gaseous fluid storage tank structures, building structures, ship structures, and aircraft structures, and automatically determines, based on the data signals, whether digital images are needed, and if so, automatically activates one or more cameras in the vicinity of (one or more) sensors to acquire images (annotated by the controller's software), and then sends an alert to the remote receiver along with the sensor data and annotated camera images.

[0055] Another embodiment of the system and method relating to the present invention detects and senses undesirable abnormal characteristics of a structure in real time compared to previously detected and sensed normal characteristics of the structure, automatically determines whether a potentially hazardous condition exists on or near the structure, automatically determines, based on the data signal, whether a digital image is required, and if so, automatically activates one or more cameras near (one or more) sensors to acquire images (annotated by the controller software), and then transmits an alert to a remote receiver along with the sensor data and annotated camera images. In a further embodiment, the system employs a sensor to determine whether undesirable corrosion is present on a structure, automatically determines, based on the data signal, whether a digital image is required, and if so, automatically activates one or more cameras near (one or more) sensors to acquire images (annotated by the controller software), and then transmits an alert to a remote receiver along with the sensor data and annotated camera images.

[0056] More specifically, the electrical circuit connected to the programmable computer controller in the structure being monitored includes an internal RAM memory or internal ROM memory connected to a microprocessor. Software instructions stored in the memory and operated by the microprocessor receive digital signals transmitted by the IMU and other sensors. The controller automatically compares the actually sensed data signals in near real-time with previously sensed or predetermined nominal safety thresholds stored in the memory. The controller then automatically determines whether any undesirable sensed characteristics or conditions exist. If an undesirable condition exists, the controller automatically operates the camera in pan / zoom / tilt modes in a preferred mobile embodiment, or, if an alternative stationary embodiment is used, causes the camera to utilize various camera lens focal lengths. The controller then automatically compares the image acquired based on the anomaly data with the previously stored nominal images and highlights the areas of concern (e.g., by enclosing the areas of concern with an annotated frame and / or adding positional or sensor data to the image). Next, the controller automatically matches the annotated image with sensing data indicating a potentially hazardous condition and sends a signal, such as a text message, email, or warning message, to a computer controller at a remote central control station and / or to a handheld or remote mobile phone, pager, or other portable communication device and / or display on a portable computer carried by a field technician user. This message may warn of an urgent and hazardous situation and may optionally automatically include corrective actions such as scheduling on-site inspection, maintenance, or replacement of the structure or surrounding area. This may include felling nearby trees, adding soil, rocks, or buttresses to eroded ground, or draining floodwaters.

[0057] Referring to Figures 14A and 14B, the typical software instructions and process steps are as follows: A. Generate real-time data of the IMU's angle / acceleration / azimuth (heading) from the sensor. Each IMU and (one or more) associated cameras may be installed separately or as part of the same pre-assembled unit, but in either case, they will work together. B. The IMU transmits the data it senses as a signal to the CPU controller via edge computing and / or cloud computing. C. The controller is used to determine whether the signal value (greater than or equal to 1) for angular motion exceeds a threshold (greater than or equal to 1). D. Using the controller, determine whether the signal value (greater than or equal to 1) for linear motion exceeds the threshold (greater than or equal to 1). E. The controller is used to determine whether the signal value (greater than or equal to 1) for gravity (g) exceeds a threshold (greater than or equal to 1). F. Using the controller, determine whether the signal value (greater than or equal to 1) for rotational motion exceeds a threshold (greater than or equal to 1). G. If the controller's determination result is positive ("YES"), the controller automatically activates the camera and causes it to pan / tilt / zoom in focus, targeting the location of real-time sensing data that exceeds a safe and desirable value (1 or more). H. One or more cameras scan one or more target regions of interest, acquire digital images (which may be still images or videos), and transmit image data signals to the CPU or the cloud. I. Using artificial intelligence computer vision analysis, determine from image data signals whether undesirable conditions exist in the monitored infrastructure structure and / or surrounding environment. Details will be described later. J. If the controller's determination result is positive ("YES"), the controller collects camera image data of the monitored infrastructure structure and / or surrounding environment area, annotates the image, links and matches the image with the sensed IMU data, and then transmits the image data. The controller sends an alert or warning message to the controller at the central control station and / or one or more remote users via edge or cloud computing and communication. K. As part of steps I and / or J before sending an alert, or after sending an alert, the controller determines whether the undesirable condition of the monitored infrastructure structure and / or surrounding environmental area matches the detected IMU signal determination result. If the controller's determination result is positive ("YES"), the controller automatically ranks the reliability of the detected and determined hazards and sends this to (one or more) remote users as part of the alert transmission for process J, or as a subsequent secondary alert. M. Subsequently, the controller allows manual override operation of (one or more) cameras, enabling the end user to change the field of view or focal length. N. The field CPU controller or the controller at the central control station can optionally and automatically save image clips and data from (one or more) cameras and IMUs, respectively. O. The on-site CPU controller or the controller at the central control station optionally communicates with first responders or crisis management personnel to schedule corrective actions, repair actions, or preventive actions manually or automatically. If a negative "NO" result is obtained, the software returns to the normal IMU sensing operation mode and / or proceeds to another determination process.

[0058] Furthermore, the general method processes for software instructions and control logic can also be described as follows: A. IMU sensing data related to angle, acceleration, and / or orientation are acquired continuously at a certain frequency. B. The sensing data is sent to the cloud and automated software analysis is performed. C. Sensing data is automatically analyzed in the cloud for anomalies related to angular motion, linear motion, rotational motion, or changes in gravity (g) (e.g., exceeding or deviating from a previously sensed or pre-stored nominal desired threshold). D. If any parameter exceeds a threshold, an alert is automatically sent via the internet or other communication channel to the end user's mobile computer, desktop computer, or personal digital assistant (PDA). These alerts may be displayed on the output display screen. E. A trigger is automatically sent to the camera to perform pan, tilt, and zoom ("PTZ") operations towards the IMU position that exceeds the parameter threshold. F. The PTZ of the programmable camera automatically scans the region of interest. G. Computer vision algorithms running on the cloud controller or on-site controller are programmed to focus on collecting detailed information when they detect any abnormal conditions within the monitored structure or its surrounding environment. Cameras located near the H.IMU, or forming part of the IMU, collect images of their respective locations and use computer vision algorithms (described later) to visually confirm the visual state of the host structure. I. If the computer vision system detects an anomaly indicated by the IMU, the controller automatically ranks that anomaly as having a higher reliability in hazard detection. J. Subsequently, the cloud controller automatically sends alerts about the hazardous situation to the end user's device via the internet. These alerts are text messages and emails accompanied by images and text descriptions of the hazard. K. End users can search for hazardous events by manually controlling the camera's PTZ (Post-to-Zero) position using a front-end software application to perform a detailed visual analysis of the situation. L. End users, such as initial responders or crisis management personnel, may make decisions regarding corrective actions and / or response measures, and save video clips of the incident. Furthermore, with respect to all software and method steps disclosed herein, the following should be understood: the order of some steps may be changed, additional steps may be added, and some steps may be omitted, depending on the specific structure being monitored, the number of IMUs and cameras at the site of the structure, the availability of a central control room, and other optional factors.

[0059] The operation of the IMU control algorithm and artificial intelligence software is described as follows: The preferred IMU at present is a nine-axis inclinometer that reports (outputs) chip temperature, chip time, and magnetic field (relative to its position on Earth). The nine-axis inclinometer measures angle (in degrees) (relative to the x, y, and z axes), angular velocity (in degrees / second) (relative to the x, y, and z axes), and acceleration (in G (1G = 9.8 m / s²) (relative to the x, y, and z axes). 2 It reports the following and is called a "nine-axis" inclinometer because it provides three measurements (indications) for three measurement items. The inclinometer is connected to the controller using an RS485 USB converter to transmit (T+ / T-) and receive (R+ / R-) signals (also known as A+ and B- in the RS485 communication protocol).

[0060] Data from the inclinometer is obtained as follows: 1) First, the controller sends a data transmission command (also called a data request) to the slave IMU. The lower and upper bytes of the CRC are appended to this transmission command to match the sensor's request format. 2) After receiving a data transmission request, the IMU transmits data once for each request. 3) The data is then received as hexadecimal data packets by Python code running on the controller. This is then converted into engineering units such as angle, acceleration, magnetic field measurements, and angular velocity. These data packets contain data from various registers of the sensor, each responsible for data corresponding to a specific measurement item. For example, register 0x30 is the first register. Registers 0x30 through 0x33 are responsible for reporting the chip time in HH:MM:SS:MSMS format and the date in (DD:MM:YY) format. Registers 0x34 through 0x36 are responsible for reporting acceleration in the X, Y, and Z axes. The Python code is responsible for all data conversion processing, including complex binary adjustments and numerous conditional branch settings. 4) A total of three main codes are executed to acquire raw data from the inclinometer, convert it to engineering units, and export it in a suitable structure for further analysis. The first code sends a data request to the IMU and outputs the data in common engineering units such as degrees, degrees / second, and G. The second code is responsible for converting the data request to the sensor's adapted format. The second code also acquires data from the sensor and converts it to integer-based data for use in display, analysis, and output by the first code. The last code is used to manage the threads that drive the above two codes in parallel, so that the acquisition and output processes can be performed simultaneously without any time loss.

[0061] This section describes the basic mathematical and parameter transformations performed within the code. When data measurements are received from the sensor, they are in hexadecimal format. Converting them from hexadecimal to integer format requires three steps: first, removing the most significant and least significant bits of the CRC (which act as tags indicating the start and end of sequences in long hexadecimal strings); then, performing bitwise operations after the conversion to derive the value; and finally, obtaining very large numbers such as 32445 or 55422. These numbers need to be converted into meaningful measurements.

[0062] The mathematical transformation is performed as follows: - Mathematical transformation of angles: Final Angle = Received Angle / 32768 * 180 (repeat for all x, y, and z coordinates) - Mathematical transformation of acceleration: Final Acceleration = Received Acceleration / 32768 * 16 (repeated for all x, y, and z coordinates) - Mathematical transformation of angular velocity: Final Angular Velocity = Final Angular Velocity / 32768 * 2000 (repeat for all x, y, and z coordinates) -After all these conversion processes, the values ​​are sent to the first code and made available for human manipulation and use.

[0063] The mathematical transformations for predicting tilt, drift, and scoliosis are as follows: Assume the sensor is in a geographical NS directional configuration and is connected flat, i.e., horizontally, to one of the tower's girders. Then, a value of +x indicates east, a value of -x indicates west, a value of +y indicates north, a value of -y indicates south, a value of +z indicates counterclockwise movement, and a value of -z indicates clockwise movement.

[0064] Lean: To calculate the lean in either the x or y direction, calculate the change in the + / -x or + / -y value. The change is calculated as follows: Δ (delta, change) = current (x / y) value - original (x / y) value The meaning of the sign of Delta is as follows: a) + Δ(X): The value of + Δ indicates the slope of "Δ" degrees in the geographical eastward direction. b) -Δ(X): The value of -Δ indicates the slope of "Δ" degrees in the geographical westward direction. c)+Δ(Y): The value of +Δ indicates the slope of "Δ" degrees in the geographical northward direction. d) -Δ(Y): The value of -Δ indicates the slope of "Δ" degrees in the geographical southward direction.

[0065] Drift: To calculate drift or footing, the following eight directions must be considered: N (North), NE (Northeast), E (East), SE (Southeast), S (South), SW (Southwest), W (West), NW (Northwest). In addition to these directions, acceleration spikes in the + / -X or + / -Y axis should also be considered. Since current IMUs do not report linear velocity, the only way to detect / report the occurrence of force or motion is through acceleration spikes. The calculation is performed as follows: Step 1: Spikes are detected along the + / -X axis, + / -Y axis, or both in the case of diagonal drift. Record them. Step 2: Calculate the synthesis: The combined acceleration of "a" is = √(a_x^2 + a_y^2) (where a_x and a_y are the deltas (changes) of the acceleration from the initial / inertial position). Given the composition of "a", the total acceleration spike in a given direction can be determined. Furthermore, the angle can be calculated with respect to x based on the following equation: θ = arctan(a_y / a_x) (Theta is used in calculating azimuth angles) The drift and its direction can be determined using these two values.

[0066] Scoliosis: Scoliosis is a phenomenon in which a tower twists around its own z-axis. This can be measured by the change in the Z-axis angle measurement returned from the IMU. The calculation is performed as follows: Δ (delta, change) = current angle (z) value - initial angle (z) value The meaning of the sign of delta is as follows: a) + Δ(Z): The value of + Δ indicates a rotation of "Δ" degrees clockwise around the central axis, representing a positive twist. b) -Δ(Z): The value of -Δ indicates a rotation of "Δ" degrees counterclockwise around the central axis, representing a negative twist. It should be noted that angular velocity is incorporated into the measurement of lean / tilt and scoliosis, at the expense of computational efficiency.

[0067] Vibration: Vibration can be measured by changes in acceleration in the X, Y, or Z axis direction. Calculations are performed based on the number of spikes per second in X, Y, or Z, as follows: V(Hz) = number of spikes / second Here, the term "spike" refers to a significant acceleration change that exceeds a threshold, and minor fluctuations may not represent a true oscillation event. Natural frequency: Often determined by the speed of vibration when a structure undergoes free vibration after a disturbance. Current IMUs do not directly calculate natural frequencies, but they can be estimated based on periodic vibrations in terms of angle or acceleration.

[0068] Detailed software and controller processes using algorithms include: (A) System initialization: Configure the cloud and / or central controller as master devices and the IMU controller as a slave device. Connect the IMU using an RS485-USB converter. (B) Main program flow: 1. Sending a data request: - The controller sends a data request command to the IMU. -To make the command conform to the IMU's request format, the lower and upper bytes of the CRC are appended to the beginning and end. 2. IMU response: - The IMU receives the data request. - The IMU sends data as a response once for each request. 3. Receiving and processing data on the controller: Receives a data packet in hexadecimal format. - Converts data packets into engineering units. 4. Analysis of data packets (parse): - Initialize `tempReg` to 0x30 (start register). - Set a variable (e.g., NRegs) that indicates the number of registers based on the length of the data packet. - Initialize `tempVals` as an empty array to temporarily store values. - For each register in the data packet (loop processing through `NRegs`). -Calculate `tempIndex` based on the register position (as a counter). - Converts 2 bytes in `tempIndex` to a single value `tempVal` - Check the register address (`tempReg`) and process the data based on the register range. -**Registers 0x30~0x33** (Chip Time and Date): -Append `tempVal` to `tempVals` -If `tempReg` is 0x33 (the last register of the chip time): Extract the time and date from the chip: `_year`=2000+(tempVals[0]&0xff) `_month`=(tempVals[0]>>8)&0xff `_day`=tempVals[1]&0xff `_hour`=(tempVals[1]>>8)&0xff `_minute`=tempVals[2]&0xff `_second`=(tempVals[2]>>8)&0xff `_millisecond`=tempVals[3] Format "Chiptime" and set it in `deviceModel`. Clear `tempVals` -**Registers 0x34~0x36** (Acceleration X, Y, Z): Calculate `tempVal` as `tempVal / 32768.0*self.accRange`. If `tempVal>=self.accRange`, adjust it to match the signed range. Add the calculated acceleration values ​​to `tempVals`. -If `tempReg` is 0x36 (the last register of acceleration): Use `tempVals` to set “accX”, “accY”, and “accZ” to `deviceModel`. Clear `tempVals` -**Register 0x40**(Temperature): Calculate `temperature` as `tempVal / 100.0` and round to two decimal places. Set “temperature” to `deviceModel` -**Registers 0x37~0x39** (Gyroscope X, Y, Z): Calculate `tempVal` as `tempVal / 32768.0*self.gyroRange`. If `tempVal>=self.gyroRange`, adjust it to match the signed range. Add the calculated gyroscope values ​​to `tempVals`. -If `tempReg` is 0x39 (the last register of the gyroscope): Use `tempVals` to set “gyroX”, “gyroY”, and “gyroZ” to `deviceModel`. Clear `tempVals` -**Register 0x3D~0x3F**(Angle X, Y, Z): Calculate `tempVal` as `tempVal / 32768.0*self.angleRange`. If `tempVal>=self.angleRange`, adjust to match the signed range. Add the calculated angle values ​​to `tempVals`. -If `tempReg` is 0x3F (the last register of angles): Use `tempVals` to set “angleX”, “angleY”, and “angleZ” to `deviceModel`. Clear `tempVals` -**Register 0x3A~0x3C**(Magnetometer X,Y,Z): Round `tempVal` to an integer and add it to `tempVals`. -If `tempReg` is 0x3C (the final register of the magnetometer): Use `tempVals` to set “magX”, “magY”, and “magZ” to `deviceModel`. Clear `tempVals` Increment `tempReg` to point to the next register. 5. Converting data to engineering units: -Regarding each parameter (angle, acceleration, angular velocity): Remove the upper and lower bytes of the CRC. Convert hexadecimal data to an integer value. - Apply a parameter-based mathematical transformation: Angle: Final_Angle = Received_Angle / 32768 * 180 (for x, y, and z) Acceleration: Final_Acceleration = Received_Acceleration / 32768 * 16 (for x, y, z) Angular velocity: Final_Angular_Velocity (Final angular velocity) = Received_Angular_Velocity (Received angular velocity) / 32768*2000 (about x, y, z) 6. Processing of data output: -The data received from the protocol resolver or the second code can be formatted and used to create a rolling plot for better insights. - The data read from TempVals about the registers is stored in a dictionary (dictionary structure) for later use. -The data can be used to perform further analyses such as: Lean calculation: The lean in either the X or Y direction is calculated by monitoring the change in the + / -X or + / -Y values. -Δ (change) is calculated as follows: delta_x = current_x - original_x delta_y = current_y - original_y Interpret the sign of -Δ and trigger an alert if the change exceeds a threshold (e.g., 5 degrees): Set lean_threshold to 5 - Defines the threshold for slope alerts. -If delta_x is greater than the slope threshold: Display "Alert: Tilt to the east by delta_x degrees" - Otherwise, if delta_x is less than the negative slope threshold: Display "Alert: The map is tilted westward by an absolute value of delta x." -When delta y is greater than the slope threshold: Display "Alert: The plane is tilted north by delta y degrees". - Otherwise, if delta y is less than the negative slope threshold: Display "Alert: The area is tilted southward by an absolute value of delta y." - Drift calculation: Set the drift threshold to 2.0 - This sets the threshold for drift alerts because wind-induced movement and noise are always present. The resultant of a is calculated as the square root of (a_x^2 + a_y^2). The calculation is performed in degrees, using theta as the arctangent to (a_y / a_x). - If the composition of "a" is greater than the drift threshold: If theta is between 0 and 22.5 degrees, or between 337.5 and 360 degrees: Set the direction to "East". -In other cases, if theta is between 22.5 degrees and 67.5 degrees: Set the direction to "Northeast". -In other cases, if theta is between 67.5 degrees and 112.5 degrees: Set the direction to "North". -In other cases, if theta is between 112.5 degrees and 157.5 degrees: Set the direction to "Northwest". -In other cases, if theta is between 157.5 degrees and 202.5 degrees: Set the direction to "West". -In other cases, if theta is between 202.5 degrees and 247.5 degrees: Set the direction to "southwest". -In other cases, if theta is between 247.5 degrees and 292.5 degrees: Set the direction to "South". -In other cases, if theta is between 292.5 degrees and 337.5 degrees: Set the direction to "southeast". - "Alert: Acceleration a_resultant m / s 2 Display "Detected drift in the [direction] direction." - "Alert: Acceleration a_resultant m / s 2 Display "Detected drift in the [direction] direction." -Vibration calculation: Set the vibration threshold to 0.1 - Taking noise into consideration, determine the vibration frequency threshold per unit Hz. - If the vibration frequency (number of spikes per second) is greater than the vibration threshold: Display "Alert: High vibration detected at vibration frequency _Hz". 7. End the program (if necessary) - If necessary, send the final data to the database or storage. - Create a graph and output word-based alerts using the conditional statements in the submitted dictionary. -End the thread and perform cleanup.

[0069] The software instructions and control logic method steps for displaying a graphical user interface (GUI) for a central control center and / or remote PDA are shown in Figures 15 and 20-25 and are described below: A. Identify / enter your target area of ​​interest. B. In the central control station 405, a map 401 with IMU / camera monitoring positions 403 is virtually displayed (see Figure 20). C. A virtual chart of the IMU / camera monitoring position coordinates 407 and status 409 is displayed on the sensor list view (see Figure 21). D. Automatically detect abnormalities in the IMU sensor at one or more monitoring locations. E. For data anomalies detected at one or more monitoring locations (e.g., wildfire 413), images 411 from one or more cameras are automatically acquired and annotated by overlaying a surrounding frame 415 with relevant text (see Figure 22). F. Data and images from the sensor are matched with (one or more) monitoring locations and automatically transmitted to the central controller. G. Automatically determine and transmit the confidence level and date / time of detected data and hazardous events (see Figure 22). H. A chart matching the judgment result with the image is virtually displayed (see Figure 22). I. Virtually display graphs / charts of IMU data sensed from inclinometers / accelerometers / magnetometers / other sensors (see Figure 23 for an overview, Figure 24 for detailed data at specific times, and Figure 25 for additional angular velocity and magnetic field parameters). J. Alerts are automatically sent to the mobile PDA. K. Automatically activate countermeasures and / or corrective actions. These may optionally appear on task lists, work orders, and / or calendars.

[0070] The method process for software instructions and control logic for automated corrosion detection and reporting is shown in Figures 16 to 19B and is described as follows: A. Identify / enter your target area of ​​interest. A map containing the B.IMU / camera monitoring locations is virtually displayed. A virtual chart showing the coordinates and status of the C.IMU / camera monitoring location is displayed. D. Automatically and periodically detect normal safety values ​​at the monitoring location and set safe or nominal (one or more) thresholds (e.g., normal and acceptable wind speed, structural harmonic vibration of sound and corrosion-free structural parts, acceptable minor impact force, temperature, acceptable contraction and expansion motion). E. Automatically detect and determine whether there are any abnormalities in IMU sensor data at (one or more) monitoring locations by correcting based on previously detected normal values. For example, determine whether (one or more) new sensor data values ​​are outside the range of (one or more) desired safety thresholds (e.g., whether the natural harmonic frequency vibration of the structure has changed due to corrosion). F. Images from one or more cameras at one or more monitoring locations where real-time sensing hazard data is measured are automatically acquired, and the images are annotated to identify which parts of the structure contain corrosion and to visually observe the severity of the corrosion. Exemplary camera images transmitted and displayed show annotated frames 441 surrounding corrosion 443 on a metal electric tower cross arm 45 (see Figures 17 and 19B), annotated frames 441 surrounding corrosion 443 on a metal riser 445 on a utility pole 43 (see Figure 18), and annotated frames 441 surrounding corrosion 443 at the base of a metal utility pole 43 (see Figure 19A). Data and images from the G.IMU sensor are matched with (one or more) monitoring locations and automatically transmitted to the central controller and / or remote user PDA. H. The system automatically determines the reliability / severity and date / time of detected data and hazardous events, and transmits the determination results to the central controller and / or remote user PDAs. I. A chart is virtually displayed that matches the judgment result with the image. J. Virtually displays graphs / charts of IMU data detected by inclinometers / accelerometers / magnetometers / other sensors. K. Automatically send alerts to your mobile PDA.

[0071] Referring to Figure 29, an optional methane sensor 473 is coupled to the IMU housing and IMU electrical circuit to monitor methane emissions 471 in the vicinity of the structure in real time. Methane is a very potent greenhouse gas and is primarily released from naturally occurring biochemical processes such as swamps, wetlands, and cattle stomachs. This system can measure naturally occurring and anthropogenic methane emissions by installing stationary methane sensors near these swamps, wetlands, and dairy / livestock farms, as well as natural gas pipelines and tanks. This includes sensing the presence and quantity of both carbon-12 (12C) and carbon-13 (13C).

[0072] The methane sensor 473 operates in conjunction with the on-site camera 35 and is automatically controlled by the on-site controller and its software commands (as in previous examples). This allows the on-site controller to distinguish between naturally occurring and anthropogenic methane emissions. The PTZ camera 35, which is automatically activated based on real-time sensing data and the controller's judgment, transmits remote visual confirmation information of the (natural or anthropogenic) methane source and on-site emission detection to the central / cloud controller 163 and / or remote user PDA 165. This offers significant advantages over conventional methods that require laboratory analysis. Furthermore, the system can advantageously monitor abnormal increases in anthropogenic methane emissions, such as those caused by pipeline and tank failures and other mechanical malfunctions resulting from natural disasters.

[0073] While various features of the present invention are disclosed, it should be understood that other modifications may be employed. For example, different or additional electronic components may be employed in the electrical circuits of the present application. However, various advantages of the present system may not be realized. As another example, it is also possible to monitor alternative stationary structures. However, certain advantages may not be obtained. Furthermore, alternative sensor configurations and installation locations may be employed. However, these may not be as advantageous as the preferred embodiment in terms of durability, performance, and cost. Furthermore, additional or different GUI displays and the data contained therein may be used in the present system. However, some advantages may not be obtained. While field IMU controllers and single central / cloud controllers have been described, this functionality can also be divided among multiple controllers. Features of each embodiment and usage may be interchangeable with each other or substituted with similar features of other embodiments, and all claims may be multiplely dependent on each other in any combination. Modifications shall not be considered departures from the present disclosure, and all such modifications shall be included within the scope and spirit of the present invention. [Brief explanation of the drawing]

[0074] [Figure 1] This is a side elevation view of the environmental and structural condition monitoring system related to the present invention. [Figure 2] This is an electrical diagram of the IMU (Infrared Measure Unit) of this system. [Figure 3] This is an electrical diagram related to the camera in this system. [Figure 4] This is a schematic diagram of the IMU (Infrared Measurement Unit) of this system used on power transmission towers. [Figure 5] This is a schematic diagram of the camera used in this system on power line support poles and power line towers. [Figure 6] This is a perspective view of the system used on roads and bridges. [Figure 7] This is a perspective view of the system used on railway tracks. [Figure 8]This is a perspective view of the system used on a wind turbine. [Figure 9] This is a perspective view of the system used on a fluid storage tank. [Figure 10] This is a schematic diagram of the system used on buildings and communication towers. [Figure 11] This is a perspective view of the system used on a ship. [Figure 12] This is a perspective view of the system used on an airplane. [Figure 13] This is a partially exploded and partially fractured perspective view showing a camera and anemometer of an alternative embodiment in this system. [Figure 14A] This is a software flow diagram for this system. [Figure 14B] This is a software flow diagram for this system. [Figure 15] This is a software flow diagram for this system. [Figure 16] This is a software flow diagram for this system. [Figure 17] This is a camera image displayed on the graphical user interface of this system, which targets surface corrosion on power transmission towers. [Figure 18] This is a camera image displayed on the graphical user interface of this system, which targets surface corrosion on the metal risers of power transmission line poles. [Figure 19A] This is a camera image displayed on the graphical user interface of this system, which targets surface corrosion on the base of utility poles. [Figure 19B] This image shows a camera view displayed on the graphical user interface of this system, which targets surface corrosion on the cross arms of power line support poles. [Figure 20] This shows a sensor location map displayed on the graphical user interface for this system. [Figure 21]This shows the sensor list displayed on the graphical user interface for this system. [Figure 22] This is a camera image matched with camera image data displayed on the graphical user interface of this system. [Figure 23] This shows an inclinometer graph of IMU data displayed on the graphical user interface of this system. [Figure 24] This shows an inclinometer graph of IMU data displayed on the graphical user interface of this system. [Figure 25] This shows an inclinometer graph of IMU data displayed on the graphical user interface of this system. [Figure 26] This document shows camera images and related GUIs displaying ground erosion and subsidence in the vicinity of structures employing this system. [Figure 27] This shows camera images and related GUIs displaying the lateral deformation of a structure using this system. [Figure 28] This image shows camera footage and a related GUI displaying rock sliding impacts on a structure employing this system. [Figure 29] This is a schematic diagram of the methane sensor used in conjunction with this system.

Claims

1. A method for monitoring structures, (a) A step of sensing characteristics related to the structure using at least one sensor; (b) A step of automatically determining whether the sensing data obtained by the sensor exceeds a threshold using at least one programmable controller; (c) If the controller determines that the sensing data exceeds the threshold, it automatically activates at least one camera on the structure or at least one camera in the vicinity of the structure to generate an image of at least a portion of the structure or an area in the vicinity of the structure; (d) The controller automatically transmits at least a portion of the sensing data and the corresponding camera image to a remote receiver based on step (c); and (e) A step in which the controller automatically transmits the position of the sensor to the remote receiver based on step (c). Methods that include...

2. The aforementioned characteristics indicate corrosion. The aforementioned sensing data is related to the harmonic frequency of the structure. The method according to claim 1, wherein the threshold is related to the harmonic frequency of the structure in a non-corrosive state.

3. At least one of the sensors includes an inertial measuring sensor, The method according to claim 2, wherein the controller automatically transmits corrosion alerts to the remote receiver, which includes a handheld and portable digital display, and to a stationary central control station having a display.

4. Solar power generation panels; Battery; antenna; The electrical circuit connected to the solar power generation panel, the battery, and the antenna. It further includes, The camera is connected to the electrical circuit, At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, At least one of the cameras is mounted in the housing, The housing is attached to the structure, The inertial measurement unit and at least one of the cameras are connected to the solar panel, the battery, and the electrical circuit. The method according to any one of claims 1 to 3, wherein at least one camera has a plurality of focal lengths such that a plurality of images having different magnifications are generated at various locations of the structure in the vicinity of the inertial measurement unit.

5. The first solar power generation panel; First battery; First antenna; A first electrical circuit connected to the first solar panel, the first battery, and the first antenna; Second solar power generation panel; Second battery; The second antenna; A second electrical circuit connected to the second solar panel, the second battery, and the second antenna. It further includes, At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the structure, At least one of the cameras is positioned in the vicinity of the housing, but at a distance from it. The method according to any one of claims 1 to 3, wherein at least one camera has a plurality of focal lengths such that a plurality of images having different magnifications are generated at various locations of the structure in the vicinity of the inertial measurement unit.

6. The aforementioned structure is a high-voltage distribution tower. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the tower, At least one of the cameras is mounted on the tower, The method according to any one of claims 1 to 3, wherein the sensor senses at least one of the following characteristics: corrosion of the tower or power line, mechanical collapse of the tower support, damage due to irregularity of the tower surface, mechanical stress on the tower or power line, mechanical strain on the tower or power line, bending of the tower or power line, twisting of the tower or power line, lateral deformation of the tower or power line, or inclination of the tower or power line.

7. The aforementioned structure is a high-voltage distribution tower. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the tower, At least one of the cameras is mounted on the tower, The method according to claim 1, wherein the sensor senses at least one of the following characteristics: a wildfire near the tower or the power line, smoke near the tower or the power line, an impact force on the tower or the power line, shaking or vibration of the tower or the power line, fluttering of the tower or the power line, aeroelastic buffeting of the tower or the power line, or galloping of the tower or the power line.

8. The aforementioned structure is a stationary pole, and a power transmission line is installed near the pole. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the column, At least one of the cameras is mounted on the column, The method according to any one of claims 1 to 3, wherein the sensor senses at least one of the following characteristics: corrosion of a pole or power line, mechanical collapse of a pole, damage due to surface irregularity of a pole, mechanical stress on a pole or power line, mechanical strain on a pole or power line, bending of a pole or power line, twisting of a pole or power line, lateral deformation of a pole or power line, or inclination of a pole or power line.

9. The aforementioned structure is a stationary pole, and a power transmission line is installed near the pole. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the column, At least one of the cameras is mounted on the column, The method according to claim 1, wherein the sensor senses at least one of the following characteristics: a wildfire near the pole or the power line, smoke near the pole or the power line, shaking or vibration of the pole or the power line, impact force on the pole or the power line, fluttering of the pole or the power line, aeroelastic buffeting of the pole or the power line, or galloping of the pole or the power line.

10. The aforementioned structure is a bridge, a road, or a railway track. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the structure, At least one of the cameras is mounted on a stationary bracket located on or near the structure, The method according to any one of claims 1 to 3, wherein the sensor senses at least one of the following characteristics: corrosion of the structure, mechanical collapse of the structure, damage due to surface irregularity of the structure, mechanical stress of the structure, mechanical strain of the structure, bending of the structure, torsion of the structure, lateral deformation of the structure, tilt of the structure, tremor or vibration of the structure, and impact force on the structure.

11. The aforementioned structure is a wind turbine, a stationary fluid storage tank, a stationary high-rise building, a stationary high-rise structure, or a stationary communication tower. At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the structure, At least one of the cameras is mounted on or near the structure, The method according to any one of claims 1 to 3, wherein the sensor senses at least one of the following characteristics: corrosion of a structure, mechanical collapse of a structure, damage due to surface irregularity of a structure, mechanical stress of a structure, mechanical strain of a structure, bending of a structure, torsion of a structure, lateral deformation of a structure, tilt of a structure, tremor or vibration of a structure, impact force on a structure, wildfire occurring outside the structure in the vicinity of the structure, or smoke occurring outside the structure in the vicinity of the structure.

12. The aforementioned structure is a ship or an aircraft, At least one of the sensors is part of an inertial measurement unit that includes an inclinometer and an accelerometer mounted within a housing, The housing is attached to the structure, At least one of the cameras is mounted on the outer surface of the structure, The method according to any one of claims 1 to 3, wherein the sensor senses at least one of the following characteristics: corrosion of the structure, mechanical collapse of the structure, damage due to surface irregularity of the structure, mechanical stress of the structure, mechanical strain of the structure, bending of the structure, torsion of the structure, lateral deformation of the structure, tilt of the structure, tremor or vibration of the structure, and impact force on the structure.

13. At least one of the programmable controllers is A map having the positions of each unit, including at least one of the sensors and at least one of the cameras; The position coordinates or address of at least one of the sensors and at least one of the cameras, and their status; An image of the structure or an area in the vicinity of the structure obtained from at least one of the cameras, the image including an annotated reference frame automatically added by the controller around an object of interest within the image; The sensing data obtained from at least one of the sensors, including inclinometer angle information; and Output of confidence or statistical judgment results based at least partially on the aforementioned sensing data The method according to any one of claims 1 to 3, wherein the above is virtually displayed on a display.

14. The method according to claim 1, wherein at least one of the programmable controllers automatically calculates at least one of the tilt, drift, and lateral deformation of a portion of the structure based on the sensing data.

15. The structure further includes a methane sensor that transmits real-time methane sensing data from the structure, The method according to any one of claims 1 to 3, wherein the controller automatically causes at least one of the cameras to acquire an image of a potential methane source and transmits the image along with the data to the end user.

16. A method for monitoring structures, (a) The step of sensing a potential hazard associated with the structure using at least one sensor, wherein the structure is a power line or a stationary power line support structure, and at least one sensor is part of an inertial measuring unit including an inclinometer and an accelerometer mounted in a housing, the housing being stationary; (b) A step of automatically determining whether the sensing data obtained by at least one of the sensors exceeds a threshold; (c) After it is determined that the sensing data exceeds the threshold, the process of automatically causing at least one camera on the structure or at least one camera in the vicinity of the structure to acquire an image of at least a portion of the structure or an area in the vicinity of the structure. (d) A step of automatically transmitting at least a portion of the sensing data and the corresponding camera image to a remote receiver. (e) The controller automatically transmits the position of the sensor to the remote receiver; and (f) A process in which at least one of the following potential hazards is detected by at least one of the sensors and shown in the camera image: corrosion, mechanical damage, undesirable structural stress, undesirable structural strain, undesirable structural bending, undesirable structural torsion, undesirable structural lateral deformation, undesirable structural tilt, impact force, movement of the structure due to flood, movement of the structure due to earthquake, movement of the structure due to ground subsidence, or movement of the structure due to ground erosion. Methods that include...

17. The aforementioned potential hazard is corrosion. The aforementioned sensing data is related to the harmonic frequency of the structure. The method according to claim 16, wherein the threshold is related to the harmonic frequency of the structure in a non-corrosive state.

18. Structural monitoring software, (a) A first set of multiple commands that receive sensing data from at least one sensor relating to the angle, inertia, acceleration, or vibration characteristics of a stationary structure; (b) A second set of commands that automatically determines whether the sensing data exceeds a threshold; (c) A third set of commands that, after it is determined that the sensing data exceeds the threshold, cause at least one camera on the structure or at least one camera in the vicinity of the structure to automatically acquire images of at least a portion of the structure or an area in the vicinity of the structure; (d) A fourth set of commands that automatically transmit at least a portion of the sensing data and the corresponding camera image to a remote receiver; (e) A fifth set of commands for automatically transmitting the position of at least one of the sensors to the remote receiver; and (f) A sixth set of commands to display the camera image on the display, which includes at least one of the following: corrosion on the structure, movement of the structure due to flooding, movement of the structure due to earthquakes, movement of the structure due to ground subsidence, or movement of the structure due to ground erosion. Software, including.

19. The aforementioned sensing data is related to the harmonic frequency during corrosion of the structure. The software according to claim 18, wherein the nominal value is related to the harmonic frequency of the structure in an uncorroded state.

20. The aforementioned structure is a power transmission tower or power transmission pole, The sensor is fixedly attached to the structure. The software according to claim 18 or 19, wherein the camera is attached to the structure.

21. A map showing the positions of the sensors, which are part of the inertial measurement unit; The position coordinates or address of the inertial measuring unit; Images of the structure or an area near the structure obtained from at least one of the cameras; The sensing data obtained from at least one of the sensors, including inclinometer angle information; and Output of confidence or statistical judgment results based at least partially on the aforementioned sensing data The software according to claim 18 or 19, further comprising additional program instructions for causing the display to be virtually shown on the display.

22. The software according to claim 18, further comprising additional program instructions for automatically calculating the tilt, drift, or lateral deformation of the structure based on the sensing data.

23. A device for monitoring structures, (a) an inertial measuring unit comprising at least an inclinometer and an accelerometer mounted within a housing, wherein the housing is fixedly attached to the structure, and the inertial measuring unit is configured to sense potential hazards associated with the structure fixed to the ground; (b) At least one programmable controller configured to automatically determine if the sensing data obtained by the inertial measurement unit exceeds a nominal value; (c) At least one programmable controller configured to, after determining that the sensing data differs from the nominal value, to automatically cause at least one camera located on the structure or at least one camera located near the structure to acquire images of at least a portion of the structure or an area near the structure; (d) An antenna coupled to the inertial measurement unit; (e) A battery coupled to the inertial measuring unit; (f) A photovoltaic panel coupled to the inertial measurement unit; (g) Remote receiver including a display; (h) at least one programmable controller configured to automatically transmit at least a portion of the sensing data and the corresponding camera image to the remote receiver; (e) at least one of the programmable controllers configured to automatically transmit the position of the inertial measuring unit to the remote receiver; and (f) The display, which displays at least a portion of the sensing data and the camera image relating to at least one of the following: corrosion of the structure, mechanical damage to the structure, undesirable structural stress, undesirable structural strain, undesirable structural bending, undesirable structural torsion, undesirable structural lateral deformation, undesirable structural tilt, impact force on the structure, movement of the structure due to flood, movement of the structure due to earthquake, movement of the structure due to ground subsidence, movement of the structure due to ground erosion, or movement of the structure due to rock slide. A device including a device.

Images