A method and system for detecting potential safety hazards of a tower

Abstract

The present application relates to the technical field of hidden danger detection, in particular to a tower safety hidden danger intelligent detection method and system, comprising the following steps: acquiring multi-modal perception data and performing space-time alignment to form component-level state data; extracting double-mode visual data to obtain hidden danger candidate area, initial hidden danger state and visual features, and associating non-visual state features; using a Transformer model to obtain fusion state features and fusion hidden danger state, inputting the fusion state features of continuous time nodes into an LSTM model to obtain degradation trend; forming state evaluation, life prediction and graded early warning results according to the current hidden danger state and the degradation trend. The present application forms a corresponding relationship between visual information and structural state of the same component through multi-modal data space-time alignment, cross-modal association and time series modeling, and reduces the influence of single observation deviation on hidden danger judgment through abnormal result re-determination, thereby improving the reliability of hidden danger state determination and prediction and early warning results.

Description

Technical Field

[0001] This invention relates to the field of hazard detection technology, and in particular to an intelligent method and system for detecting safety hazards in iron towers. Background Technology

[0002] The field of hazard detection technology mainly involves the collection, identification, analysis, and judgment of the operating status, appearance defects, component anomalies, and potential risk characteristics of equipment, facilities, and structures during the construction and operation maintenance of communication systems. It is widely applied in scenarios such as communication tower construction acceptance, daily inspections, operation maintenance, and safety monitoring. Among these, the intelligent hazard detection method for communication towers refers to the method of detecting abnormal states and hazard characteristics of the main body, tower materials, connecting parts, and auxiliary components of communication signal towers. This typically involves collecting images of the communication tower, component status information, or on-site monitoring data to identify and judge corrosion, deformation, missing components, connection anomalies, and other hazard characteristics. It also combines image processing, feature extraction, pattern recognition, and data analysis techniques to detect and analyze information related to safety hazards during the construction and maintenance of communication towers.

[0003] Current methods for detecting safety hazards in communication towers primarily rely on image recognition or sensor monitoring to assess the tower's structural appearance and operational status. However, during the construction, acceptance, operation, and maintenance of communication towers, data from different sources differ in collection time and spatial location. It is difficult to establish a correspondence between visual information and structural response information of the same component. Single identification results lack correlation with continuous state changes, and local observation deviations lack cross-verification with other modal states and historical degradation information. Consequently, the assessment of communication tower hazards is easily influenced by a single detection result, and there is insufficient data integration between state assessment and lifespan prediction. The graded early warning results fail to take into account both current hazards and degradation trends, which is detrimental to the continuous assessment of the safety status of communication towers during construction and subsequent maintenance. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent detection method and system for safety hazards in iron towers. By performing component-level spatiotemporal alignment of multimodal sensing data of iron towers collected by multi-source sensing nodes, and by associating non-visual state features based on bimodal visual recognition, the method combines cross-modal association processing of the Transformer model and temporal state analysis of the LSTM model to continuously correlate the current hazard status, degradation trend, state assessment results, and life prediction results of iron tower components. When different detection results or inconsistent directions of change occur, cross-modal state re-judgment is performed to form a graded early warning result corresponding to the current state and degradation process of the iron tower components.

[0005] To achieve the above objectives, the present invention provides an intelligent detection method for safety hazards in iron towers, comprising the following steps: S1: Acquire multimodal sensing data of the tower collected by multi-source sensing nodes, and perform spatiotemporal alignment according to component identification, spatial location and acquisition time to form component-level status data; S2: Extract bimodal visual data from the component-level state data, perform candidate region identification and visual feature extraction to obtain hazard candidate regions, bimodal visual features and initial hazard states, and extract corresponding non-visual state features using the hazard candidate regions as indexes; S3: Input the bimodal visual features and the non-visual state features into the Transformer model to perform cross-modal association, obtain fused state features and fused potential state, and input the fused state features of the same tower component at consecutive time nodes into the LSTM model to obtain the degradation trend; S4: Compare the initial hidden danger state and the fused hidden danger state. When the two are consistent and the degradation trend is consistent with the change direction corresponding to the fused hidden danger state, determine the current hidden danger state. When the two are inconsistent or the change direction is inconsistent, extract the corresponding data from the component-level state data according to the hidden danger candidate region to perform cross-modal state determination, and determine the current hidden danger state according to the determination result. S5: Generate a status assessment result and a life prediction result based on the current hidden danger status and the degradation trend, and determine a graded early warning result based on the status assessment result and the life prediction result.

[0006] Furthermore, the multi-source sensing node consists of a drone, a fixed sensor, and a mobile detection unit. The drone collects visual data and spatial morphological data of the tower, the fixed sensor collects structural response data and environmental data of the tower, and the mobile detection unit collects supplementary detection data of designated tower components. The multi-source sensing node records the collection time according to a unified time base and binds the data collected by the drone, the fixed sensor, and the mobile detection unit to the corresponding component identifier and spatial location, so that the multimodal sensing data of the tower from different sources are spatiotemporally aligned according to the same tower component.

[0007] Furthermore, the bimodal visual data consists of spatiotemporally aligned visible light and infrared images. The visible light and infrared images are input into an improved YOLOv8s model. Based on the tower component locations and hazard locations output by the improved YOLOv8s model, the hazard candidate regions are determined. The visible light and infrared images corresponding to the hazard candidate regions are input into a ResNet-18 model for feature extraction and state recognition to obtain the bimodal visual features and the initial hazard state. Based on the component identifier corresponding to the hazard candidate region and the acquisition time, the non-visual state features corresponding to the hazard candidate region are extracted from the component-level state data.

[0008] Furthermore, the bimodal visual features and the non-visual state features are matched according to the component identifier and the acquisition time. The feature dimensions of the matched bimodal visual features and the non-visual state features are unified and arranged according to the modality source to form a multimodal feature sequence. The multimodal feature sequence is input into the Transformer model. The correlation between each modality feature and the current state of the tower component is determined through the attention correlation between different modal features. The fused state feature is generated according to the correlation. The fused potential hazard state is determined by the state output of the Transformer model.

[0009] Furthermore, multiple fusion state features corresponding to the same tower component are arranged in chronological order of acquisition time to form a component state time series; the component state time series is input into the LSTM model, and the LSTM model recursively associates the state memory of the previous time node with the fusion state feature of the current time node, and determines the degradation trend based on the state change direction of consecutive time nodes.

[0010] Furthermore, the initial hidden danger state and the merged hidden danger state are compared for consistency in state category, and the change direction of the degradation trend representation is compared for consistency in trend with the state change direction corresponding to the merged hidden danger state. When the initial hidden danger state and the merged hidden danger state have the same state category, and the change direction of the degradation trend representation is the same as the state change direction corresponding to the merged hidden danger state, the merged hidden danger state is determined as the current hidden danger state. When the state category is inconsistent or the change direction is inconsistent, the corresponding hidden danger candidate region is determined as the region to be verified.

[0011] Furthermore, for the area to be verified, the bimodal visual data and the non-visual state features are re-extracted from the component-level state data according to the corresponding component identifier and the acquisition time. Visual features are extracted from the re-extracted bimodal visual data to obtain bimodal visual features for verification. The bimodal visual features for verification and the non-visual state features are input into the Transformer model for cross-modal state determination to obtain the verification hazard status. The current hazard status is updated according to the verification hazard status.

[0012] Furthermore, the hazard category of the tower component is determined based on the current hazard status, and the corresponding state change direction is determined based on the degradation trend. The hazard category and the state change direction are correlated to form the state assessment result. The state evolution prediction is performed based on the fusion state characteristics, the current hazard status, and the degradation trend of the same tower component at consecutive time points to obtain the life prediction result.

[0013] Further, candidate warning levels are determined based on the status assessment results, and the lifetime prediction results are compared with the warning conditions corresponding to the candidate warning levels. When the lifetime prediction results meet the warning conditions, the candidate warning level is determined as the graded warning result. When the lifetime prediction results do not meet the warning conditions, adjacent warning levels are selected according to a preset order of risk levels corresponding to the warning levels, and the lifetime prediction results are compared with the warning conditions corresponding to the adjacent warning levels until the corresponding warning conditions are met or the boundary of the preset order is reached. The graded warning result is then determined based on the comparison results.

[0014] Furthermore, an intelligent detection system for safety hazards of iron towers is provided. This system is applied to the intelligent detection method for safety hazards of iron towers and includes a multimodal data processing module, a visual recognition module, a fusion time series analysis module, a status determination module, and a prediction and early warning module. The multimodal data processing module acquires multimodal sensing data of the tower collected by multi-source sensing nodes, and performs spatiotemporal alignment according to component identification, spatial location and acquisition time to form component-level status data; The visual recognition module extracts bimodal visual data from the component-level state data, performs candidate region recognition and visual feature extraction, obtains hazard candidate regions, bimodal visual features and initial hazard states, and extracts corresponding non-visual state features using the hazard candidate regions as indexes. The fusion time series analysis module inputs the dual-modal visual features and the non-visual state features into the Transformer model to perform cross-modal correlation, thereby obtaining fusion state features and fusion potential state. It also inputs the fusion state features of the same tower component at consecutive time nodes into the LSTM model to obtain the degradation trend. The state determination module compares the initial hidden danger state and the fused hidden danger state. When the two are consistent and the degradation trend is consistent with the change direction corresponding to the fused hidden danger state, the current hidden danger state is determined. When the two are inconsistent or the change direction is inconsistent, the corresponding data is re-extracted from the component-level state data according to the hidden danger candidate region to perform cross-modal state determination, and the current hidden danger state is determined according to the determination result. The prediction and early warning module generates a status assessment result and a life prediction result based on the current hidden danger status and the degradation trend, and determines a graded early warning result based on the status assessment result and the life prediction result.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By spatiotemporally aligning multimodal sensing data collected during the construction and maintenance of communication towers according to component identification, spatial location, and acquisition time, and by associating visual and non-visual state features with potential hazard candidate areas, a correspondence is established between multi-source detection information of the same communication tower component. Since the Transformer model associates different modal features, and the LSTM model performs temporal modeling of fused state features at continuous time points, a link is established between the hazard status and degradation trend of communication tower components. When the initial hazard status is inconsistent with the fused hazard status or the direction of change is inconsistent, component-level state data is retrieved again for cross-modal state determination. This reduces the impact of single observation results on hazard judgment during the construction acceptance and operation and maintenance of communication towers, improves the reliability of hazard status determination, and generates state assessment, life prediction, and graded early warning results, providing a basis for the construction acceptance, operation inspection, and maintenance of communication towers. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the data processing link for methods of detecting safety hazards in iron towers; Figure 2 This is a schematic diagram illustrating the data binding relationship between multi-source sensing nodes and tower components. Figure 3 This is a schematic diagram illustrating the association between bimodal visual recognition and non-visual state features. Figure 4 This diagram illustrates the data processing relationship between the Transformer model and the LSTM model. Figure 5 This is a schematic diagram illustrating the consistency determination and verification relationship of potential hazards. Figure 6 This is a schematic diagram of the module interaction of the tower safety hazard detection system. Detailed Implementation

[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings. The present invention uses tower components as the basic object for multimodal data organization and state analysis. Data acquired by multi-source sensing nodes at different locations and times are correlated through component identification, spatial location, and acquisition time. Around the same tower component, component-level state data, hazard candidate areas, bimodal visual features, initial hazard state, non-visual state features, fused state features, fused hazard state, degradation trend, current hazard state, state assessment results, lifespan prediction results, and graded early warning results are sequentially formed. Component identification is used to uniquely indicate tower components within the current detection range; spatial location describes the positional correspondence between the acquired data and the corresponding tower component; and acquisition time identifies the moment the data was acquired. Component-level state data is a data set formed by organizing multimodal sensing data belonging to the same tower component and having a temporal correspondence according to a unified data object. Bimodal visual data refers to visible light images and infrared images that have achieved spatiotemporal alignment. Non-visual state features refer to spatial morphology, structural response, and environmental state-related features corresponding to hazard candidate areas. Fusion state features are component state representations formed by the Transformer model through cross-modal correlation of bimodal visual and non-visual state features. Degradation trend is the direction of state change determined by the LSTM model based on the fusion state features of the same tower component at consecutive time points. The aforementioned data objects will use the same names and referential relationships in subsequent processing stages.

[0019] The improvements to the YOLOv8s model enable it to simultaneously detect tower components and potential hazard regions using spatially corresponding visible light and infrared images. Specifically, the image input structure of the YOLOv8s model is adjusted to include a visible light input branch and an infrared input branch. The visible light input branch receives visible light images, and the infrared input branch receives infrared images. Before input, both types of images are registered according to the spatial correspondence of the same tower component and adjusted to the same model input size. The visible light and infrared input branches respectively extract shallow features through convolutional layers, obtaining visible light shallow features and infrared shallow features. These visible light and infrared shallow features with the same spatial position are concatenated along the feature channel direction, and then fused using convolutional layers to form a dual-modal fused feature. This dual-modal fused feature is then input into the subsequent backbone feature extraction network, feature fusion network, and detection head of the YOLOv8s model. The detection head outputs the tower component location, hazard location, and corresponding detection results.

[0020] The location of the tower component is used to determine the range of the tower component corresponding to the current detection object, while the location of the hidden danger is used to determine the local area within the tower component range that needs further analysis, and the corresponding local area is determined as the candidate region for hidden danger. When multiple hidden danger locations are identified in the same tower component, multiple corresponding candidate regions for hidden danger are formed, and each candidate region for hidden danger continues to maintain the corresponding component identification and acquisition time. When improving the YOLOv8s model training, visible light images and infrared images that have completed spatial registration are used as paired training samples. When performing geometric data enhancements such as cropping, scaling, rotation, or flipping on paired images, the same spatial transformation relationship is used for both types of images to maintain the correspondence between the same tower component regions in the visible light images and infrared images. For the enhancement of image attributes such as brightness, contrast, and grayscale of the visible light images and infrared images respectively, they are processed separately according to the data characteristics of the corresponding modality. During the training process, the labeled tower component locations and hidden danger locations are used to supervise the detection results, enabling joint training of the visible light input branch, infrared input branch, dual-modal feature fusion part, and subsequent YOLOv8s detection network. The training loss employed includes at least a classification loss to constrain target category recognition and a bounding box regression loss to constrain the locations of tower components and potential hazards, enabling the model to simultaneously learn the correspondence between bimodal visual features and the locations of tower components and potential hazards. During the inference phase, the spatial correspondence between the visible light image and the infrared image is maintained, ensuring that the corresponding regions of the same tower component in both types of images enter the corresponding input branches. Through the aforementioned dual-branch feature extraction and bimodal feature fusion method, the component appearance, contour, and surface state information in the visible light image, along with the infrared response information of the corresponding region in the infrared image, jointly participate in candidate region localization, outputting the tower component location and potential hazard location for subsequent visual feature extraction and state recognition around the same potential hazard candidate region.

[0021] After identifying candidate regions for potential hazards, visible light and infrared candidate region images are extracted from the corresponding visible light and infrared images, respectively, according to the same candidate region boundaries. These two types of candidate region images are then input into a lightweight ResNet-18 model. The lightweight ResNet-18 model is based on the ResNet-18 residual network and performs structured channel pruning on redundant feature channels in the network to reduce the computational cost of convolutions and the model's storage requirements. Specifically, the ResNet-18 model is first trained using training samples of potential hazards in iron towers. Based on the importance of the corresponding feature channels in each convolutional layer during training, channels to be retained and channels to be pruned are determined, and less important feature channels are deleted.

[0022] During channel pruning, the main branch and shortcut branch involved in residual addition within the same residual module are adjusted accordingly to ensure that the two branches maintain the same feature dimension at the residual addition position, thus preventing channel pruning from disrupting the original residual connections of ResNet-18. For network layers that directly handle input feature extraction or final state classification, their input and output functions remain unchanged. After structured channel pruning, the pruned ResNet-18 model is retrained or fine-tuned using training samples of tower hazards to recover the loss of visual feature representation ability caused by the deletion of some feature channels.

[0023] Furthermore, during model deployment, batch normalization layer parameters can be fused with corresponding convolutional layers, and the model weights of convolutional layers and state classification layers can be represented with low precision to further reduce model storage space and inference computation. This lightweighting process does not change the basic processing relationship of the ResNet-18 model for visual feature extraction and state recognition of hazard candidate regions. For the same hazard candidate region, visible light and infrared candidate region images are respectively processed by the lightweight ResNet-18 model to extract features, obtaining visible light and infrared region features. The visible light and infrared region features are then concatenated or mapped and fused to form a bimodal visual feature corresponding to the current hazard candidate region. The bimodal visual feature retains the corresponding component identification, acquisition time, and hazard candidate region, and is input into the state classification layer for state recognition to obtain the initial hazard state. The initial hazard state adopts the same state category system as the subsequently fused hazard state, enabling a consistency comparison between the initial hazard state and the fused hazard state output by the Transformer model.

[0024] Therefore, the improved YOLOv8s model achieves joint candidate region identification of dual-modal visual data by setting visible light input branches, infrared input branches, and a dual-modal feature fusion structure; the lightweight ResNet-18 model reduces the computational load while maintaining the functions of visual feature extraction and state recognition of hazard candidate regions through structured channel pruning, fine-tuning after pruning, and network compression during the model deployment stage, and completes the output of dual-modal visual features and initial hazard state respectively.

[0025] Example 1 Please see Figure 1-5This embodiment provides a method for detecting safety hazards in iron towers. The method sequentially executes steps S1 to S5: multimodal data acquisition and spatiotemporal alignment, bimodal visual recognition and non-visual state feature association, cross-modal fusion and temporal state analysis, hazard state consistency determination and verification, and state assessment and predictive warning. The component-level state data generated in S1 serves as the data foundation for S2 and S4. The bimodal visual features, initial hazard state, and non-visual state features generated in S2 are input to S3. The fused hazard state and degradation trend generated in S3 are input to S4. The current hazard state determined in S4 and the degradation trend generated in S3 are jointly input to S5, thus forming a continuous data processing chain from multi-source sensing data to hierarchical warning results.

[0026] S1: Acquire multimodal sensing data of the tower collected by multi-source sensing nodes, and perform spatiotemporal alignment according to component identification, spatial location and acquisition time to form component-level status data.

[0027] The multi-source sensing node consists of a drone, fixed sensors, and a mobile detection unit. The drone, acting as an aerial data acquisition platform, obtains visual and spatial morphological data of the tower. The visual data corresponds to visible light and infrared images, while the spatial morphological data corresponds to the spatial geometry of the tower components. Fixed sensors are installed at their respective locations and establish a fixed association with the corresponding tower components. They are used to acquire structural response data and environmental data. The structural response data characterizes the vibration state, strain state, or structural response reflected by relevant sensing quantities of the corresponding component during the inspection. The environmental data records the environmental conditions corresponding to the same acquisition time. The mobile detection unit is a data acquisition platform capable of changing its detection position on-site and facing a designated tower component to obtain supplementary detection data. When the inspection task specifies a particular tower component, the mobile detection unit writes the acquired data into the corresponding component's data record, using the same component identification and time recording methods as other multi-source sensing nodes.

[0028] Each multi-source sensing node synchronously writes the acquisition time when outputting collected data. In addition to the acquisition time, the data output by the UAV also includes spatial location, which maps the image area and spatial morphology data to the corresponding tower component. Fixed sensors establish component identification relationships with the tower components being measured during deployment, and the structural response data and environmental data output by the fixed sensors use the corresponding component identification. When performing supplementary acquisition of a designated component, the mobile detection unit receives the corresponding component identification and outputs the supplementary detection data along with the component identification and acquisition time. Therefore, data from different sources all have component identification, spatial location or a component correspondence equivalent to the spatial location, and acquisition time before entering subsequent processing.

[0029] Spatiotemporal alignment includes object alignment and time alignment. Object alignment uses component identifiers as the basic index. For UAV data, the corresponding component is determined based on its spatial location; for fixed sensor data, the corresponding component is determined based on its installation association; and for mobile detection unit data, the corresponding component is determined based on the component identifier corresponding to the acquisition task. After object alignment is completed, data for the same component enters the same component data set. Time alignment uses the acquisition time of the current visual data as the corresponding moment. Spatial morphology data, structural response data, environmental data, and supplementary detection data corresponding to the time are read from the same component data set. For data sequences formed by continuous acquisition, the data sequence is maintained according to a unified time reference, so that data entering the same time node all represent the state of the same tower component near the corresponding acquisition time.

[0030] Component-level status data uses the component identifier as the primary index and the acquisition time as the temporal index, while maintaining the source category of each modal data. Therefore, visible light images, infrared images, spatial morphology data, structural response data, environmental data, and supplementary detection data present at the time of acquisition for the same component can be re-read in subsequent steps based on the component identifier and acquisition time. Component-level status data only establishes object and temporal correspondences between data from different sources.

[0031] After S1 outputs component-level state data, it can read the record corresponding to any component identifier and check the acquisition time and modal source to confirm that data from different sources have been organized under the same component object. This component-level organization relationship serves as the basis for S2 to extract bimodal visual data and non-visual state features, and also as the backtracking entry point for S4 to re-extract data when state inconsistencies occur.

[0032] S2: Extract bimodal visual data from component-level state data, perform candidate region identification and visual feature extraction to obtain hazard candidate regions, bimodal visual features and initial hazard states, and extract corresponding non-visual state features using hazard candidate regions as indexes.

[0033] First, based on component identification and acquisition time, spatiotemporally aligned visible light and infrared images are retrieved from the component-level status data. Visible light and infrared images of the same component at the same corresponding time are treated as a set of bimodal visual data. The two images in the bimodal visual data maintain a correspondence with the same tower component. The visible light image is used to represent the component's appearance, outline, and surface condition, while the infrared image is used to represent the infrared response of the same area. The two types of images are not directly output as independent detection results but are jointly incorporated into the candidate region identification and visual feature extraction process.

[0034] Visible light and infrared images are input into the improved YOLOv8s model. The YOLOv8s model receives the visible light and infrared images with established spatial correspondences and outputs the locations of tower components and potential hazards. The component locations output by the model are used to confirm the tower component to which the detected object belongs, and the hazard locations output by the model are used to identify local areas requiring further analysis within the corresponding component. The local area defined by the component location and the hazard location is defined as a hazard candidate region. If multiple hazard locations exist for the same component, each hazard location corresponds to a hazard candidate region, and the same component identifier is retained in subsequent processing.

[0035] Improvements to the YOLOv8s model support pairwise bimodal visual data as detection input, with the output of tower component locations and hazard locations as boundaries. During the inference phase, the input image size is kept consistent with the model input size, and the spatial correspondence between the visible light and infrared images is maintained. The model output should be able to locate at least the tower component and the hazard candidate region within the component, enabling the subsequent ResNet-18 model to extract visual features around the same region.

[0036] The visible light image region and infrared image region corresponding to each potential hazard region are input into a lightweight ResNet-18 model. The lightweight ResNet-18 model performs feature extraction and state recognition on the two types of visual data, outputting bimodal visual features and an initial hazard state. The bimodal visual features represent the visual state corresponding to the current potential hazard region, preserving the corresponding component identification and acquisition time. The initial hazard state is the state recognition result formed by the visual processing chain for the current potential hazard region, and it adopts a state category system that allows for category consistency comparison in subsequent fused hazard states. The compression method of the ResNet-18 model only needs to maintain the relationship between the visual feature extraction and state recognition output of the potential hazard region.

[0037] The improved YOLOv8s model and the lightweight ResNet-18 model have different responsibilities in S2. The improved YOLOv8s model is responsible for identifying the detection object and hazard candidate regions, while the lightweight ResNet-18 model receives the identified hazard candidate regions and outputs bimodal visual features and the initial hazard state. Therefore, the visual recognition process first completes the candidate region localization, and then completes the candidate region-level feature representation, avoiding the direct use of irrelevant areas of the entire tower image as subsequent cross-modal association objects.

[0038] After obtaining candidate hazard regions, the system returns to the component-level state data based on the component identifier and acquisition time corresponding to the candidate region. Non-visual state features are extracted from data records of the same component at the same corresponding time. These non-visual state features originate from at least the spatial morphology data, structural response data, and environmental data associated with the current component. Spatial morphology data extracts the corresponding spatial geometric state according to the component location corresponding to the candidate hazard region; structural response data reads the corresponding state records according to the component identifier and acquisition time; and environmental data is read according to the same acquisition time. Modal source identifiers are retained for all types of non-visual state features, enabling S3 to identify the source of different features.

[0039] When the mobile detection unit has obtained supplementary detection data for the current tower component, the supplementary detection data is incorporated into the current component's status record according to the corresponding component identifier and acquisition time, and serves as supplementary status input associated with the candidate hazard area. The supplementary detection data does not change the output definition of bimodal visual recognition; it is only used as component status information that can be read during subsequent cross-modal status analysis.

[0040] At the end of S2, each potential hazard area has a corresponding component identifier, acquisition time, bimodal visual features, initial hazard state, and non-visual state features. By checking whether these data objects share the same component identifier and corresponding acquisition time, it can be confirmed that the visual state and non-visual state have established a correlation for the same tower component, and S3 performs cross-modal fusion accordingly.

[0041] S3: Input the dual-modal visual features and non-visual state features into the Transformer model to perform cross-modal correlation, obtain the fused state features and fused potential state, and input the fused state features of the same tower component at continuous time nodes into the LSTM model to obtain the degradation trend.

[0042] Before entering the Transformer model, bimodal visual features and non-visual state features are matched according to component identification and acquisition time, and only features belonging to the same tower component and corresponding in time are grouped into the same input group. Since the original feature lengths and data representation formats of bimodal visual features, spatial morphology features, structural response features, and environmental features differ, dimensionality mapping is performed separately for each modality. Specifically, a corresponding feature mapping layer is set for each modality, and each feature mapping layer uses a fully connected layer to linearly map the input features of the corresponding modality, converting features of different original dimensions into feature vectors consistent with the input dimensions of the Transformer model. For spatial morphology features, structural response features, and environmental features containing multiple continuous state variables, the original feature vector of the corresponding modality is first formed according to a preset field order, and then input into the corresponding fully connected mapping layer; for bimodal visual features, the visual feature extraction results are directly input into the corresponding fully connected mapping layer. After fully connected mapping, each modality feature is normalized by a normalization layer, ensuring that features from different sources have a consistent feature representation scale before entering the Transformer model. Different modalities employ independent feature mapping layers. The input dimension of each mapping layer is set according to the dimension of the original features of the corresponding modality, while the output dimension of each mapping layer remains consistent. This allows bimodal visual features, spatial morphology features, structural response features, and environmental features to be converted into feature representations of the same dimension. To maintain the source distinction between different modalities, after dimension mapping, a corresponding modality identifier is assigned to each modality. The embedded features corresponding to the modality identifier are combined with the features of that modality after dimension mapping, enabling the Transformer model to distinguish whether the current input feature originates from bimodal visual data, spatial morphology data, structural response data, or environmental data. After the above processing, according to the preset modality arrangement order, the modal features with a unified feature dimension and carrying the corresponding modality identifier are arranged sequentially to form a multimodal feature sequence. This multimodal feature sequence is then input into the Transformer model for cross-modal association. Thus, different modal features maintain the distinction of their respective data sources while having a unified input dimension, enabling the Transformer model to establish associations between different modal state information within the same feature space.

[0043] Features are arranged according to their modal origins to achieve dimensional uniformity, forming a multimodal feature sequence. Each feature item in the multimodal feature sequence corresponds to a state source of the current tower component, and the sequence as a whole corresponds to a component and a time node. After receiving the multimodal feature sequence, the Transformer model uses attention association to calculate the interrelationships between the modal features, enabling information interaction between visual state, spatial morphology, structural response, and environmental state within the same component. The attention association operates on feature items that have already undergone object matching and time matching, avoiding the direct establishment of cross-modal associations between data from different components or different time nodes.

[0044] The Transformer model outputs fused state features based on the relationships formed between different modal features. These fused state features are still bound to the current component identifier and the current acquisition time, representing the overall state of the tower component at that time point. The Transformer model also forms a fused hazard state, which uses the same state category caliber as the initial hazard state in S2, allowing for direct state category consistency comparison in S4. The fused hazard state is not a copy of the initial hazard state, but rather a state output formed by the cross-modal association of bimodal visual features and non-visual state features.

[0045] For the same tower component, multiple time points are generated through multiple data acquisitions. Each time point can yield a fused state feature according to the aforementioned processing. The multiple fused state features corresponding to the same component are arranged in chronological order of acquisition time to form a component state time series. The component state time series maintains its chronological order, with the fused state feature of the previous time point preceding the current time point, and the fused state feature of the current time point serving as the current state input at the end of the sequence.

[0046] The component state time series is input into the LSTM model. While processing the fused state features of the current time node, the LSTM model simultaneously reads the state memory retained after processing the previous time node, recursively associating historical states with the current fused state features. As time nodes are sequentially input into the LSTM model, the model output reflects the continuous relationship between the state of the same tower component and the acquisition time. The degradation trend is determined based on the LSTM state outputs of consecutive time nodes, representing the direction of change of the state of the same tower component over time.

[0047] The degradation trend and the fusion risk state are compared using a comparable direction of state change. The direction of state change corresponding to the current fusion risk state is formed by the relationship between the current state at the current time node and the state at the previous effective time node, while the degradation trend is formed by the LSTM model performing temporal correlation on the continuous fusion state features. S4 judges whether the current visual and cross-modal recognition results are consistent with the historical state evolution relationship by comparing the consistency of the directions of the two.

[0048] In this embodiment, the Transformer model and the LSTM model employ a cascaded data processing relationship. The Transformer model processes the state relationships between different modes at the same time point, while the LSTM model processes the state relationships between the same component at different time points. The fused state features output by the Transformer model serve as the direct temporal input to the LSTM model, which does not directly receive the raw acquired data from incomplete component matching and cross-modal fusion. Therefore, the multi-source state relationships at the same time point and the state evolution relationships across time are divided into two consecutive processing stages.

[0049] S4: Compare the initial hazard state and the merged hazard state. When the two are consistent and the degradation trend is consistent with the change direction corresponding to the merged hazard state, determine the current hazard state. When the two are inconsistent or the change direction is inconsistent, extract the corresponding data from the component-level state data according to the hazard candidate area to perform cross-modal state judgment, and determine the current hazard state based on the judgment result.

[0050] S4 first performs a state category consistency comparison. The comparison objects are the initial hazard state and the fused hazard state corresponding to the same hazard candidate area. The initial hazard state comes from the bimodal visual recognition process, and the fused hazard state comes from the Transformer cross-modal association process. Both share component identification, acquisition time, and hazard candidate area. If the two state categories are the same, the state category comparison result is consistent; if the two state categories are different, the state category comparison result is inconsistent.

[0051] Beyond state category comparison, S4 further performs trend consistency comparison. The degradation trend characterizes the state change direction formed by the LSTM model at consecutive time nodes. The state change direction corresponding to the fused hazard state represents the change direction of the current fused hazard state relative to the fused hazard state at the previous valid time node. By comparing whether the two point to the same state evolution direction, it is determined whether the cross-modal state at the current time node is consistent with the continuous temporal relationship.

[0052] When the initial hazard state and the fused hazard state have the same state category, and the direction of change of the degradation trend is consistent with the direction of change of the state corresponding to the fused hazard state, it indicates that there is no conflict between the current visual recognition result, the current cross-modal state, and the continuous temporal state. The fused hazard state is then determined as the current hazard state. At this point, there is no need to call the component-level state data again for verification, and the current hazard state directly proceeds to S5.

[0053] When the initial hazard state and the fused hazard state have different state categories, or the direction of change in the degradation trend is inconsistent with the direction of change in the fused hazard state, it indicates a conflict between the visual recognition result, the cross-modal fusion result, and the relationship between state changes over continuous time at the current time point. The corresponding hazard candidate area is then designated as a region to be verified. The region to be verified retains the corresponding component identifier, acquisition time, and regional location information to limit the tower components and local areas targeted by subsequent verification processing. For the region to be verified, based on the corresponding component identifier, acquisition time, and regional location information, the corresponding bimodal visual data is reread from the component-level state data, and simultaneously, the non-visual state data corresponding to the current time point and the state data of adjacent effective time points for the same tower component are extracted. For regions to be verified that have already obtained supplementary detection data from the mobile detection unit, supplementary detection data corresponding to the component identifier and regional location is simultaneously read, so that the data input in the verification stage includes adjacent time state information and supplementary detection information compared to the initial cross-modal fusion stage.

[0054] The re-read visible light and infrared images are re-cropped according to the location of the area to be verified, maintaining the spatial correspondence between the two types of images. Visual information of the hazard location and its adjacent component areas is preserved, centered on the area to be verified, and visual features are extracted again to obtain the bimodal visual features for verification. The bimodal visual features for verification are re-extracted from the original bimodal visual data corresponding to the area to be verified, without directly reusing the bimodal visual features already formed in the initial visual recognition stage.

[0055] Feature extraction and dimensional unification are performed on the non-visual state data corresponding to the current time point, the state data of adjacent valid time points, and the supplementary detection data when they exist, while retaining the corresponding data source identifiers. The state data of adjacent valid time points is used to characterize the state changes of the component to be verified before and after the current acquisition time, while the supplementary detection data provides additional state information for the component or area to be verified. The dual-modal visual features for verification, the non-visual state features corresponding to the current time point, the state features corresponding to adjacent valid time points, and the supplementary detection features when they exist are combined according to their corresponding modal sources to form a multi-modal feature sequence for verification, which is then input into the Transformer model for cross-modal state determination. The Transformer model, by associating the visual state of the current area to be verified, the current non-visual state, the adjacent time state, and the supplementary detection state, reconstructs the state representation of the corresponding tower component and outputs the verified hazard status.

[0056] Verifying the hazard status differs from the initial cross-modal fusion, which directly relies on the dual-modal visual and non-visual state features of the current time point. The verification process further incorporates re-extracted visual information from the area to be verified, state information from adjacent valid time points of the same component, and supplementary detection information when the hazard exists. When the initial identification result is affected by partial occlusion, shooting deviation, instantaneous state fluctuations, or insufficient observation at a single time point, cross-modal association is re-performed by expanding the sources of state information in the verification stage to reduce the impact of single observation deviations on the state determination result. After obtaining the verified hazard status, the current hazard status is updated based on the verified hazard status.

[0057] When a mobile detection unit is used and supplementary detection data has been obtained for the corresponding tower component in the area to be verified, the corresponding supplementary detection data is read synchronously during the process of re-extracting the component-level state data. This ensures that the supplementary detection data maintains the same component identification and acquisition time association with the original non-visual state features. The supplementary detection data participates in the component state representation during the verification phase but does not change the input / output responsibilities of the Transformer model.

[0058] The verified hazard status, obtained by re-evaluating the cross-modal status for inconsistent situations, is used to update the current hazard status. After the update, the current hazard status has a unique result, without retaining conflicting initial and merged hazard statuses as the final output. Therefore, S4 forms two mutually exclusive processing branches: if both the consistent state category and the consistent direction of change are met, the current hazard status is directly determined; if either condition is not met, the process enters the verification stage, updating the current hazard status with the verified hazard status. Both branches use the current hazard status as a unified output, giving S5 a definite input.

[0059] After S4 is completed, the processing result can be verified by checking whether the current hazard status has a status record consistent with the corresponding component identifier and collection time. For verified objects, the current hazard status is also recorded as the verified status; for objects that do not require verification, the current hazard status is directly determined by the merged hazard status. Both types of status use the same status category caliber and adopt the same assessment and prediction processing path after entering S5.

[0060] S5: Generate status assessment results and life prediction results based on the current hazard status and degradation trend, and determine the graded early warning results based on the status assessment results and life prediction results.

[0061] S5 first reads the current hazard status, determines the hazard category of the corresponding tower component, and reads the degradation trend corresponding to the same component to determine the direction of state change of the current hazard over continuous time. It then establishes a correlation between the hazard category and the direction of state change to form a state assessment result. The state assessment result should at least express the hazard status of the currently detected object and the direction of change of that status over continuous time, thus avoiding relying solely on a single visual inspection status as the basis for subsequent early warnings.

[0062] The lifespan prediction process continues to utilize the fused state features, current hazard status, and degradation trend of the same tower component across consecutive time points. The fused state features across consecutive time points have been temporally correlated using an LSTM model. The current hazard status defines the state at the current time point after consistency assessment or verification, and the degradation trend defines the evolution direction of the current state. These three elements together constitute the input for state evolution prediction, ensuring that the lifespan prediction result corresponds to the currently confirmed component state, rather than to an unverified initial visual result.

[0063] In this embodiment, the LSTM model has established a state memory for the fused state features of the same component at consecutive time points. The lifespan prediction stage uses this temporal state relationship to predict the state evolution after the addition of the current potential hazard state, outputting the lifespan prediction result. The lifespan prediction result is used to characterize the remaining service state boundary of the current tower component under the state evolution relationship. The lifespan prediction output is bound to the component identifier and, together with the state assessment result, is used in the early warning level determination.

[0064] Candidate warning levels are determined based on the status assessment results. These levels are mapped from the hazard categories and the direction of status change identified in the status assessment. Different warning levels are assigned in a pre-defined order according to risk severity, with each warning level corresponding to a lifespan warning condition. Lifespan warning conditions are used to determine whether the current lifespan prediction results match the candidate warning levels. All warning conditions use a unified data caliber for lifespan prediction results to avoid direct comparisons of different data scales.

[0065] The life prediction results are compared with the warning conditions corresponding to the candidate warning levels. When the life prediction results meet the warning conditions corresponding to the candidate warning level, the candidate warning level is determined as a graded warning result. When the life prediction results do not meet the corresponding warning conditions, the mismatched candidate warning level is not directly retained. Instead, adjacent warning levels are selected according to the preset order of risk levels corresponding to the warning levels, and then the life prediction results are compared with the warning conditions corresponding to the adjacent warning levels.

[0066] The comparison of adjacent warning levels continues in a preset order until the lifetime prediction result meets the warning conditions corresponding to the currently compared warning level, or the comparison reaches the boundary of the preset order. When the warning conditions are met, the currently compared warning level is used as the graded warning result; when the preset order boundary is reached, the graded warning result is determined based on the last comparison result at the boundary. Therefore, the warning level is not directly output from the underlying sensor data, but is formed sequentially through current hazard status confirmation, status assessment, lifetime prediction, and level condition matching.

[0067] The condition assessment results, lifespan prediction results, and graded early warning results are all associated with the component identifiers of the current tower components. When the detection end reads the graded early warning results, it can trace back the corresponding current hidden danger status and degradation trend. When the condition analysis end reads the lifespan prediction results, it can trace back the fusion state characteristics of the same component at consecutive time points. The above results maintain a clear data source and usage relationship, forming a continuous closed loop between detection results, time-series status, and early warning output.

[0068] Example 2 Please see Figure 6 This embodiment provides a tower safety hazard detection system for carrying out and executing the detection method of Embodiment 1. The difference between this embodiment and Embodiment 1 is that Embodiment 1 describes the data processing sequence according to S1 to S5, while this embodiment assigns the same data processing responsibilities to the multimodal data processing module, visual recognition module, fusion time series analysis module, status determination module, and prediction and early warning module, and clarifies the data input, processing output, and receiving relationships among the five modules.

[0069] The multimodal data processing module receives multimodal sensing data of the tower from the multi-source sensing nodes. The input data includes component identifiers, spatial locations or corresponding component relationships, and acquisition times. The multimodal data processing module merges objects according to component identifiers, establishes positional relationships between UAV-collected data and corresponding tower components according to spatial locations, and completes time mapping of data from different sources for the same component according to acquisition times, forming component-level status data. This component-level status data is read by the visual recognition module and is also retained for the status determination module to read again during verification.

[0070] The visual recognition module receives component-level state data and reads the bimodal visual data corresponding to the same tower component. It then uses an improved YOLOv8s model to determine the location of the tower component and potential hazards, forming hazard candidate regions. A lightweight ResNet-18 model is then used to extract visual features and identify the state of these candidate regions, yielding bimodal visual features and the initial hazard state. Based on the component identifier and acquisition time corresponding to the hazard candidate region, the visual recognition module synchronously reads non-visual state features from the component-level state data and sends the bimodal visual features, initial hazard state, non-visual state features, component identifier, acquisition time, and hazard candidate region to the fusion time-series analysis module and the state determination module.

[0071] The fusion time-series analysis module receives bimodal visual and non-visual state features, matches and unifies the dimensions of features belonging to the same component and corresponding time, forming a multimodal feature sequence. The module then calls a Transformer model to process the multimodal feature sequence, outputting fused state features and fused hazard states, and maintains the fused state feature sequence corresponding to the same tower component according to the acquisition time. Once the data input condition for continuous time nodes is met, the module inputs the corresponding fused state feature sequence into an LSTM model, outputting the degradation trend. The fused hazard state, fused state features, and degradation trend are sent to the state determination module, where the fused state features are also provided to the prediction and early warning module for lifespan prediction.

[0072] The status determination module receives the initial hazard status, the merged hazard status, the degradation trend, the component identifier, the acquisition time, and the hazard candidate area. The module first compares the status categories of the initial and merged hazard statuses, then compares the direction of change represented by the degradation trend with the direction of change corresponding to the merged hazard status. When both comparisons are consistent, the module outputs the merged hazard status as the current hazard status. When either comparison is inconsistent, the module uses the hazard candidate area, component identifier, and acquisition time as re-reading conditions, retrieves the corresponding data from the component-level status data stored in the multimodal data processing module, and calls the visual recognition process and the Transformer cross-modal status determination process to form a verified hazard status, updating the current hazard status accordingly.

[0073] After the status determination module outputs the current hazard status, it sends the current hazard status to the prediction and early warning module. The prediction and early warning module simultaneously receives the degradation trend and continuous time-node fused status characteristics output by the fusion time-series analysis module. The prediction and early warning module determines the hazard category based on the current hazard status and the direction of status change based on the degradation trend, correlating the two to form a status assessment result. Then, it combines the continuous time-node fused status characteristics, the current hazard status, and the degradation trend to form a lifetime prediction result. Subsequently, it determines candidate early warning levels based on the status assessment results, compares the lifetime prediction results with the early warning conditions corresponding to the candidate early warning levels, and performs necessary comparisons of adjacent levels according to a preset level order, ultimately forming a graded early warning result.

[0074] When an inconsistency arises in the status determination, the status determination module generates a readback request for component-level status data. The readback conditions consist of the component identifier corresponding to the candidate hazard area and the acquisition time. The multimodal data processing module returns the corresponding data based on the readback conditions. The visual recognition module re-extracts features from the corresponding bimodal visual data. The fusion time-series analysis module uses the Transformer model to perform cross-modal status determination, and finally returns the verified hazard status to the status determination module. This readback path constitutes a closed loop for anomaly verification in the system and is triggered only when the status category or the direction of change is inconsistent.

[0075] The data sources, data organization methods, model input-output relationships, state comparison relationships, early warning level comparison relationships, and module collaboration relationships in the above embodiments are used to illustrate how the technical solutions described in this invention are implemented. Without departing from the technical solutions described in this invention, those skilled in the art can replace or adjust equivalent data carrying formats, equivalent field names, equivalent execution entities, equivalent module correspondences, or data transmission formats that do not change the processing logic. As long as the component-level spatiotemporal correspondence of multimodal data, bimodal visual candidate recognition, cross-modal state fusion, continuous-time state analysis, cross-modal verification under inconsistent conditions, and the processing relationship between state assessment and prediction / early warning are maintained, all of these fall within the implementation forms covered by the technical solutions described in this invention.

Claims

1. A method for intelligent detection of safety hazards in iron towers, characterized in that, The method includes: Acquire multimodal sensing data of the iron tower collected by multi-source sensing nodes, and perform spatiotemporal alignment according to component identification, spatial location and acquisition time to form component-level status data; Bimodal visual data is extracted from the component-level state data, and candidate region identification and visual feature extraction are performed to obtain hazard candidate regions, bimodal visual features and initial hazard states. The corresponding non-visual state features are then extracted using the hazard candidate regions as indexes. The bimodal visual features and the non-visual state features are input into the Transformer model for cross-modal correlation to obtain fused state features and fused potential state. The fused state features of the same tower component at consecutive time nodes are input into the LSTM model to obtain the degradation trend. Compare the initial hazard state and the fused hazard state. When the two are consistent and the degradation trend is consistent with the change direction corresponding to the fused hazard state, determine the current hazard state. When the two are inconsistent or the change direction is inconsistent, extract the corresponding data from the component-level state data according to the hazard candidate region to perform cross-modal state determination, and determine the current hazard state based on the determination result. Based on the current hazard status and the degradation trend, a status assessment result and a lifespan prediction result are generated, and a graded early warning result is determined based on the status assessment result and the lifespan prediction result.

2. The intelligent detection method for safety hazards of iron towers according to claim 1, characterized in that, The multi-source sensing node consists of a drone, fixed sensors, and a mobile detection unit. The drone collects visual data and spatial morphology data of the tower, the fixed sensors collect structural response data and environmental data of the tower, and the mobile detection unit collects supplementary detection data of designated tower components. The multi-source sensing nodes record the acquisition time according to a unified time base, and bind the data collected by the UAV, the fixed sensor and the mobile detection unit to the corresponding component identifier and spatial location, so that the multi-modal sensing data of the iron tower from different sources are spatiotemporally aligned according to the same iron tower component.

3. The intelligent detection method for safety hazards of iron towers according to claim 1, characterized in that, The bimodal visual data consists of spatiotemporally aligned visible light images and infrared images. The visible light images and infrared images are input into the YOLOv8s model. Based on the location of the tower components and the location of the hidden dangers output by the YOLOv8s model, the candidate areas of the hidden dangers are determined. The visible light image and the infrared image corresponding to the candidate hazard region are input into the ResNet-18 model for feature extraction and state recognition to obtain the bimodal visual features and the initial hazard state. Based on the component identifier corresponding to the candidate hazard region and the acquisition time, the non-visual state features corresponding to the candidate hazard region are extracted from the component-level state data.

4. The intelligent detection method for safety hazards of iron towers according to claim 1, characterized in that, The bimodal visual features and non-visual state features are matched according to the component identifier and the acquisition time. The feature dimensions of the matched bimodal visual features and non-visual state features are unified and arranged according to the modality source to form a multimodal feature sequence. The multimodal feature sequence is input into the Transformer model. The correlation between each modal feature and the current state of the tower component is determined through the attention association between different modal features. The fusion state feature is generated based on the correlation. The fusion potential state is determined by the state output of the Transformer model.

5. The intelligent detection method for safety hazards of iron towers according to claim 4, characterized in that, Arrange multiple fusion state features corresponding to the same tower component in chronological order of the acquisition time to form a component state time sequence; The component state time series is input into the LSTM model, which recursively associates the state memory of the previous time node with the fused state features of the current time node, and determines the degradation trend based on the state change direction of consecutive time nodes.

6. The intelligent detection method for safety hazards of iron towers according to claim 1, characterized in that, The initial hidden danger state and the fused hidden danger state are compared for consistency in state category, and the change direction of the degradation trend characterization is compared for consistency in trend with the state change direction corresponding to the fused hidden danger state. When the initial hidden danger state is consistent with the state category of the merged hidden danger state, and the direction of change of the degradation trend is consistent with the direction of change of the state corresponding to the merged hidden danger state, the merged hidden danger state is determined as the current hidden danger state. When the state categories are inconsistent or the directions of change are inconsistent, the corresponding hidden danger candidate area is determined as the area to be verified.

7. The intelligent detection method for safety hazards of iron towers according to claim 6, characterized in that, For the area to be verified, the bimodal visual data and the non-visual state features are re-extracted from the component-level state data according to the corresponding component identifier and the acquisition time, and visual features are extracted from the re-extracted bimodal visual data to obtain bimodal visual features for verification. The bimodal visual features and the non-visual state features used for verification are input into the Transformer model to perform cross-modal state determination, obtain the verified hazard status, and update the current hazard status based on the verified hazard status.

8. The intelligent detection method for safety hazards of iron towers according to claim 1, characterized in that, The hazard category of the tower component is determined based on the current hazard status, and the corresponding state change direction is determined based on the degradation trend. The hazard category and the state change direction are correlated to form the state assessment result. Based on the fusion state characteristics, current hidden danger state, and degradation trend of the same tower component at continuous time nodes, state evolution prediction is performed to obtain the life prediction result.

9. The intelligent detection method for safety hazards of iron towers according to claim 8, characterized in that, Based on the status assessment results, candidate warning levels are determined, and the lifetime prediction results are compared with the warning conditions corresponding to the candidate warning levels. When the life prediction result meets the warning conditions, the candidate warning level is determined as the graded warning result. When the life prediction result does not meet the warning conditions, adjacent warning levels are selected according to the preset order of risk level corresponding to the warning level, and the life prediction result is compared with the warning conditions corresponding to the adjacent warning level until the corresponding warning conditions are met or the boundary of the preset order is reached. The graded warning result is determined based on the comparison result.

10. An intelligent detection system for safety hazards in iron towers, the system being used to implement the intelligent detection method for safety hazards in iron towers as described in any one of claims 1-9, characterized in that, The system includes a multimodal data processing module, a visual recognition module, a fusion time series analysis module, a state determination module, and a prediction and early warning module; The multimodal data processing module acquires multimodal sensing data of the tower collected by multi-source sensing nodes, and performs spatiotemporal alignment according to component identification, spatial location and acquisition time to form component-level status data; The visual recognition module extracts bimodal visual data from the component-level state data, performs candidate region recognition and visual feature extraction, obtains hazard candidate regions, bimodal visual features and initial hazard states, and extracts corresponding non-visual state features using the hazard candidate regions as indexes. The fusion time series analysis module inputs the dual-modal visual features and the non-visual state features into the Transformer model to perform cross-modal correlation, thereby obtaining fusion state features and fusion potential state. It also inputs the fusion state features of the same tower component at consecutive time nodes into the LSTM model to obtain the degradation trend. The state determination module compares the initial hidden danger state and the fused hidden danger state. When the two are consistent and the degradation trend is consistent with the change direction corresponding to the fused hidden danger state, the current hidden danger state is determined. When the two are inconsistent or the change direction is inconsistent, the corresponding data is re-extracted from the component-level state data according to the hidden danger candidate region to perform cross-modal state determination, and the current hidden danger state is determined according to the determination result. The prediction and early warning module generates a status assessment result and a life prediction result based on the current hidden danger status and the degradation trend, and determines a graded early warning result based on the status assessment result and the life prediction result.

Images