Active fine management and predictive operation and maintenance system based on multi-modal sensor fusion and "digital twin pole"

Abstract

The application discloses an active fine management and predictive operation system based on multi-modal sensing fusion and a digital twin pole, relates to the technical field of city intelligent operation management, and comprises a multi-modal data acquisition module, a multi-modal data fusion module, a digital twin pole construction module, a predictive operation module, a fine management module and a visual management and control platform; through spatiotemporal alignment and feature-level fusion of multi-modal data, the application breaks the data island among different modules, realizes comprehensive correlation analysis of multi-dimensional data, realizes real-time mapping and bidirectional interaction of physical entities and virtual models by constructing a single-pole full-factor digital twin mirror image, provides intuitive visual support for equipment full-life cycle management, judges deterioration trends and gives early warning of faults through an equipment health evaluation model, changes passive repair to active predictive maintenance, and improves system operation stability and public service continuity.

Description

Technical Field

[0001] This invention relates to the field of urban smart operation and maintenance management technology, specifically to a proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and "digital twin poles". Background Technology

[0002] With the deepening of smart city construction, smart streetlights, as one of the core infrastructures for urban digital transformation, have evolved from traditional single-function lighting to multi-functional composite nodes. Smart streetlights are composite public facilities built upon traditional municipal streetlights, integrating technologies such as IoT sensing, edge computing, and wireless communication. They incorporate multiple functional modules including environmental monitoring, security monitoring, information dissemination, 5G micro base stations, new energy vehicle charging stations, and emergency broadcasting. Leveraging the inherent advantages of wide coverage, even distribution, and stable power supply in urban roads, they serve as the peripheral nerves of the smart city's sensing network. They can collect various basic data on urban operations around the clock, providing data support for multiple fields such as traffic management, environmental governance, public safety, and public services. They are an important carrier for achieving refined urban governance and improving the quality of public services. Currently, most cities have launched large-scale smart streetlight renovation and construction projects, and the number and coverage of smart pole clusters continue to expand. Their operational stability and management efficiency are directly related to the normal operation of urban public services.

[0003] Digital twin poles are a typical application of digital twin technology in the field of smart infrastructure, and are the product of digitally mapping the entire lifecycle of physical smart street light poles. They construct a virtual mirror that corresponds one-to-one with the physical entity, synchronizes in real time, and allows for two-way interaction by performing high-precision digital modeling of all elements of the physical pole, including its hardware structure, component composition, spatial coordinates, operating parameters, and historical status. Compared to traditional digital ledger management, digital twin poles can intuitively and dynamically present the entire operational picture of the equipment, supporting various simulation operations such as fault simulation, strategy deduction, and performance evaluation. This provides a core technological foundation for the transformation of smart street light management from a "post-event handling" to a "pre-event prevention" model. Currently, digital twin pole technology is still in its initial application stage. Most existing solutions only achieve static modeling and basic data display of the physical pole, and have not yet deeply integrated with multi-source sensor data, making it difficult to realize its core value in predictive maintenance and refined management.

[0004] Existing smart street light operation and maintenance management systems still have certain shortcomings. They suffer from significant data silos, with different modal data such as video, environmental, and equipment electrical parameters belonging to different subsystems. This lack of a unified fusion and processing mechanism prevents multi-dimensional comprehensive analysis, resulting in a passive and outdated operation and maintenance model that relies primarily on manual inspections and fault reporting. This makes it difficult to predict equipment degradation trends in advance, leading to low fault handling efficiency. Energy consumption management is also inefficient, with a single lighting control strategy that cannot dynamically adjust output power based on real-time scenarios, resulting in substantial energy waste. Furthermore, the lack of global control capabilities prevents unified operational assessment and resource scheduling of smart pole clusters within a region, leading to high operation and maintenance costs. Therefore, developing a proactive, refined management and predictive operation and maintenance system based on multi-modal sensor fusion and "digital twin poles" is of great significance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and "digital twin poles". It can break down data silos between different modules through spatiotemporal alignment and feature-level fusion of multimodal data, realize comprehensive correlation analysis of multi-dimensional data, and achieve real-time mapping and bidirectional interaction between physical entities and virtual models by constructing a single pole full-element digital twin image. This provides intuitive visualization support for the full life cycle management of equipment and enables the assessment of degradation trends and early warning of faults through equipment health assessment models.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a proactive refined management and predictive operation and maintenance system based on multimodal sensor fusion and "digital twin poles", the system comprising: a multimodal data acquisition module, a multimodal data fusion module, a digital twin pole construction module, a predictive operation and maintenance module, a refined management module, and a visual control platform;

[0007] The multimodal data acquisition module is deployed on the physical smart pole body to collect video images, environmental parameters, equipment electrical parameters and spatial location data;

[0008] The multimodal data fusion module receives the raw data transmitted by the multimodal data acquisition module, performs spatiotemporal alignment preprocessing, extracts feature vectors of different modalities, and generates a unified fusion feature dataset.

[0009] The digital twin pole construction module constructs a single pole full-element digital twin image based on the fused feature dataset, and establishes a two-way data transmission channel between the physical entity and the virtual model;

[0010] The predictive maintenance module uses historical fusion feature datasets and real-time fusion feature datasets to train a device health assessment model and generate predictive maintenance work orders.

[0011] The refined management module performs simulation based on a digital twin image to generate lighting control commands and inspection paths;

[0012] The visualization management and control platform integrates functions such as equipment status monitoring, fault early warning reception, operation and maintenance work order management, and energy consumption statistical analysis.

[0013] Furthermore, the multimodal data fusion module performs the following operations during data fusion:

[0014] It receives raw data transmitted from the multimodal data acquisition module, performs timestamp calibration and spatial coordinate matching on different modal data, and completes spatiotemporal alignment preprocessing;

[0015] Visual features of video images, temporal features of environmental parameters, frequency domain features of device electrical parameters, and geographical features of spatial location are extracted respectively to generate feature vectors corresponding to each mode;

[0016] The adaptive fusion weights of the feature vectors of each modality are calculated based on the information entropy of each modality data. A unified-dimensional fused feature dataset is generated using the adaptive weighted feature fusion formula, which is as follows: ,in, To fuse feature vectors, where n is the number of modalities. Let i be the eigenvector of the i-th mode. Let i be the fusion weights for the i-th mode. It is determined by the ratio of the information entropy of the i-th modal data to the sum of the information entropies of all modal data.

[0017] Furthermore, the digital twin pole construction module performs the following operations when constructing a single pole digital twin image:

[0018] Import the 3D structural model and hardware component parameters of the smart pole to establish a basic static model;

[0019] Receive the fused feature dataset output by the multimodal data fusion module and map the fused features to the corresponding component nodes of the basic static model;

[0020] Establish a two-way data transmission channel between physical entities and virtual models to achieve real-time synchronization between the operating status of physical entities and the display status of virtual models.

[0021] Furthermore, the predictive maintenance module performs the following operations when generating a predictive maintenance work order:

[0022] Retrieve historical fusion feature datasets and corresponding equipment fault records to train an equipment health assessment model;

[0023] The real-time fused feature dataset is input into the trained equipment health assessment model to extract multi-dimensional operational features of each key component. The degradation degree values ​​of the power supply, drive, and sensing key components are output using the component degradation degree calculation formula: ,in, Let be the degradation level value of the j-th critical component, and m be the corresponding operational characteristic dimension of that component. For the j-th component, the k-th operational characteristic value is... The degradation weight corresponding to the k-th operational feature of the j-th component. The correlation coefficient between this feature and the occurrence of component failures is determined from historical failure data;

[0024] Based on the degradation level, a corresponding maintenance level is matched to generate a predictive maintenance work order that includes the device location, fault type, and suggested handling method.

[0025] Furthermore, the video acquisition unit of the multimodal data acquisition module uses a high-definition network camera, deployed on the upper part of the smart pole, to collect video image data of road pedestrian and vehicle traffic and the surrounding environment of the equipment. The environmental sensing unit integrates temperature and humidity sensors, air pressure sensors, and wind speed sensors, deployed in the middle of the smart pole, to collect meteorological parameters of the surrounding environment. The equipment status monitoring unit connects to the power module, drive module, and various sensor modules of the smart pole, and collects the operating voltage, operating current, and operating time data of each module. The positioning unit uses a satellite positioning module to collect the latitude and longitude coordinates and altitude data of the smart pole.

[0026] Furthermore, the multimodal data fusion module is deployed on an edge computing node, which is integrated inside the smart pole. It performs local preprocessing and feature-level fusion on the collected raw data, and transmits the generated fused feature dataset to a cloud server for storage and subsequent processing. The cloud server stores all historical operating data, fault records and digital twin model data of the smart pole, providing data support for each module.

[0027] Furthermore, the digital twin pole construction module constructs a single pole full-element digital twin image that includes a three-dimensional model and operating status parameters of the pole structure, power supply components, drive components, sensing components, and communication components. Each component corresponds to a unique virtual node. The virtual node stores the historical and real-time operating data of the corresponding physical component. The virtual node supports the modification of component parameters and the simulation of operating status, providing a basis for strategy deduction.

[0028] Furthermore, the refined management module performs simulation based on the digital twin pole image. According to real-time lighting conditions, pedestrian and vehicle flow data, and environmental parameters, it generates lighting output power adjustment instructions and sends the adjustment instructions to the corresponding smart pole's drive module. The drive module adjusts the output power of the lighting fixtures according to the instructions. The refined management module divides lighting control units by region and performs unified lighting strategy control on smart poles within the same unit.

[0029] Furthermore, the refined management module generates the optimal inspection path based on the predictive maintenance work orders of all smart poles in the area and the real-time location of maintenance personnel. The inspection path is then sent to the mobile terminal of the maintenance personnel. At the same time, the maintenance task display status of the visual management and control platform is updated. The refined management module supports the priority adjustment of maintenance tasks and reallocates maintenance resources according to the urgency of the fault.

[0030] Furthermore, the visualization and control platform provides a three-dimensional digital twin scene interface, which displays the location and operating status of all smart poles in the area. Smart poles with different operating statuses are identified by different colors. The platform receives and displays fault warning information and predictive maintenance work orders, supports the dispatch, tracking and closed-loop management of maintenance work orders, and generates daily, weekly and monthly energy consumption statistics reports and maintenance statistics reports.

[0031] Compared with existing technologies, this proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin poles" has the following beneficial effects:

[0032] This invention breaks down data silos between different modules through spatiotemporal alignment and feature-level fusion of multimodal data, enabling comprehensive correlation analysis of multidimensional data. By constructing a single-pole full-element digital twin mirror image, it achieves real-time mapping and bidirectional interaction between physical entities and virtual models, providing intuitive visualization support for equipment lifecycle management. Through equipment health assessment models, it analyzes degradation trends and provides early warnings of faults, transforming passive emergency repairs into proactive predictive maintenance, improving system operational stability and public service continuity. Through digital twin simulation, it dynamically optimizes lighting strategies and maintenance paths, achieving refined control of energy consumption and efficient scheduling of maintenance resources, significantly reducing energy consumption and manual maintenance costs.

[0033] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0035] Figure 1 This is a schematic diagram of the structure of a proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and a "digital twin pole";

[0036] Figure 2 This is a flowchart of a proactive, refined management and predictive maintenance system based on multimodal sensor fusion and a "digital twin pole".

[0037] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0038] The technical solution protected by this invention is a proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and digital twin poles. This system relies on smart pole hardware to build a complete technical architecture. Through a multimodal data acquisition module, it acquires four core data types from the physical smart pole: video images, environmental parameters, equipment electrical parameters, and spatial location. The multimodal data fusion module extracts spatiotemporal alignment features and performs adaptive weighted feature fusion to form a unified fused feature dataset. The digital twin pole construction module builds a full-element digital twin image of a single pole based on this fused feature dataset, enabling real-time bidirectional data interaction between the physical entity and the virtual model. The predictive maintenance module relies on historical and real-time fusion data to train the equipment health assessment model, calculate the degradation degree of key components, and generate predictive maintenance work orders. The refined management module completes lighting strategy simulation and optimal inspection path planning based on digital twin mirrors. The visual control platform integrates equipment monitoring, fault early warning work order management, and energy consumption statistics functions to achieve full-domain visual control of smart pole clusters. The entire technical solution deeply combines multimodal data fusion and digital twin technology to solve the problems of data silos, passive maintenance, and scattered energy consumption control in traditional smart pole maintenance, and builds a closed-loop management system for the entire process from data collection, fusion, and modeling to predictive maintenance and refined control.

[0039] This embodiment addresses the daily operation and management of smart pole clusters along main roads in urban core areas. It fully implements a proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and digital twin poles. The overall hardware and software architecture of the system is as follows: Figure 1 As shown, the execution logic of the entire process is as follows: Figure 2As shown, the implementation process strictly follows the modular collaborative operation rules, and sequentially completes system deployment, data collection, data fusion, digital twin modeling, predictive operation and maintenance generation, refined management and control, execution, and visualization platform presentation of all aspects of the operation, ensuring that the system functions are fully implemented and adapted to the actual operation and maintenance needs of smart poles on urban roads.

[0040] This embodiment selects a cluster of smart poles along a six-lane dual carriageway in the city's core area as the implementation platform. These smart poles serve multiple functions, including road lighting, traffic monitoring, environmental sensing, and public services, and are core peripheral nodes in the city's smart city sensing network. In the initial implementation phase, the physical deployment of the multimodal data acquisition modules is completed. The acquisition units are fixedly installed on each physical smart pole according to functional zones, ensuring that the acquisition range of each unit covers the target area without any data acquisition blind spots.

[0041] The video acquisition unit uses a high-definition network camera, fixedly mounted on the cantilever of the physical smart pole. The lens faces the main road and sidewalk area, continuously collecting real-time video image data of pedestrian and vehicle traffic, the surrounding environment of the smart pole, and the status of roadside facilities. This unit continuously outputs visual data at a fixed frame rate, providing basic visual information for subsequent scene perception equipment status recognition. The environmental sensing unit integrates temperature, humidity, air pressure, and wind speed sensors, sealed within a protective chamber in the middle of the physical smart pole to prevent data deviation caused by outdoor environmental interference. It collects real-time air temperature, humidity, atmospheric pressure, wind speed, and meteorological parameters around the road, accurately reflecting the external environmental conditions for the smart pole's operation and providing environmental data support for lighting control and environmental monitoring.

[0042] The equipment status monitoring unit connects to the power module, drive module, and various sensor modules of the physical smart pole via wired connection. It continuously collects data on the operating voltage, current, continuous operating time, and internal temperature of each functional module, directly reflecting the operating status of the smart pole's core hardware and serving as a core data source for equipment health assessment. The positioning unit uses a satellite positioning module embedded within the central control unit of the smart pole. It receives satellite signals in real time, acquiring the latitude, longitude, and altitude coordinates of each smart pole to determine its precise spatial location, providing geographic information for subsequent digital twin modeling and inspection path planning.

[0043] After all acquisition units are deployed, they are synchronously debugged to ensure that the clocks of each unit are transmitted stably and uniformly. The four types of raw data are transmitted in real time to the multimodal data fusion module through the communication module built into the smart pole, completing the raw acquisition and transmission of multimodal data and providing a complete raw data foundation for subsequent data processing.

[0044] This embodiment deploys a multimodal data fusion module within an edge computing node integrated inside the smart pole. Leveraging the low-latency local processing capabilities of edge computing, it completes preprocessing and feature-level fusion of raw data, avoiding network congestion caused by directly transmitting large amounts of raw data to the cloud, while simultaneously improving the real-time performance of data processing. The data fusion process consists of three core steps: spatiotemporal alignment preprocessing, feature vector extraction, and adaptive weighted feature fusion.

[0045] First, a spatiotemporal alignment preprocessing operation is performed. After receiving four types of raw data, the multimodal data fusion module first uniformly calibrates the timestamps of all data to eliminate time deviations caused by hardware differences between the video acquisition unit, environmental sensing unit, device status monitoring unit, and positioning unit, ensuring that all data correspond to the same time node. Subsequently, spatial coordinate matching is performed on the data, accurately associating the image coordinates of the video image, the acquisition location of the environmental parameters, the device's geographical location information of the electrical parameter data, and ensuring that all data point to the same spatial location of the same physical smart pole. This completes the spatiotemporal unification of different modal data and eliminates spatiotemporal dimensional errors for subsequent feature fusion.

[0046] Next, multimodal feature vector extraction is performed. Specific features are extracted for different types of raw data to form independent feature vectors. For video image data, visual feature extraction algorithms are used to extract visual features such as target contours, pedestrian density, and vehicle speed, generating video modal feature vectors. For environmental parameter data, time-series analysis algorithms are used to extract the continuous variation trends of temperature, humidity, air pressure, and wind speed, generating environmental modal feature vectors. For equipment electrical parameter data, frequency domain analysis algorithms are used to extract the fluctuation frequency features of operating voltage and current, generating equipment electrical parameter modal feature vectors. For spatial location data, geographic information parsing algorithms are used to extract latitude, longitude, and altitude features, generating location modal feature vectors. These four types of feature vectors correspond to different dimensions of equipment and environmental information, providing standardized feature data for subsequent fusion.

[0047] In the specific implementation of this embodiment, adaptive weighted feature fusion calculation is carried out. The four types of feature vectors are integrated into a unified dimension fusion feature dataset through the adaptive weighted feature fusion formula, which is expressed as follows: ,in This represents the final generated fused feature vector, where n represents the total number of modalities participating in the fusion. In this embodiment, n is taken as the value of the four modalities of video environment device electrical parameter positioning. This represents the feature vector corresponding to the i-th modality data. Let represent the fusion weight of the i-th modal feature vector, with a value range of (0, 1), and satisfy . .

[0048] The fusion weights are determined using information entropy quantification, and the specific calculation process is as follows:

[0049] Single-modal information entropy calculation: for the i-th type of modal data First calculate the probability distribution of the data. Then through the formula Calculate the modal information entropy Information entropy threshold range: , The higher the value, the greater the amount of information in the modality data;

[0050] Total information entropy calculation: Calculate the sum of information entropy for all modalities. ;

[0051] Weight allocation: through formula Calculate adaptive fusion weights. When the single-modal information entropy is lower than the threshold of 0.5, automatically assign the lowest weight of 0.1 to ensure data validity.

[0052] Through this quantification method, the system can automatically allocate weights based on the actual information value of each modality's data without manual intervention, achieving adaptive optimization of feature-level fusion and ensuring that the fused feature dataset fully retains the core effective information of each modality.

[0053] The unified feature dataset after fusion is transmitted to the cloud server through the smart pole's communication module. The cloud server builds a dedicated database to uniformly store all the smart pole's historical operation data, historical fault records, digital twin model data, and fused feature dataset, providing continuous and stable data support for building a predictive operation and maintenance refined management module for the digital twin pole.

[0054] After the multimodal data fusion is completed, the digital twin pole construction module is launched. Based on the fused feature dataset, a full-element digital twin image of a single pole is built to realize real-time mapping and two-way interaction between the physical smart pole and the virtual model. The construction process is divided into three steps: basic static model construction, fused feature mapping, and establishment of two-way data channels.

[0055] First, a basic static model for the digital twin is built. The module imports the 3D structural model of the physical smart pole and all hardware component parameters. The 3D structural model completely restores the physical form of the pole's shape, cantilever layout, installation position, and physical form. The hardware component parameters include the model specifications, assembly relationships, and operating parameters of the power supply components, drive components, sensor components, and communication components. Based on the above data, a static virtual model that is completely consistent with the physical smart pole is constructed. This model provides the basic carrier for the digital twin mirror image.

[0056] Subsequently, a feature mapping operation is performed. The digital twin pole construction module receives the fused feature dataset output by the multimodal data fusion module and precisely maps the fused features to the corresponding component nodes of the basic static model according to data type and hardware affiliation. Each physical hardware component corresponds to a unique virtual node in the virtual model. The virtual node is synchronously bound to the component's historical running data, real-time running data, and fused feature data, realizing a one-to-one correspondence between the virtual model and the physical component's data.

[0057] Finally, a bidirectional data transmission channel was established between the physical entity and the virtual model. Real-time operational data from the physical smart pole is continuously transmitted to the virtual model via the acquisition and fusion modules. The virtual model synchronously updates component status and operating parameters, achieving a forward mapping from the physical entity to the virtual model. Simultaneously, the virtual model supports simulation operations for modifying component parameters and operating status. Simulation results can be transmitted back to the control unit of the physical smart pole via the bidirectional channel, achieving a reverse interaction between the virtual model and the physical entity. The completed single-pole full-element digital twin image fully includes 3D models and full-cycle operating parameters of the pole's structure, power supply components, drive components, sensing components, and communication components, providing a visualization and simulation foundation for subsequent predictive maintenance simulation and refined management.

[0058] Once the digital twin mirror is built, the predictive maintenance module is activated. Based on historical and real-time fused data, equipment health assessments are conducted to achieve early warning of faults and generation of predictive maintenance work orders. The operation process consists of three steps: health model training, component degradation degree calculation, and maintenance work order generation.

[0059] First, the equipment health assessment model is trained. The module retrieves historical fusion feature datasets and corresponding equipment fault records stored on the cloud server. Using this historical data as training samples, machine learning algorithms are employed to train and iteratively optimize the model. During training, the model continuously learns the characteristic patterns of different states of equipment operation and early warning faults, constantly optimizing its recognition accuracy. Ultimately, this results in a health assessment model capable of accurately judging equipment degradation trends and identifying potential faults.

[0060] In the specific implementation of this embodiment, the degradation degree of key components is calculated. The real-time fused feature dataset is input into the trained equipment health assessment model. The model automatically extracts multi-dimensional operating features of three core key components: power supply, drive, and sensing. The degradation degree of each component is calculated using the component degradation degree calculation formula, which is expressed as follows: ,in represents the degradation level of the j-th critical component, with a value range of [0, 100], and m represents the total number of operational feature dimensions corresponding to this critical component. The value representing the k-th operational characteristic of the j-th component, after normalization, has a range of [0, 1]. The degradation weight represents the k-th operational characteristic of the j-th component, and its value ranges from (0, 1), and satisfies the following conditions: .

[0061] The degradation weight is determined using the Pearson correlation coefficient quantification method. The specific calculation process and threshold settings are as follows:

[0062] Pearson correlation coefficient calculation: For the k-th operating characteristic of the j-th component and the fault occurrence state, the correlation coefficient is calculated using the formula... Calculate the correlation coefficient , The value range is [-1, 1];

[0063] Weight threshold determination: when When it is determined to be a strong correlation, it is assigned... ;when When it is determined to be moderately relevant, it is assigned... ;when When it is determined to be a weak correlation, it is assigned... ;

[0064] Weight normalization: After converting all correlation coefficients into degraded weights, normalization is performed to ensure that the sum of the weights is 1.

[0065] This quantitative method ensures that the calculation of the degree of degradation closely matches the actual operating patterns of the equipment, guaranteeing that the assessment results accurately reflect the true degradation state of the components.

[0066] After calculating the degradation level of each key component, the module is matched with the corresponding maintenance level according to the degradation level value. The specific correspondence between degradation level and maintenance level is as follows:

[0067] Basic maintenance (general hidden dangers): Deterioration level value The components are in good working order and only require routine inspection; no immediate action is needed.

[0068] Intermediate Maintenance (Early Warning of Potential Risks): Deterioration Level Value The component shows a slight deterioration trend and preventive maintenance should be arranged within 72 hours;

[0069] Advanced Maintenance (Emergency Faults): Degradation Level Value The components are severely deteriorated, posing an immediate risk of failure, and on-site repairs are required within 2 hours.

[0070] The higher the degree of degradation, the higher the maintenance level and the higher the processing priority. The system automatically generates predictive maintenance work orders, which include the precise location of the smart pole, the type of faulty component, the degree of degradation, the suggested handling method, and other core information. The work orders are transmitted in real time to the refined management module and the visual control platform, enabling early warning of equipment failures and transforming reactive emergency repairs into proactive predictive maintenance.

[0071] The refined management module is based on digital twin images and predictive maintenance work orders to carry out refined control of lighting and optimized scheduling of maintenance resources. The execution of the operation is divided into three stages: lighting strategy control, inspection path planning, and maintenance resource scheduling.

[0072] The lighting strategy control relies on real-time data from a digital twin mirror, combined with real-time lighting conditions, pedestrian and vehicle traffic data, and environmental parameters. It conducts lighting operation simulations in a virtual model, simulating road lighting effects and energy consumption data under different lighting powers, and generating lighting output power adjustment commands adapted to the current scenario. These commands are sent to the corresponding smart pole's drive module via a communication network. The drive module dynamically adjusts the output power of the lighting fixtures according to the commands, achieving on-demand lighting. Simultaneously, the system divides lighting control units according to the functional areas of urban roads and administrative regions, implementing unified lighting strategy control for all smart poles within the same control unit. This avoids uneven lighting caused by independent control of individual poles, achieving refined energy consumption management and reducing energy waste.

[0073] The inspection route planning integrates predictive maintenance work orders from all smart poles within the area, combined with the real-time location, mobile terminal status, and maintenance skill information of maintenance personnel. It employs a combined ant colony algorithm and Dijkstra's algorithm to generate the optimal inspection route: Dijkstra's algorithm calculates the shortest path to a single point, while the improved ant colony algorithm optimizes the inspection sequence of multiple nodes, balancing the principles of shortest distance, handling the highest priority faults first, and minimizing detours to maximize maintenance efficiency. The planned inspection route is sent to the maintenance personnel's mobile terminals in real time, simultaneously updating the maintenance task display status on the visual management platform, achieving real-time synchronization of maintenance tasks.

[0074] The operation and maintenance resource scheduling supports dynamic adjustment of task priorities. The system automatically adjusts the priority of operation and maintenance tasks according to the urgency and degradation level of the fault. High-level faults are given priority in the allocation of operation and maintenance personnel and equipment resources. At the same time, it supports manual adjustment of priorities by managers to ensure that sudden faults and major hidden dangers are dealt with quickly and improve the utilization efficiency of operation and maintenance resources.

[0075] The visualization management and control platform serves as the unified management and control entry point for the system. It integrates the functional data of all modules to achieve full-domain visualization management and control of the smart pole cluster. The platform's functions are divided into four parts: 3D scene display, fault early warning reception, work order closed-loop management, and data report generation.

[0076] The platform provides a 3D digital twin scene interface that fully restores the geographical environment of the city's main roads and the layout of the smart pole cluster. It accurately displays the spatial location and real-time operating status of each smart pole and uses different colors to indicate the operating status of the smart poles, intuitively distinguishing between normal operation, early warning status, fault status, and maintenance pending status. Managers can quickly grasp the operating status of all devices in the area through the interface.

[0077] The platform receives fault warnings and maintenance work orders from the predictive maintenance module in real time, and displays high-level warning information via pop-up windows to ensure that managers are promptly aware of potential equipment hazards. Simultaneously, the platform supports closed-loop management of maintenance work order dispatch, tracking, and confirmation. From work order generation and dispatch to execution and archiving, the entire process is traceable and monitorable, achieving standardized and regulated maintenance management.

[0078] The platform automatically collects energy consumption data, operation and maintenance data, and fault data of smart poles. It automatically generates energy consumption statistics reports, operation and maintenance statistics reports, and fault analysis reports according to the time dimensions of daily, weekly, and monthly. The reports include core data such as total energy consumption, energy consumption comparison, operation and maintenance completion rate, and fault occurrence rate, providing accurate data support for the formulation, optimization, and adjustment of urban smart pole management policies.

[0079] In summary, this embodiment, through the complete implementation of a proactive, refined management and predictive maintenance system based on multimodal sensor fusion and digital twin poles, strictly adheres to... Figure 1 System architecture and Figure 2 The system successfully achieves intelligent management and control of the entire process of smart pole clusters along urban main roads. At the data processing level, multimodal data spatiotemporal alignment and adaptive weighted fusion completely break down the data silos of traditional systems, enabling comprehensive correlation analysis of multi-dimensional data on the spatial location of video environment equipment electrical parameters, thus enhancing data utilization value. At the digital modeling level, the full-element digital twin mirror of a single pole enables real-time bidirectional interaction between the physical entity and the virtual model, providing intuitive visualization support for the entire lifecycle management of equipment and supporting simulation and strategy modeling. At the operation and maintenance level, the equipment health assessment model and quantitative calculation of degradation degree enable early prediction of equipment failures, completely changing the passive operation and maintenance mode of traditional manual inspection and fault reporting, and significantly improving system stability and public service continuity. At the management efficiency level, dynamic lighting control enables refined energy consumption management, and optimal inspection paths and resource scheduling significantly reduce operation and maintenance costs. The entire system effectively solves various pain points in traditional smart pole operation and maintenance management, providing a replicable and scalable technical solution for proactive and refined management and predictive operation and maintenance of smart city infrastructure.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A proactive, refined management and predictive maintenance system based on multimodal sensor fusion and a "digital twin pole," characterized in that: The system includes: a multimodal data acquisition module, a multimodal data fusion module, a digital twin pole construction module, a predictive operation and maintenance module, a refined management module, and a visual control platform; The multimodal data acquisition module is deployed on the physical smart pole body to collect video images, environmental parameters, equipment electrical parameters and spatial location data; The multimodal data fusion module receives the raw data transmitted by the multimodal data acquisition module, performs spatiotemporal alignment preprocessing, extracts feature vectors of different modalities, and generates a unified fusion feature dataset. The digital twin pole construction module constructs a single pole full-element digital twin image based on the fused feature dataset, and establishes a two-way data transmission channel between the physical entity and the virtual model; The predictive maintenance module uses historical fusion feature datasets and real-time fusion feature datasets to train a device health assessment model and generate predictive maintenance work orders. The refined management module performs simulation based on a digital twin image to generate lighting control commands and inspection paths; The visualization management and control platform integrates functions such as equipment status monitoring, fault early warning reception, operation and maintenance work order management, and energy consumption statistical analysis.

2. The proactive, refined management and predictive operation and maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The multimodal data fusion module performs the following operations during data fusion: It receives raw data transmitted from the multimodal data acquisition module, performs timestamp calibration and spatial coordinate matching on different modal data, and completes spatiotemporal alignment preprocessing; Visual features of video images, temporal features of environmental parameters, frequency domain features of device electrical parameters, and geographical features of spatial location are extracted respectively to generate feature vectors corresponding to each mode; The adaptive fusion weights of the feature vectors of each modality are calculated based on the information entropy of each modality data. A unified-dimensional fused feature dataset is generated using the adaptive weighted feature fusion formula, which is as follows: ,in, To fuse feature vectors, where n is the number of modalities. Let i be the eigenvector of the i-th mode. Let be the fusion weight for the i-th mode.

3. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The digital twin pole construction module performs the following operations when constructing a single-pole digital twin image: Import the 3D structural model and hardware component parameters of the smart pole to establish a basic static model; Receive the fused feature dataset output by the multimodal data fusion module and map the fused features to the corresponding component nodes of the basic static model; Establish a two-way data transmission channel between physical entities and virtual models to achieve real-time synchronization between the operating status of physical entities and the display status of virtual models.

4. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The predictive maintenance module performs the following operations when generating a predictive maintenance work order: Retrieve historical fusion feature datasets and corresponding equipment fault records to train an equipment health assessment model; The real-time fused feature dataset is input into the trained equipment health assessment model to extract multi-dimensional operational features of each key component. The degradation degree values ​​of the power supply, drive, and sensing key components are output using the component degradation degree calculation formula: ,in, Let be the degradation level value of the j-th critical component, and m be the corresponding operational characteristic dimension of that component. For the j-th component, the k-th operational characteristic value is... This is the degradation weight corresponding to the kth operational characteristic of the jth component; Based on the degradation level, a corresponding maintenance level is matched to generate a predictive maintenance work order that includes the device location, fault type, and suggested handling method.

5. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The video acquisition unit of the multimodal data acquisition module uses a high-definition network camera, deployed on the upper part of the smart pole, to collect video image data of road pedestrian and vehicle traffic and the surrounding environment of the equipment. The environmental sensing unit integrates temperature and humidity sensors, air pressure sensors, and wind speed sensors, deployed in the middle of the smart pole, to collect meteorological parameters of the surrounding environment. The equipment status monitoring unit connects to the power module, drive module, and various sensor modules of the smart pole, and collects the operating voltage, operating current, and operating time data of each module. The positioning unit uses a satellite positioning module to collect the latitude and longitude coordinates and altitude data of the smart pole.

6. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The multimodal data fusion module is deployed on an edge computing node, which is integrated inside the smart pole. It performs local preprocessing and feature-level fusion on the collected raw data, and transmits the generated fused feature dataset to a cloud server for storage and subsequent processing. The cloud server stores all historical operating data, fault records and digital twin model data of the smart pole, providing data support for each module.

7. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The digital twin pole construction module constructs a single pole full-element digital twin image, which includes a 3D model and operating status parameters of the pole structure, power supply components, drive components, sensing components, and communication components. Each component corresponds to a unique virtual node. The virtual node stores the historical and real-time operating data of the corresponding physical component. The virtual node supports the modification of component parameters and the simulation of operating status, providing a basis for strategy deduction.

8. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The refined management module performs simulation based on the digital twin pole image. According to real-time lighting conditions, pedestrian and vehicle flow data and environmental parameters, it generates lighting output power adjustment instructions and sends the adjustment instructions to the corresponding smart pole's drive module. The drive module adjusts the output power of the lighting fixtures according to the instructions. The refined management module divides the lighting control units by area and performs unified lighting strategy control on the smart poles in the same unit.

9. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The refined management module generates the optimal inspection route based on the predictive maintenance work orders of all smart poles in the area and the real-time location of maintenance personnel. The inspection route is then sent to the mobile terminals of maintenance personnel. At the same time, the maintenance task display status of the visual management and control platform is updated. The refined management module supports the priority adjustment of maintenance tasks and reallocates maintenance resources according to the urgency of the fault.

10. The proactive, refined management and predictive maintenance system based on multimodal sensor fusion and "digital twin pole" as described in claim 1, characterized in that, The visualization and management platform provides a three-dimensional digital twin scene interface, which displays the location and operating status of all smart poles in the area. Smart poles with different operating statuses are identified by different colors. The platform receives and displays fault warning information and predictive maintenance work orders, supports the dispatch, tracking and closed-loop management of maintenance work orders, and generates daily, weekly and monthly energy consumption statistics reports and maintenance statistics reports.

Images