A multimodal data fusion intelligent inspection method for hydropower plants

Abstract

The present invention relates to the field of electrical digital data processing technology. The present invention discloses a multimodal data fusion intelligent inspection method for hydropower plants, comprising the following steps: determining the equipment inspection sequence with the shortest path based on the position data of the internal equipment of the hydropower plant to obtain an initial inspection route; planning the inspection route and updating the inspection route information based on the frequency of key data characteristics generated by the internal equipment of the hydropower plant and the specific positions between the equipment; obtaining the inspection point information on the inspection route during the inspection process, establishing an inspection point list, and collecting multimodal data of the equipment; preprocessing, feature extraction and fusion of the collected data, and identifying anomalies based on the real-time operating condition information of the hydropower plant. The present invention improves the inspection efficiency and the response speed of emergency tasks, reduces the missed detection rate, can significantly improve the flexibility and safety of hydropower plant inspections, and improves the anomaly detection coverage and reduces the false alarm rate through multimodal complementarity and operating condition adaptive analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of digital data processing for inspection of hydropower plants, and in particular to a multi-modal data fusion intelligent inspection method for hydropower plants. Background Art

[0002] During the operation of hydropower plant equipment, inspections are necessary to promptly detect potential equipment failures and safety hazards, ensuring safe and stable operation. However, manual inspections are currently labor-intensive and inefficient, and operators vary in their experience, leading to inconsistent inspection results. Therefore, robots are urgently needed to replace some of these manual inspection tasks.

[0003] CN117742334A proposes a robot-based intelligent inspection method and system for hydropower plants. This method uses equipment operating information, attributes, and location to determine inspection priorities and generate an initial inspection route. After the robot performs a single equipment inspection, it analyzes the inspection data. If no anomalies are detected, a secondary equipment inspection is performed based on the remaining route. If an anomaly is detected, the anomalous device is marked, data acquisition parameters are obtained, and the route is generated based on these parameters for a secondary equipment inspection. However, the inspection system employs a fixed inspection schedule and fails to flexibly adjust to the differences in the frequency of key data features generated by various equipment in the hydropower plant. Furthermore, the data collection process fails to consider the impact of ambient sound on vibration signals. Furthermore, when the hydropower plant's operating conditions fluctuate significantly, the lack of emergency inspections for key equipment can easily lead to missed equipment anomalies. Furthermore, the system cannot accurately distinguish between data feature changes caused by normal operating conditions and abnormal conditions, resulting in false alarms and requiring manual review or intervention.

[0004] CN119129953A proposes a hydropower plant safety inspection method and system based on blockchain technology. Blockchain technology is used to process and protect data, a particle swarm optimization algorithm is used to dynamically adjust inspection task allocation and path planning, and an improved dynamic time warping algorithm is used to match time series inspection data to identify potential equipment anomalies and problems. Finally, inspection adjustment suggestions are provided to achieve safe management, intelligent scheduling, and real-time early warning of hydropower plant inspection data. Its shortcomings are that it fails to make reasonable inspection plans based on the significant differences in the frequency of key data characteristics of various equipment in the hydropower plant and changes in operating conditions. It also fails to consider the problem of vibration signal acquisition under environmental sound interference, especially under high head and large flow conditions. Furthermore, the differences in data characteristics of core equipment under different flow rates and loads are not analyzed, resulting in deficiencies in data anomaly identification. Data review is required to avoid errors and omissions.

[0005] Therefore, while existing technologies can improve the efficiency of manual inspections and reduce errors and omissions in manually processing inspection data, they still have limitations. First, existing technologies fail to flexibly adjust inspection times and routes based on the differences in the frequency of key data characteristics generated by various equipment in hydropower plants. Second, during data collection, the presence of ambient sound can interfere with vibration signals, reducing the accuracy of anomaly identification. Third, when the operating conditions of a hydropower plant fluctuate significantly, there is a lack of emergency inspections of key equipment, and the technology fails to distinguish between data characteristic changes caused by normal operating conditions and actual data anomalies, increasing the false alarm rate. Summary of the Invention

[0006] To address current technical problems, the main purpose of this invention is to provide a multimodal data fusion intelligent inspection method for hydropower plants. This method addresses the problem that traditional fixed inspection modes are unable to adapt to dynamic changes in equipment data and unexpected operating conditions, improves inspection efficiency and emergency task response speed, reduces missed detection rates, and significantly enhances the flexibility and safety of hydropower plant inspections. By performing underwater acoustic interference decomposition on multi-point vibration signals, the accuracy of vibration signal anomaly detection is improved, providing reliable input for subsequent multimodal data fusion and anomaly identification. Through multimodal complementarity and adaptive operating condition analysis, the anomaly detection coverage rate is improved and the false alarm rate is reduced.

[0007] To overcome the problems of the prior art, the present invention adopts a technical solution: a multimodal data fusion intelligent inspection method for a hydropower plant, comprising the following steps:

[0008] S1. Determine the equipment inspection sequence with the shortest path based on the location data of the internal equipment of the hydropower plant to obtain the initial inspection route;

[0009] S2. When the number of inspections reaches the path update period, the inspection path is planned based on the frequency of key data characteristics of the hydropower plant's internal equipment and the specific locations between the equipment, and the inspection route information is updated. The inspection path planning includes regular inspection path planning and emergency inspection path planning, and regular inspection route information and emergency inspection route information are obtained respectively.

[0010] S3. During the inspection process, obtain information about inspection points along the inspection route, establish a list of inspection points, and collect multimodal data from the equipment. The multimodal data includes equipment operation video data, equipment infrared temperature data, environmental acoustic signals, and multi-point vibration signals. During the collection process, perform underwater acoustic interference decomposition on the multi-point vibration signals.

[0011] S4. Preprocess, extract features and fuse the collected data to identify anomalies based on the real-time operating information of the hydropower plant;

[0012] S5. Record routine inspection reports, emergency inspection reports and abnormality reports, and send them to the terminal device.

[0013] In S1, an inspection robot is used to conduct inspections according to the initial inspection route information in the inspection route resource library to collect the location data of the hydropower plant equipment.

[0014] In S2, the regular inspection route information includes the equipment inspection sequence and the inspection start time; the emergency inspection route information includes the key equipment selection and the equipment inspection sequence.

[0015] The routine inspection route is planned as follows:

[0016] By analyzing the historical data of each device in the time window corresponding to the most recent path update cycle, a device data generation frequency characteristic model is established. :

[0017] ;

[0018] in: Representation device The fluctuation range of basic data, represents the time variable, and Respectively represent the peak time point and discrete degree of data change, 、 and Represent the amplitude, phase and frequency of periodic data changes respectively, Indicates the baseline value of device data;

[0019] Calculate the optimal inspection time for each device based on the frequency characteristic model generated by the data and establish the objective function and , To minimize the total path length, It is used to minimize the deviation between the actual inspection time and the optimal inspection time. The objective function formula is:

[0020] ;

[0021] ;

[0022] in: Indicates the number of devices, Representation device To device distance, Is a decision variable, indicating whether to Go directly to the device ; Indicates the device The actual inspection time is calculated by the inspection start time, the time required to pass the path and the time required for data collection. Represents a device that generates frequency model calculations based on data The best inspection time;

[0023] Set constraints: Constraints include each device being inspected at most once, flow conservation constraints, and sub-loop elimination constraints;

[0024] Each device can be inspected at most once: ;

[0025] Flow conservation constraints: ;

[0026] Sub-loop elimination constraints: ;in and Respectively represent devices and equipment Sequence number in the equipment inspection sequence;

[0027] For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,conventional inspection route information.

[0028] The emergency inspection route is planned as follows:

[0029] Collect the working condition information of the hydropower plant in real time, define the working condition change index, and when the working condition change index exceeds the preset threshold When the emergency inspection is triggered;

[0030] Working condition information includes Working condition parameters; working condition change indicators The calculation formula is:

[0031] ;

[0032] in, Indicates time No. Working condition parameters, and Respectively represent The maximum and minimum values ​​of each working condition parameter;

[0033] Based on the real-time collected hydropower plant operating information, a dynamic equipment importance assessment model is established:

[0034] ;

[0035] in, Indicates time Working conditions Download equipment The importance of Represents a collection of attributes for a device. Indicates the An evaluation function, Indicates the The weight of the evaluation function, Represents the total number of evaluation functions; the evaluation functions include the correlation function between equipment and safe operation, the equipment failure probability function, and the equipment current health status function;

[0036] Establish objective function based on dynamic evaluation model of equipment importance , which is used to minimize the negative value of the sum of the importance of inspection equipment. The formula is:

[0037] ;

[0038] in, is a decision variable, representing the device whether it is selected;

[0039] For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,emergency inspection route information.

[0040] Particle encoding is performed using binary encoding, where each particle is represented as a dimensional vector, each vector component takes the value 0 or 1, representing whether the corresponding device is selected, and the constraint condition also includes exceeding the importance threshold The device must be selected.

[0041] The adaptive particle swarm genetic hybrid algorithm solution process is:

[0042] Use real number encoding to encode particles, and each particle is represented by a dimensional vector ,in, Indicates the inspection start time. Representation device Sequence number in the equipment inspection sequence;

[0043] Constructing a fitness function ;in, and Represent the weights of the two objectives, , and Represent the maximum value of the two targets respectively;

[0044] According to the fitness function, the speed and position of each particle are updated through the standard update rule of the particle swarm algorithm. The genetic algorithm is used to perform selection, crossover and mutation operations in each generation of particles and generate new solutions.

[0045] When the fitness change of the particle reaches the preset stop condition, the optimization process stops, the optimal equipment inspection sequence and the optimal inspection start time are output, and the regular inspection route information is obtained;

[0046] In the optimization process of the adaptive particle swarm genetic hybrid algorithm, the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm are adaptively adjusted:

[0047] Inertia Weight Adaptive adjustment formula:

[0048] ;

[0049] Mutation probability Adaptive adjustment formula:

[0050] ;

[0051] in, and Represent the maximum and minimum values ​​of the inertia weight, Indicates the current iteration number, Indicates the maximum number of iterations; and represent the maximum and minimum values ​​of the mutation probability, respectively. Represents an individual The fitness of and They represent the maximum fitness and minimum fitness of the current population respectively.

[0052] In S3, the environmental sound interference is decomposed for the multi-point vibration signal as follows:

[0053] Building an overcomplete dictionary of equipment vibration signals and environmental acoustic signals ,in and represent the vibration feature dictionary and the environmental acoustic feature dictionary respectively;

[0054] Building a sparse representation model based on an overcomplete dictionary ;

[0055] in, Indicates the vibration signal under the interference of environmental sound, and represents the sparse coefficient, represents additive noise;

[0056] Solution , iteratively solve the sparse coefficients and ;in, , and Represent the weight matrices of the sparse coefficients of vibration and ambient acoustic signals, and represents the regularization parameter;

[0057] Automatically adjust the elements of the weight matrix according to the time-frequency characteristics of the signal and :

[0058] ;

[0059] ;

[0060] in, Represents a small positive number to prevent the denominator from being zero. Indicates that the vibration signal part is in The coefficients on the frequency components, Indicates that the ambient acoustic signal is in the The coefficients on the frequency components, and represent the variance of vibration signal and ambient acoustic signal respectively;

[0061] The reconstructed vibration signal is obtained according to the solution results .

[0062] In S4, the steps of preprocessing, feature extraction and fusion of collected data are as follows:

[0063] Preprocess the collected data, including denoising, sharpening, contrast enhancement, correction, filtering and time alignment;

[0064] For the preprocessed data, the convolutional neural network is used to analyze the video frames and extract the video data features. ;

[0065] Extract infrared temperature data features by statistically analyzing temperature mean, standard deviation and change rate ;

[0066] Extract vibration signal features through time domain and frequency domain analysis ;

[0067] Extract acoustic signal features through Mel-frequency cepstral coefficients, loudness and frequency ;

[0068] The weighted fusion of each modal feature is used to obtain the fusion feature , the formula is as follows:

[0069] ;

[0070] in, 、 、 and Represents the weight of each modal data feature;

[0071] Based on fusion features , combined with the hydropower plant operating condition information at the corresponding time, identify equipment abnormalities. The steps are as follows:

[0072] Obtain operating condition information during inspection data collection, including equipment load, water level, and ambient temperature;

[0073] Based on historical data, obtain the normal characteristics of the corresponding working conditions at the time of inspection data collection ;

[0074] Calculate fusion features With normal characteristics When the cosine similarity is less than the threshold , then the data is judged to be abnormal; or, according to the fusion feature With normal characteristics Calculate the anomaly score. When the anomaly score is greater than or equal to the adaptive threshold , then the data is judged to be abnormal.

[0075] In S5, the regular inspection report includes inspection route information, equipment data characteristics, the deviation between the optimal inspection time and the actual inspection time of each device, and resource consumption statistics;

[0076] Emergency inspection reports include operating condition change index values, specific operating condition parameters exceeding thresholds, a list of key equipment, and details of the emergency inspection route;

[0077] Abnormal reports include abnormal equipment identification, abnormal data characteristics, abnormality determination information, and related working condition information;

[0078] Reports are pushed based on differentiated terminal role permissions. Operation and maintenance personnel receive real-time abnormality alerts on their mobile terminals, management terminals obtain summary reports, and the monitoring screen displays inspection route maps and equipment abnormalities in real time.

[0079] The present invention has the following beneficial effects:

[0080] 1. The present invention proposes a dynamic inspection path planning mechanism, which combines the characteristics of equipment data with real-time operating condition information to achieve dual optimization of routine inspections and emergency inspections. In routine inspections, the equipment data frequency model is established by analyzing the historical data of the equipment, and the multi-objective optimization problem is solved based on the adaptive particle swarm genetic hybrid algorithm. The algorithm takes minimizing the total path length and the deviation between the actual inspection time and the optimal time as the core goal, and combines constraints such as flow conservation and sub-loop elimination to generate a global optimal path. In emergency inspections, the operating parameters of the hydropower plant are monitored in real time, the importance of the equipment is dynamically evaluated, and emergency tasks are triggered when the operating condition changes exceed the threshold. The same algorithm is used to quickly generate inspection paths covering key equipment to ensure that high-risk equipment is not missed. This mechanism solves the problem that the traditional fixed inspection mode cannot adapt to the dynamic changes of equipment data and sudden operating conditions, improves inspection efficiency and emergency task response speed, reduces the missed detection rate, and can significantly improve the flexibility and safety of hydropower plant inspections.

[0081] 2. To address the problem of interference from the complex acoustic environment of hydropower plants on equipment vibration signals, this paper proposes a signal separation technology based on an overcomplete dictionary and adaptive sparse representation. By constructing an overcomplete dictionary that includes equipment vibration characteristics and environmental acoustic characteristics, a sparse representation model is established to decompose the mixed signal into a target vibration component and an environmental noise component. The ADMM algorithm is used to iteratively solve the sparse coefficients, and the weight parameters are dynamically adjusted according to the signal's time-frequency characteristics to achieve high-precision separation of signal and noise. This improves the accuracy of vibration signal anomaly detection and provides reliable input for subsequent multimodal data fusion and anomaly identification.

[0082] 3. The present invention proposes a precise anomaly identification method that combines multimodal data fusion features with a feature reference library under dynamic working conditions. By integrating video, infrared temperature, vibration, and acoustic multi-source data, corresponding methods are used for processing and feature extraction, and weighted fusion is used to generate a comprehensive feature vector. At the same time, a feature reference library under dynamic working conditions is established by combining real-time working condition parameters with historical data, and anomalies are identified by calculating the similarity between real-time features and reference features. Through multimodal complementarity and working condition adaptive analysis, the anomaly detection coverage rate can be improved, the false alarm rate can be reduced, the robustness of the system in complex operating environments can be significantly improved, the need for manual review can be reduced, and the inspection efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0084] Figure 1 Flowchart of the method of the present invention.

[0085] Figure 2 This is a flowchart of conventional inspection route planning in the present invention.

[0086] Figure 3 This is a flow chart of emergency inspection route planning in the present invention. DETAILED DESCRIPTION

[0087] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0088] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0089] See also Figure 1 This embodiment provides a multimodal data fusion intelligent inspection method for a hydropower plant, comprising the following steps:

[0090] S1. Determine the equipment inspection sequence with the shortest path based on the location data of the internal equipment of the hydropower plant to obtain the initial inspection route.

[0091] Specifically, inspection robots were used to conduct inspections based on the initial inspection route information stored in the inspection route resource library and collect data from hydropower plant equipment. The initial inspection route information was obtained by determining the shortest equipment inspection sequence based on the location of the hydropower plant equipment. A topological network was constructed based on the spatial distribution of the hydropower plant equipment. Each device node was mapped as a weighted graph vertex, with the inter-device travel distance as the edge weight. A dynamic programming algorithm was used to generate the shortest device visit sequence without duplicate coverage, thus obtaining the initial inspection route.

[0092] S2. When the number of inspections reaches the path update cycle, the inspection path is planned based on the frequency of key data characteristics of the internal equipment of the hydropower plant and the specific location between the equipment, and the inspection route information is updated; among them, the inspection path planning includes regular inspection path planning and emergency inspection path planning, and regular inspection route information and emergency inspection route information are obtained respectively.

[0093] Specifically, when the number of inspections reaches the path update cycle, the inspection robot's inspection path is rationally planned and updated based on the frequency of key data characteristics generated by the hydropower plant's internal equipment and the specific locations between the equipment. The path update cycle can be set to a preset number of inspections, such as performing routine inspection path planning every 10 routine inspections. Emergency inspection path planning is triggered by real-time operating condition information.

[0094] Inspection route planning includes regular inspection route planning and emergency inspection route planning, which respectively obtain regular inspection route information and emergency inspection route information. Regular inspection route information includes equipment inspection sequence and inspection start time; emergency inspection route information includes key equipment selection and equipment inspection sequence.

[0095] See also Figure 2 , the conventional inspection route planning is as follows:

[0096] (1) Establish a device data generation frequency characteristic model by analyzing the historical data of each device in the time window corresponding to the most recent path update cycle :

[0097] ;

[0098] in, Representation device The fluctuation range of basic data, represents the time variable, and Respectively represent the peak time point and discrete degree of data change, 、 and Represent the amplitude, phase and frequency of periodic data changes respectively, Indicates the baseline value of device data.

[0099] The fluctuation range of basic data is obtained by calculating the standard deviation of the normalized difference value of historical data;

[0100] By marking all the fluctuation peak points exceeding the preset threshold in the historical data, clustering analysis is performed on the timestamps of the peak points to identify the time windows where the main fluctuations are concentrated. The weighted average and variance of the offset of the time points in each cluster are calculated to obtain the peak time point and the degree of dispersion, with the weight being the amplitude value of the corresponding peak.

[0101] Apply fast Fourier transform to detect the main frequency component of historical data and determine the periodic frequency of the equipment;

[0102] Use a time series decomposition algorithm to separate the periodic components of the data and calculate their amplitude and phase.

[0103] Apply sliding average to historical data to filter out high-frequency fluctuations, and take the median of the smoothed curve as the baseline value.

[0104] (2) Calculate the optimal inspection time for each device based on the frequency characteristic model generated by the data and establish the objective function and , To minimize the total path length, It is used to minimize the deviation between the actual inspection time and the optimal inspection time. The objective function formula is:

[0105] ;

[0106] ;

[0107] in, Indicates the number of devices, Representation device To device distance, Is a decision variable, indicating whether to Go directly to the device ; Indicates the device The actual inspection time is calculated by the inspection start time, the time required to pass the path and the time required for data collection. Represents a device that generates frequency model calculations based on data The best inspection time.

[0108] The optimal inspection time for each device is calculated based on the data generation frequency characteristic model: the first-order derivative of the data generation frequency characteristic model is taken and its zero point is solved. The extreme point is numerically approximated using the Newton iteration method. The global maximum point is ensured by verifying that the second-order derivative is negative, thereby obtaining the optimal inspection time.

[0109] (3) Setting constraints: Constraints include each device being inspected at most once, flow conservation constraints, and sub-loop elimination constraints;

[0110] Each device can be inspected at most once: ;

[0111] Flow conservation constraints: ;

[0112] Sub-loop elimination constraints: ;in and Respectively represent devices and equipment The sequence number in the device inspection sequence.

[0113] (4) For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,conventional inspection route information.

[0114] The adaptive particle swarm genetic hybrid algorithm solution process is:

[0115] Use real number encoding to encode particles, and each particle is represented by a dimensional vector ,in, Indicates the inspection start time. Representation device The sequence number in the device inspection sequence.

[0116] Constructing a fitness function ;in, and Represent the weights of the two objectives, , and Represent the maximum value of the two targets respectively.

[0117] According to the fitness function, the speed and position of each particle are updated through the standard update rule of the particle swarm algorithm. The genetic algorithm is used to perform selection, crossover and mutation operations in each generation of particles to explore new solution spaces and generate new solutions.

[0118] When the fitness change of the particle reaches the preset stop condition, the optimization process stops, the optimal equipment inspection sequence and the optimal inspection start time are output, and the regular inspection route information is obtained.

[0119] In the optimization process of the adaptive particle swarm genetic hybrid algorithm, the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm are adaptively adjusted:

[0120] Inertia Weight Adaptive adjustment formula:

[0121] ;

[0122] Mutation probability Adaptive adjustment formula:

[0123] ;

[0124] in, and Represent the maximum and minimum values ​​of the inertia weight, Indicates the current iteration number, Indicates the maximum number of iterations; and represent the maximum and minimum values ​​of the mutation probability, respectively. Represents an individual The fitness of and They represent the maximum fitness and minimum fitness of the current population respectively.

[0125] See also Figure 3 , the emergency inspection route planning is as follows:

[0126] (1) Collect working condition information in real time and define working condition change indicators. When the working condition change indicators exceed the preset threshold When the emergency inspection is triggered.

[0127] If the regular inspection is in progress when the emergency inspection is triggered, the inspection plan will not be changed; if the regular inspection has not started when the emergency inspection is triggered, and the time from the start of the regular inspection exceeds the threshold , an emergency inspection is performed; if the regular inspection has not started when the emergency inspection is triggered, and the time from the start of the regular inspection does not exceed the threshold , the inspection plan will not be changed.

[0128] Working condition information includes Working condition parameters; working condition change indicators The calculation formula is:

[0129] ;

[0130] in, Indicates time No. Working condition parameters, and Respectively represent The maximum and minimum values ​​of the parameters of each working condition.

[0131] (2) Based on the real-time collected hydropower plant operating information, a dynamic equipment importance assessment model is established:

[0132] ;

[0133] in, Indicates time Working conditions Download equipment The importance of Represents a collection of attributes for a device. Indicates the An evaluation function, Indicates the The weight of the evaluation function, Represents the total number of evaluation functions; the evaluation functions include the function of the relevance of equipment to safe operation, the function of the probability of equipment failure, and the function of the current health status of the equipment.

[0134] The evaluation functions in the dynamic evaluation model of equipment importance include:

[0135] Function of the correlation between equipment and safe operation: ;

[0136] Equipment failure probability function: ;

[0137] Device current health status function: ;

[0138] in, Representation device Safety Importance Index, Indicates the current working conditions The load factor under Indicates the maximum value of the safety importance index of all equipment; Representation device The number of failures under similar historical conditions, Representation device Current health index; Representation device health index, and They represent the minimum and maximum values ​​of the health index respectively; based on the data collected in the observation window, the health of the equipment is evaluated using machine learning, statistical analysis or signal processing methods to obtain the equipment health index.

[0139] (3) Establishing the objective function based on the dynamic evaluation model of equipment importance , which is used to minimize the negative value of the sum of the importance of inspection equipment. The formula is:

[0140] ;

[0141] in, is a decision variable, representing the device Whether it is selected.

[0142] (4) For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,emergency inspection route information.

[0143] Among them, binary encoding is used for particle encoding, and each particle is represented as a dimensional vector, each vector component takes the value 0 or 1, representing whether the corresponding device is selected, and the constraint condition also includes exceeding the importance threshold The device must be selected.

[0144] S3. During the inspection process, the inspection point information on the inspection route is obtained, a list of inspection points is established, and multimodal data of the equipment is collected. The multimodal data includes equipment operation video data, equipment infrared temperature data, environmental acoustic signals, and multi-point vibration signals. During the collection process, the multi-point vibration signals are subjected to underwater acoustic interference decomposition.

[0145] The decomposition of environmental sound interference for multi-point vibration signals is as follows:

[0146] (1) Establish an overcomplete dictionary of equipment vibration signals and environmental acoustic signals ,in and Represent the vibration feature dictionary and environmental acoustic feature dictionary respectively.

[0147] (2) Establish a sparse representation model based on an overcomplete dictionary ;

[0148] in, Indicates the vibration signal under the interference of environmental sound, and represents the sparse coefficient, represents additive noise;

[0149] (3) Solve using ADMM algorithm , iteratively solve the sparse coefficients and ;

[0150] in, , and Represent the weight matrices of the sparse coefficients of vibration and ambient acoustic signals, and represents the regularization parameter.

[0151] Automatically adjust the elements of the weight matrix according to the time-frequency characteristics of the signal and :

[0152] ;

[0153] ;

[0154] in, Represents a small positive number to prevent the denominator from being zero. Indicates that the vibration signal part is in The coefficients on the frequency components, Indicates that the ambient acoustic signal is in the The coefficients on the frequency components, and represent the variance of vibration signal and ambient acoustic signal respectively;

[0155] The reconstructed vibration signal is obtained according to the solution results .

[0156] By building a feature library of hydropower plant environmental noise, which includes typical noises under different water flow rates and unit loads, we can quickly identify interference components through transfer learning to achieve noise decomposition, and train the model to build basic feature extraction capabilities on public acoustic datasets.

[0157] S4. Preprocess, extract features and fuse the collected data, and identify anomalies based on the real-time operating information of the hydropower plant.

[0158] Specifically, in S4, the steps of preprocessing, feature extraction and fusion of the collected data are as follows:

[0159] Preprocess the collected data, including denoising, sharpening, contrast enhancement, correction, filtering and time alignment;

[0160] For the preprocessed data, the convolutional neural network is used to analyze the video frames and extract the video data features. ;

[0161] Extract infrared temperature data features by statistically analyzing temperature mean, standard deviation and change rate ;

[0162] Extract vibration signal features through time domain and frequency domain analysis ;

[0163] Extract acoustic signal features through Mel-frequency cepstral coefficients, loudness and frequency ;

[0164] The weighted fusion of each modal feature is used to obtain the fusion feature , the formula is as follows:

[0165] ;

[0166] in, 、 、 and The weights representing the features of each modal data can be set through training or based on data quality.

[0167] Based on fusion features , combined with the hydropower plant operating condition information at the corresponding time, identify equipment abnormalities. The steps are as follows:

[0168] Obtain working condition information during inspection data collection, including equipment load, water level, ambient temperature, etc.;

[0169] Based on historical data, obtain the normal characteristics of the corresponding working conditions at the time of inspection data collection ;

[0170] Calculate fusion features With normal characteristics When the cosine similarity is less than the threshold , then the data is judged to be abnormal.

[0171] In another embodiment, based on the fusion features With normal characteristics Calculate the anomaly score. When the anomaly score is greater than or equal to the adaptive threshold , then the data is judged to be abnormal.

[0172] Specifically, the anomaly score The calculation formula is:

[0173] ;

[0174] ;

[0175] in, Represents the absolute deviation between the fusion feature and the normal feature, The wavelet packet coherence coefficient representing the fusion feature and the normal feature; and Represent the wavelet packet coefficients of fusion features and normal features respectively, s Indicates scale, j Indicates location;

[0176] Calculating adaptive thresholds : Get the abnormal score sequence in the most recent observation period, calculate the first quartile and the third quartile, and get the interquartile range ; Calculate the threshold based on the interquartile range :

[0177] ;

[0178] represents the 95% quantile in the anomaly score sequence, Represents the risk factor.

[0179] S5. Record routine inspection reports, emergency inspection reports and abnormality reports, and send them to the terminal device.

[0180] Specifically, in S5, the regular inspection report includes inspection route information, equipment data characteristics, the deviation between the optimal inspection time and the actual inspection time of each device, and resource consumption statistics;

[0181] Emergency inspection reports include operating condition change index values, specific operating condition parameters exceeding thresholds, a list of key equipment, and details of the emergency inspection route;

[0182] Abnormal reports include abnormal equipment identification, abnormal data characteristics, abnormality determination information, and related working condition information;

[0183] Reports are pushed based on differentiated terminal role permissions. Operation and maintenance personnel receive real-time abnormality alerts on their mobile terminals, management terminals obtain summary reports, and the monitoring screen displays inspection route maps and equipment abnormalities in real time.

[0184] This method analyzes historical data characteristics of hydropower plant equipment to establish a frequency model for device data generation. Incorporating device location information, it uses an adaptive particle swarm genetic hybrid algorithm to dynamically plan inspection routes. With the dual goals of minimizing total path length and the deviation between actual inspection time and optimal time, a globally optimal path is generated using constraints. By encoding inspection sequences and times and iteratively optimizing the path, this method overcomes the limitations of traditional fixed inspection models, which are unable to adapt to dynamic changes in device data. This improves inspection efficiency, reduces missed inspections due to time deviations, and reduces inspection costs.

[0185] Based on real-time monitoring of hydropower plant operating parameters, this method quantifies equipment importance through a dynamic assessment model. When operating conditions suddenly change beyond a threshold, it prioritizes the generation of emergency inspection routes covering critical equipment. A hybrid algorithm is used to rapidly solve the objective function, ensuring that high-risk equipment is prioritized under extreme conditions such as high head and high flow. This mechanism improves emergency response speed, reduces the risk of missed inspections, effectively avoids missed inspections of equipment anomalies due to unexpected operating conditions, and significantly enhances hydropower plant operational safety.

[0186] This invention designs multi-objective optimization functions to address the different needs of routine and emergency inspections. For routine inspections, it achieves optimal matching of resource utilization and timeliness by balancing the weight coefficients of path length and time deviation. For emergency inspections, it uses a hybrid algorithm to rapidly generate paths, focusing on maximizing coverage of key equipment. This avoids the one-sidedness of single-objective optimization while simultaneously improving resource utilization.

[0187] This invention utilizes a dynamic parameter optimization mechanism within a particle swarm genetic hybrid algorithm. By providing real-time feedback on iteration status and individual fitness, it adaptively adjusts inertia weight and mutation probability, improving path planning efficiency and global optimality. The inertia weight dynamically decays based on iteration progress, while the mutation probability is dynamically adjusted based on individual fitness differences. This increases population diversity, preserves high-quality genes, and avoids falling into local optimal solutions.

[0188] This method addresses the interference of high-noise environments in hydropower plants on vibration signals by constructing an overcomplete dictionary encompassing both vibration and ambient acoustic features. This method then uses a sparse representation model to separate target signals from background noise. The ADMM algorithm dynamically adjusts weight parameters and iteratively solves based on the signal's time-frequency characteristics, ultimately reconstructing a high-precision vibration signal. This improves the signal-to-noise ratio (SNR) and the accuracy of anomaly detection. This method is particularly suitable for scenarios with turbulent water flows or high-speed equipment operation, resolving the problem of misjudgment caused by noise interference in traditional methods.

[0189] This method integrates multimodal data from video, infrared temperature, vibration, and acoustics. It uses convolutional neural networks to extract video features, statistical methods to analyze temperature changes, time-frequency analysis to capture vibration characteristics, and Mel-frequency cepstral coefficients to analyze acoustic features. This method then fuses these data using a dynamic weighting strategy to generate a comprehensive feature vector. By leveraging the complementarity of multi-source data, it improves anomaly detection coverage and reduces false alarm rates, overcoming the limitations of a single data source. Furthermore, it assigns weights based on data quality, strengthens the contribution of high-reliability modalities, and enhances overall detection robustness.

[0190] This method dynamically establishes a normal feature baseline library by combining real-time operating parameters with historical data. It identifies anomalies by calculating the cosine similarity between the real-time and baseline data features, and sets adaptive thresholds to distinguish between operating fluctuations and true faults. For example, normal data changes under high load during the flood season will not be misidentified as anomalies, while feature deviations caused by equipment failures can be accurately captured, reducing false alarm rates. The system also supports long-term adaptive updating of the baseline library to adapt to evolving hydropower plant operating conditions, ensuring accurate anomaly identification and system sustainability.

[0191] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Any modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal data fusion intelligent inspection method for hydropower plants, characterized in that: The following steps are involved: S1. Determine the equipment inspection sequence with the shortest path based on the location data of the internal equipment of the hydropower plant to obtain the initial inspection route; S2. When the number of inspections reaches the path update period, the inspection path is planned based on the frequency of key data characteristics of the hydropower plant's internal equipment and the specific locations between the equipment, and the inspection route information is updated. The inspection path planning includes regular inspection path planning and emergency inspection path planning, and regular inspection route information and emergency inspection route information are obtained respectively. S3. During the inspection process, obtain information about inspection points along the inspection route, establish a list of inspection points, and collect multimodal data from the equipment. The multimodal data includes equipment operation video data, equipment infrared temperature data, environmental acoustic signals, and multi-point vibration signals. During the collection process, perform underwater acoustic interference decomposition on the multi-point vibration signals. S4. Preprocess, extract features and fuse the collected data to identify anomalies based on the real-time operating information of the hydropower plant; S5. Record routine inspection reports, emergency inspection reports, and abnormality reports, and send them to the terminal device; Among them, the routine inspection route planning is as follows: 1) Establish a device data generation frequency characteristic model by analyzing the historical data of each device within the time window corresponding to the most recent path update cycle : ； in: Representation device The fluctuation range of basic data, represents the time variable, and Respectively represent the peak time point and discrete degree of data change, 、 and Represent the amplitude, phase and frequency of periodic data changes respectively, Indicates the baseline value of device data; 2) Calculate the optimal inspection time for each device based on the frequency characteristic model generated by the data and establish the objective function and , To minimize the total path length, It is used to minimize the deviation between the actual inspection time and the optimal inspection time. The objective function formula is: ； ； in: Indicates the number of devices, Representation device To device distance, Is a decision variable, indicating whether to Go directly to the device ; Indicates the device The actual inspection time is calculated by the inspection start time, the time required to pass the path and the time required for data collection. Represents a device that generates frequency model calculations based on data The best inspection time; 3) Set constraints: Constraints include a maximum of one inspection per device, flow conservation constraints, and sub-loop elimination constraints; Each device can be inspected at most once: ; Flow conservation constraints: ; Sub-loop elimination constraints: ;in and Respectively represent devices and equipment Sequence number in the equipment inspection sequence; 4) For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,conventional inspection route information; The emergency inspection route is planned as follows: 1) Real-time collection of hydropower plant operating information, definition of operating condition change indicators, when the operating condition change indicators exceed the preset threshold When the emergency inspection is triggered; Working condition information includes Working condition parameters; working condition change indicators The calculation formula is: ； in, Indicates time No. Working condition parameters, and Respectively represent The maximum and minimum values ​​of each working condition parameter; 2) Based on the real-time collected hydropower plant operating information, a dynamic equipment importance assessment model is established: ； in, Indicates time Working conditions Download equipment The importance of Represents a collection of attributes for a device. Indicates the An evaluation function, Indicates the The weight of the evaluation function, Represents the total number of evaluation functions; the evaluation functions include the correlation function between equipment and safe operation, the equipment failure probability function, and the equipment current health status function; 3) Establish objective function based on dynamic evaluation model of equipment importance , which is used to minimize the negative value of the sum of the importance of inspection equipment. The formula is: ； in, is a decision variable, representing the device whether it is selected; 4) For the objective function and ,Under the constraints, the adaptive particle swarm genetic hybrid algorithm is used to solve the ,emergency inspection route information.

2. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1 is characterized in that: In S1, an inspection robot is used to conduct inspections according to the initial inspection route information in the inspection route resource library to collect the location data of the hydropower plant equipment.

3. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1 is characterized in that: In S2, the regular inspection route information includes the equipment inspection sequence and the inspection start time; the emergency inspection route information includes the key equipment selection and the equipment inspection sequence.

4. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1 is characterized in that: Particle encoding is performed using binary encoding, where each particle is represented as a dimensional vector, each vector component takes the value 0 or 1, representing whether the corresponding device is selected, and the constraint condition also includes exceeding the importance threshold The device must be selected.

5. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1 is characterized in that: The adaptive particle swarm genetic hybrid algorithm solution process is: Use real number encoding to encode particles, and each particle is represented by a dimensional vector ,in, Indicates the inspection start time. Representation device Sequence number in the equipment inspection sequence; Constructing a fitness function ;in, and Represent the weights of the two objectives, , and Represent the maximum value of the two targets respectively; According to the fitness function, the speed and position of each particle are updated through the standard update rule of the particle swarm algorithm. The genetic algorithm is used to perform selection, crossover and mutation operations in each generation of particles and generate new solutions. When the fitness change of the particle reaches the preset stop condition, the optimization process stops, the optimal equipment inspection sequence and the optimal inspection start time are output, and the regular inspection route information is obtained; In the optimization process of the adaptive particle swarm genetic hybrid algorithm, the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm are adaptively adjusted: Inertia Weight Adaptive adjustment formula: ； Mutation probability Adaptive adjustment formula: ； in, and Represent the maximum and minimum values ​​of the inertia weight, Indicates the current iteration number, Indicates the maximum number of iterations; and represent the maximum and minimum values ​​of the mutation probability, respectively. Represents an individual The fitness of and They represent the maximum fitness and minimum fitness of the current population respectively.

6. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1, characterized in that: In S3, the environmental sound interference is decomposed for the multi-point vibration signal as follows: 1) Establish an over-complete dictionary of equipment vibration signals and environmental acoustic signals ,in and represent the vibration feature dictionary and the environmental acoustic feature dictionary respectively; 2) Establish a sparse representation model based on an overcomplete dictionary ; in, Indicates the vibration signal under the interference of environmental sound, and represents the sparse coefficient, represents additive noise; 3) Solution , iteratively solve the sparse coefficients and ;in, , and Represent the weight matrices of the sparse coefficients of vibration and ambient acoustic signals, and represents the regularization parameter; Automatically adjust the elements of the weight matrix according to the time-frequency characteristics of the signal and : ； ； in, Represents a small positive number to prevent the denominator from being zero. Indicates that the vibration signal part is in The coefficients on the frequency components, Indicates that the ambient acoustic signal is in the The coefficients on the frequency components, and represent the variance of vibration signal and ambient acoustic signal respectively; 4) Obtain the reconstructed vibration signal based on the solution results .

7. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1, characterized in that: In S4, the steps of preprocessing, feature extraction and fusion of collected data are as follows: Preprocess the collected data, including denoising, sharpening, contrast enhancement, correction, filtering and time alignment; For the preprocessed data, the convolutional neural network is used to analyze the video frames and extract the video data features. ; Extract infrared temperature data features by statistically analyzing temperature mean, standard deviation and change rate ; Extract vibration signal features through time domain and frequency domain analysis ; Extract acoustic signal features through Mel-frequency cepstral coefficients, loudness and frequency ; The weighted fusion of each modal feature is used to obtain the fusion feature , the formula is as follows: ； in, 、 、 and Represents the weight of each modal data feature; Based on fusion features , combined with the hydropower plant operating condition information at the corresponding time, identify equipment abnormalities. The steps are as follows: Obtain operating condition information during inspection data collection, including equipment load, water level, and ambient temperature; Based on historical data, obtain the normal characteristics of the corresponding working conditions at the time of inspection data collection ; Calculate fusion features With normal characteristics When the cosine similarity is less than the threshold , then the data is judged to be abnormal; or, according to the fusion feature With normal characteristics Calculate the anomaly score. When the anomaly score is greater than or equal to the adaptive threshold , then the data is judged to be abnormal.

8. The multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1, characterized in that: In S5, the regular inspection report includes inspection route information, equipment data characteristics, the deviation between the optimal inspection time and the actual inspection time of each device, and resource consumption statistics; Emergency inspection reports include operating condition change index values, specific operating condition parameters exceeding thresholds, a list of key equipment, and details of the emergency inspection route; Abnormal reports include abnormal equipment identification, abnormal data characteristics, abnormality determination information, and related working condition information; Reports are pushed based on differentiated terminal role permissions. Operation and maintenance personnel receive real-time abnormality alerts on their mobile terminals, management terminals obtain summary reports, and the monitoring screen displays inspection route maps and equipment abnormalities in real time.

Images