Intelligent inspection method for multi-modal data fusion of hydraulic power plant

Abstract

The invention relates to the technical field of electric digital data processing, and discloses an intelligent inspection method for multi-modal data fusion of a hydraulic power plant, which comprises the following steps of: determining an equipment inspection sequence with a shortest path according to position data of internal equipment of the hydraulic power plant to obtain an initial inspection route; aiming at the frequency generated by key data characteristics of internal equipment of the hydraulic power plant and the specific position between the equipment, planning an inspection path, and updating inspection path information; in the inspection process, inspection point information on an inspection route is obtained, an inspection point position list is established, and multi-modal data of equipment is collected; and preprocessing, feature extraction and fusion are carried out on the collected data, and abnormity is identified based on real-time working condition information of the hydraulic power plant. According to the method, the inspection efficiency and the emergency task response speed are improved, the omission ratio is reduced, the flexibility and the safety of hydraulic power plant inspection can be remarkably improved, the anomaly detection coverage rate is improved through multi-mode complementarity and working condition self-adaptive analysis, and the false alarm rate is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of hydropower plant inspection electric digital data processing, and in particular to an intelligent inspection method for multimodal data fusion in hydropower plants. Background Art

[0002] During the operation of equipment in a hydropower plant, inspections need to be carried out to timely detect potential faults and safety hazards of the equipment, and ensure the safe and stable operation of the hydropower plant. At present, the workload of manual inspection is large and the inspection efficiency is not high. The experience of operators varies, and there are differences in inspection results. It is urgent to use robots to replace some of the manual inspection work.

[0003] In the prior art, CN117742334A proposes an intelligent inspection method and system for a hydropower plant based on an inspection robot. The inspection priority is determined through equipment operation information, attributes, and locations to generate an initial inspection route. After the robot performs an equipment inspection once, the inspection data is analyzed. If there is no abnormality, a second equipment inspection is carried out based on the remaining route. If an abnormality is found, the abnormal equipment is marked and data acquisition parameters are obtained, and a route is generated in combination with the parameters for a second equipment inspection. Its disadvantages are that the inspection system uses a fixed inspection time and fails to flexibly adjust according to the differences in the generation frequencies of the key data characteristics of each equipment in the hydropower plant. At the same time, when collecting data, the influence of environmental sound on vibration signals is not considered. In addition, when the working conditions of the hydropower plant fluctuate greatly, there is a lack of emergency inspections of key equipment, which is easy to miss equipment abnormalities, and the distinction between data characteristic changes caused by normal working condition changes and abnormal states is not accurate enough, resulting in false alarms, and still relying on manual recheck or intervention.

[0004] In CN119129953A, a safety inspection method and system for a hydropower plant based on blockchain technology is proposed. The data is processed and protected through blockchain technology, the inspection task allocation and path planning are dynamically adjusted by using the particle swarm optimization algorithm, and the time series inspection data is matched by an improved dynamic time warping algorithm to identify potential equipment abnormalities and problems. Finally, inspection adjustment suggestions are provided to realize the safe management, intelligent scheduling, and real-time warning of hydropower plant inspection data. Its disadvantages are: it fails to make a reasonable inspection plan for the significant differences in the generation frequencies of the key data characteristics of each equipment in the hydropower plant and the working condition changes. At the same time, the problem of vibration signal acquisition under the interference of environmental sound is not considered, especially under the conditions of high water head and large flow. In addition, the data characteristic differences of core equipment under different flows and loads are not analyzed, resulting in insufficient identification of data abnormalities, and data recheck is required to avoid errors and omissions.

[0005] Therefore, although the existing technologies can improve the efficiency of manual inspections and reduce errors and omissions in manually processing inspection data, there are still limitations. The deficiencies are as follows: First, the existing technologies fail to flexibly adjust the inspection time and route according to the differences in the frequencies of the key data features of each device in the hydropower plant; Second, during data collection, environmental sounds can interfere with vibration signals, reducing the accuracy of anomaly recognition; Third, when the operating conditions of the hydropower plant fluctuate greatly, there is a lack of emergency inspections of key equipment, and no distinction is made between data feature changes caused by normal operating condition changes and actual data anomalies, increasing the false alarm rate. Summary of the Invention

[0006] To solve the existing technical problems, the main object of the present invention is to provide an intelligent inspection method for multimodal data fusion in a hydropower plant, which solves the problem that the traditional fixed inspection mode cannot adapt to the dynamic changes of equipment data and sudden operating conditions, improves the inspection efficiency and the response speed to emergency tasks, reduces the missed inspection rate, and can significantly improve the flexibility and safety of hydropower plant inspections. By decomposing the underwater acoustic interference of multi-point vibration signals, the accuracy of vibration signal anomaly detection is improved, providing reliable input for subsequent multimodal data fusion and anomaly recognition. Through multimodal complementarity and operating condition adaptive analysis, the anomaly detection coverage rate is improved, and the false alarm rate is reduced.

[0007] To overcome the problems existing in the prior art, the technical solution adopted by the present invention is: an intelligent inspection method for multimodal data fusion in a hydropower plant, including the following steps: S1. Determine the device inspection sequence with the shortest path according to the position data of the internal devices of the hydropower plant to obtain the initial inspection route; S2. When the number of inspections reaches the path update period, plan the inspection path according to the frequencies of the key data features of the internal devices of the hydropower plant and the specific positions between the devices, and update the inspection route information; among them, the planning of the inspection path includes the planning of the regular inspection path and the emergency inspection path, and the regular inspection route information and the emergency inspection route information are obtained respectively; S3. During the inspection process, obtain the inspection point information on the inspection route, establish an inspection point list, and collect the multimodal data of the devices. The multimodal data includes device operation video data, device infrared temperature data, environmental acoustic signals, and multi-point vibration signals. During the collection process, decompose the underwater acoustic interference of the multi-point vibration signals; S4. Preprocess, extract features, and fuse the collected data, and identify anomalies based on the real-time operating condition information of the hydropower plant; S5. Record the regular inspection report, emergency inspection report, and anomaly report, and send them to the terminal device.

[0008] In S1, an inspection robot is used to perform inspections according to the initial inspection route information in the inspection route resource library, and collect the equipment location data of the hydropower plant.

[0009] In S2, the regular inspection route information includes the equipment inspection sequence and the inspection start time; the emergency inspection route information includes the key equipment selection and the equipment inspection sequence.

[0010] The regular inspection path planning is as follows: By analyzing the historical data of each device within the time window corresponding to the most recent path update cycle, an equipment data generation frequency feature model is established : ; Where: represents the fluctuation range of the basic data of the device , represents the time variable, and represent the peak time point and the dispersion degree of data change respectively, , and represent the amplitude, frequency and phase of periodic data change respectively, represents the baseline value of the device data; Calculate the optimal inspection time of each device according to the data generation frequency feature model, and establish the objective functions and , for minimizing the total path length, for minimizing the deviation between the actual inspection time and the optimal inspection time. The objective function formula is: ; ; Where: represents the number of devices, represents the distance from device to device , is a decision variable, indicating whether to go directly from device to device ; represents the actual inspection time of device . The actual inspection time is calculated through the inspection start time, the time required to pass through the path, and the time required for data collection. represents the optimal inspection time of device calculated according to the data generation frequency model; Set the constraint conditions: The constraint conditions include that each device is inspected at most once, the flow conservation constraint, and the sub-loop elimination constraint; Each device is inspected at most once: ; Flow conservation constraint: ; Sub-loop elimination constraint: ; where and respectively represent the sequence numbers of device and device in the device inspection sequence; For the objective function and , it is solved by the adaptive particle swarm genetic hybrid algorithm under the constraint conditions to obtain the information of the regular inspection route.

[0011] The emergency inspection path planning is as follows: Real-time collect the working condition information of the hydropower plant, define the working condition change index, and trigger an emergency inspection when the working condition change index exceeds the preset threshold ; The working condition information includes working condition parameters; the working condition change index The calculation formula is: ; Among them, represents the th working condition parameter at time and respectively represent the maximum and minimum values of the th working condition parameter; For the real-time collected working condition information of the hydropower plant, establish a dynamic evaluation model of equipment importance: ; Among them, represents the importance of device at time under the working condition , represents the set of device attributes, represents the th evaluation function, represents the weight of the th evaluation function, represents the total number of evaluation functions; the evaluation functions include the correlation function between the device and safe operation, the device failure probability function, and the device current health status function; Establish the objective function according to the dynamic evaluation model of equipment importance, which is used to minimize the negative value of the total importance of the inspected devices. The formula is: ; Among them, is a decision variable indicating whether the device is selected; For the objective function and , under the constraint conditions, it is solved by an adaptive particle swarm genetic hybrid algorithm to obtain the emergency inspection route information.

[0012] Particle coding is carried out using binary coding. Each particle is represented as a -dimensional vector, and each vector component takes a value of 0 or 1, representing whether the corresponding device is selected respectively. And the constraint conditions also include that devices exceeding the importance threshold must be selected.

[0013] The solution process of the adaptive particle swarm genetic hybrid algorithm is as follows: Particle coding is carried out using real number coding. Each particle is represented as a -dimensional vector , among which, represents the inspection start time, represents the device in the sequence number in the device inspection sequence; Construct a fitness function ; among which, and represent the weights of the two objectives respectively, , and represent the maximum values of the two objectives respectively; According to the fitness function, update the speed and position of each particle through the standard update rules of the particle swarm algorithm, perform selection, crossover and mutation operations on the particles in each generation through the genetic algorithm, and generate new solutions; When the fitness change of the particle reaches the preset stop condition, stop the optimization process, output the optimal device inspection sequence and the optimal inspection start time, and obtain the regular inspection route information; During the optimization process of the adaptive particle swarm genetic hybrid algorithm, adaptively adjust the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm: Inertia weight Adaptive adjustment formula: ; Mutation probability Adaptive adjustment formula: ; Among them, and represent the maximum value and the minimum value of the inertia weight respectively, represents the current iteration number, represents the maximum number of iterations; and represent the maximum and minimum values of the mutation probability respectively, represents an individual 's fitness, and represent the maximum fitness and the minimum fitness of the current population respectively.

[0014] In S3, the environmental sound interference decomposition for multi-point vibration signals is as follows: Establish an over-complete dictionary of the device vibration signal and the environmental acoustic signal , where and represent the vibration feature dictionary and the environmental acoustic feature dictionary respectively; Establish a sparse representation model based on the over-complete dictionary ; Among them, represents the vibration signal under environmental sound interference, and represent the sparse coefficients, represents the additive noise; Solve , iteratively solve the sparse coefficients and ; Among them, , and represent the weight matrices of the sparse coefficients of the vibration and environmental acoustic signals respectively, and represent the regularization parameters; Automatically adjust the elements of the weight matrices according to the time-frequency characteristics of the signals and : ; ; Among them, represents a small positive number used to prevent the denominator from being zero, represents the coefficient of the vibration signal part at the th frequency component, represents the coefficient of the environmental acoustic signal part at the th frequency component, and represent the variances of the vibration signal and the environmental acoustic signal respectively; Obtain the reconstructed vibration signal according to the solution result .

[0015] In S4, the steps for preprocessing, feature extraction, and fusion of the collected data are as follows: Preprocess the collected data, including denoising, sharpening, contrast enhancement, calibration, filtering, and time alignment processing; For the preprocessed data, analyze the video frames through a convolutional neural network to extract video data features ; Extract infrared temperature data features by statistically calculating the temperature mean, standard deviation, and change rate ; Extract vibration signal features through time-domain and frequency-domain analysis ; Extract acoustic signal features through Mel-frequency cepstral coefficients, loudness, and frequency ; Perform weighted fusion on the features of each modality to obtain fused features , and the formula is as follows: ; where, , , and represent the weights of the data features of each modality; Based on the fused features , combined with the operating condition information of the hydropower plant at the corresponding time, identify equipment anomalies, and the steps are as follows: Obtain the operating condition information at the time of inspection data collection, including equipment load, water level, and ambient temperature; According to historical data, obtain the normal features under the corresponding operating condition information at the time of inspection data collection ; Calculate the cosine similarity between the fused features and the normal features . When the cosine similarity is less than the threshold , it is determined that the data is abnormal; or, calculate the anomaly score based on the fused features and the normal features . When the anomaly score is greater than or equal to the adaptive threshold , it is determined that the data is abnormal.

[0016] In S5, the regular inspection report includes inspection route information, equipment data features, the deviation value between the optimal inspection time and the actual inspection time of each equipment, and resource consumption statistics; The emergency inspection report includes the values of operating condition change indicators, specific operating condition parameters exceeding the threshold, the list of key equipment, and the details of the emergency inspection path; The anomaly report includes the anomaly equipment identifier, anomaly data features, anomaly determination information, and associated operating condition information; Push reports based on the differentiated permissions of terminal roles. Operation and maintenance personnel receive real-time anomaly alerts on their mobile devices, management terminals obtain summary reports, and monitoring dashboards display the inspection path map and equipment anomaly status in real time.

[0017] The present invention has the following beneficial effects: 1. The present invention proposes a dynamic inspection path planning mechanism, which combines equipment data characteristics and real-time operating conditions information to achieve dual optimization of regular inspections and emergency inspections. In regular inspections, by analyzing historical equipment data, an equipment data frequency model is established, and a multi-objective optimization problem is solved based on an adaptive particle swarm genetic hybrid algorithm. The algorithm takes minimizing the total path length and the deviation between the actual inspection time and the optimal time as the core objectives, and combines constraints such as flow conservation and sub-loop elimination to generate a globally optimal path. In emergency inspections, the operating parameters of the hydropower plant are monitored in real time, and the importance of equipment is dynamically evaluated. When the operating conditions change beyond the threshold, an emergency task is triggered, and the same algorithm is used to quickly generate an inspection path covering key equipment to ensure that high-risk equipment is not missed. This mechanism solves the problem that the traditional fixed inspection mode cannot adapt to the dynamic changes of equipment data and sudden operating conditions, improves the inspection efficiency and the response speed of emergency tasks, reduces the missed inspection rate, and can significantly enhance the flexibility and safety of hydropower plant inspections.

[0018] 2. Aiming at the interference problem of the complex acoustic environment in the hydropower plant on the equipment vibration signal, the present invention proposes a signal separation technology based on an overcomplete dictionary and adaptive sparse representation. By constructing an overcomplete dictionary containing equipment vibration characteristics and environmental acoustic characteristics, a sparse representation model is established, and the mixed signal is decomposed into a target vibration component and an environmental noise component. The ADMM algorithm is used to iteratively solve the sparse coefficients, and the weight parameters are dynamically adjusted according to the time-frequency characteristics of the signal to achieve high-precision separation of the signal and the noise. The accuracy of anomaly detection of vibration signals is improved, providing reliable input for subsequent multi-modal data fusion and anomaly recognition.

[0019] 3. The present invention proposes a precise anomaly recognition method that combines multi-modal data fusion features with a feature reference library under dynamic operating conditions. By integrating multi-source data such as video, infrared temperature, vibration, and acoustics, corresponding methods are used for processing and feature extraction respectively, and weighted fusion is performed to generate a comprehensive feature vector. At the same time, a feature reference library under dynamic operating conditions is established by combining real-time operating parameters and historical data, and anomalies are recognized by calculating the similarity between real-time features and reference features. Through multi-modal complementarity and operating condition adaptive analysis, the coverage rate of anomaly detection can be improved, the false alarm rate can be reduced, the robustness of the system in a complex operating environment can be significantly enhanced, the need for manual review can be reduced, and the inspection efficiency can be improved. Description of the Drawings

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of the method of the present invention.

[0022] Figure 2 It is a flowchart of the regular inspection path planning in the present invention.

[0023] Figure 3 It is a flowchart of the emergency inspection path planning in the present invention. Specific Embodiments

[0024] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present invention with reference to the accompanying drawings of the specification.

[0025] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0026] See Figure 1 , this embodiment provides a multi-modal data fusion intelligent inspection method for a hydropower plant, including the following steps: S1. Determine the equipment inspection sequence with the shortest path according to the position data of the internal equipment of the hydropower plant to obtain the initial inspection route.

[0027] Specifically, use an inspection robot to conduct inspections according to the initial inspection route information in the inspection route resource library and collect hydropower plant equipment data. Determine the equipment inspection sequence with the shortest path according to the positions of the internal equipment of the hydropower plant to obtain the initial inspection route information. Construct a topological network based on the spatial position distribution of the hydropower plant equipment, map each equipment node to the vertex of a weighted graph, use the moving distance between equipment as the edge weight, and solve through the dynamic programming algorithm to generate the shortest equipment access order without duplicate coverage to obtain the initial inspection route.

[0028] S2. When the number of inspections reaches the path update period, plan the inspection path according to the frequency of the key data features generated by the internal equipment of the hydropower plant and the specific positions between the equipment, and update the inspection route information; among them, the planning of the inspection path includes regular inspection path planning and emergency inspection path planning, and the regular inspection route information and emergency inspection route information are obtained respectively.

[0029] Specifically, when the inspection frequency reaches the path update cycle, the inspection path of the inspection robot is reasonably planned according to the frequency of key data features generated by the internal equipment of the hydropower plant and the specific positions between the equipment, and the inspection route information is updated. The path update cycle can be set to a preset inspection frequency, such as performing a regular inspection path planning every 10 regular inspections. The emergency inspection path planning is triggered by real-time working condition information.

[0030] The inspection path planning includes regular inspection path planning and emergency inspection path planning, and the regular inspection route information and emergency inspection route information are obtained respectively. The regular inspection route information includes the equipment inspection sequence and the inspection start time; the emergency inspection route information includes the selection of key equipment and the equipment inspection sequence.

[0031] See Figure 2 , the regular inspection path planning is as follows: (1) By analyzing the historical data of each device within the time window corresponding to the most recent path update cycle, establish a frequency feature model of device data generation : ; Among them, represents the fluctuation amplitude of the basic data of device , represents the time variable, and represent the peak time point and the degree of dispersion of data change respectively, , and represent the amplitude, frequency and phase of periodic data change respectively, represents the baseline value of device data.

[0032] The fluctuation amplitude of the basic data is obtained by calculating the standard deviation of the normalized difference values of the historical data; By marking all the fluctuation peak points exceeding the preset threshold in the historical data, clustering analysis is performed on the timestamps of the peak points, the time window where the main fluctuations are concentrated is identified, and the weighted average value and the variance of the offset are calculated for the time points within each cluster to obtain the peak time point and the degree of dispersion, and the weight is the amplitude value of the corresponding peak; Apply the fast Fourier transform to detect the main frequency component of the historical data and determine the periodic frequency of the device; Use the time series decomposition algorithm to separate the periodic component of the data and calculate its amplitude and phase.

[0033] Apply the moving average to the historical data, filter out the high-frequency fluctuations, and take the median of the smoothed curve as the baseline value.

[0034] (2) Calculate the optimal inspection time for each device according to the data generation frequency feature model, and establish the objective function and , for minimizing the total path length, for minimizing the deviation between the actual inspection time and the optimal inspection time. The objective function formula is: ; ; where, represents the number of devices, represents device to device distance, is a decision variable, indicating whether to go directly from device to device ; represents the actual inspection time of device . The actual inspection time is calculated through the inspection start time, the time required for the passing path, and the time required for data collection. represents the optimal inspection time of device calculated according to the data generation frequency model.

[0035] Calculate the optimal inspection time for each device according to the data generation frequency feature model: Take the first derivative of the data generation frequency feature model and solve its zero point. Combine the Newton iteration method for numerical approximation of the extreme point, and ensure that it is the global maximum point by verifying that the second derivative is negative to obtain the optimal inspection time.

[0036] (3) At the same time, set the constraint conditions: The constraint conditions include that each device is inspected at most once, the flow conservation constraint, and the sub-circuit elimination constraint; Each device is inspected at most once: ; Flow conservation constraint: ; Sub-circuit elimination constraint: ; where and respectively represent the sequence numbers of device and device in the device inspection sequence.

[0037] (4) For the objective function and , solve under the constraint conditions through the adaptive particle swarm genetic hybrid algorithm to obtain the regular inspection route information.

[0038] The solution process of the adaptive particle swarm genetic hybrid algorithm is: Particle coding is performed using real - number coding, and each particle is represented as a dimensional vector , where represents the start time of the inspection tour, represents the equipment sequence number in the equipment inspection sequence.

[0039] Construct the fitness function ; where and represent the weights of the two objectives respectively, , and represent the maximum values of the two objectives respectively.

[0040] According to the fitness function, update the velocity and position of each particle through the standard update rules of the particle swarm algorithm, and perform selection, crossover, and mutation operations on the particles in each generation through the genetic algorithm to explore a new solution space and generate new solutions.

[0041] When the fitness change of the particle reaches the preset stop condition, stop the optimization process, output the optimal equipment inspection sequence and the optimal inspection start time, and obtain the regular inspection route information.

[0042] During the optimization process of the adaptive particle swarm genetic hybrid algorithm, adaptively adjust the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm: Inertia weight Adaptive adjustment formula: ; Mutation probability Adaptive adjustment formula: ; where and represent the maximum and minimum values of the inertia weight respectively, represents the current iteration number, represents the maximum iteration number; and represent the maximum and minimum values of the mutation probability respectively, represents the fitness of the individual , and represent the maximum fitness and minimum fitness of the current population respectively.

[0043] See Figure 3 , the emergency inspection path planning is as follows: (1) Collect the working condition information in real - time, define the working condition change index, and when the working condition change index exceeds the preset threshold When it is triggered, an emergency inspection is initiated.

[0044] If an emergency inspection is triggered while a regular inspection is in progress, the inspection plan remains unchanged. If an emergency inspection is triggered when the regular inspection has not started and the time until the start of the regular inspection exceeds the threshold , then the emergency inspection is executed. If an emergency inspection is triggered when the regular inspection has not started and the time until the start of the regular inspection does not exceed the threshold , then the inspection plan remains unchanged.

[0045] The operating condition information includes operating condition parameters; the operating condition change index The calculation formula is: ; Among them, represents the th operating condition parameter at time and respectively represent the maximum and minimum values of the th operating condition parameter.

[0046] (2) Establish a dynamic evaluation model for equipment importance based on the real-time collected operating condition information of the hydropower plant: ; Among them, represents the importance of equipment under the operating condition at time , represents the set of equipment attributes, represents the th evaluation function, represents the weight of the th evaluation function, represents the total number of evaluation functions; the evaluation functions include the relevance function of the equipment to safe operation, the equipment failure probability function, and the equipment current health status function.

[0047] The evaluation functions in the dynamic evaluation model of equipment importance include: The relevance function of the equipment to safe operation: ; The equipment failure probability function: ; The equipment current health status function: ; Among them, represents the safety importance index of the equipment, represents the load factor under the current operating condition , ​​​​​​Represents the maximum value of the safety importance index of all devices; Represents the device The number of failures under historical similar operating conditions, Represents the device Current health index; Represents the device Health index, And Represent the minimum and maximum values of the health index respectively; According to the data collected within the observation window, the health of the device is evaluated using machine learning, statistical analysis or signal processing methods to obtain the device health index.

[0048] (3) Establish an objective function according to the device importance dynamic evaluation model , which is used to minimize the negative value of the total importance of the inspected devices. The formula is: ; Among them, Is a decision variable, representing whether the device Is selected.

[0049] (4) For the objective function And , solve it through an adaptive particle swarm genetic hybrid algorithm under the constraint conditions to obtain the emergency inspection route information.

[0050] Among them, the binary coding method is used for particle coding. Each particle is represented as a -dimensional vector, and each vector component takes a value of 0 or 1, representing whether the corresponding device is selected respectively. And the constraint conditions also include that the devices exceeding the importance threshold Must be selected.

[0051] S3. During the inspection process, obtain the inspection point information on the inspection route, establish an inspection point list, and collect the multimodal data of the device. The multimodal data includes device operation video data, device infrared temperature data, environmental acoustic signals and multi-point vibration signals. During the collection process, the multi-point vibration signals are decomposed by underwater acoustic interference.

[0052] The environmental sound interference decomposition of the multi-point vibration signals is as follows: (1) Establish an over-complete dictionary of the device vibration signal and the environmental acoustic signal , where And Represent the vibration feature dictionary and the environmental acoustic feature dictionary respectively.

[0053] (2) Establish a sparse representation model according to the over-complete dictionary ; Among them, Represents the vibration signal under environmental sound interference, and represents the sparse coefficient, represents the additive noise; (3) Use the ADMM algorithm to solve , and iteratively solve the sparse coefficient and ; Among them, , and respectively represent the weight matrices of the sparse coefficients of the vibration and environmental acoustic signals, and represent the regularization parameters.

[0054] Automatically adjust the elements of the weight matrix according to the time-frequency characteristics of the signal and : ; ; Among them, represents a small positive number used to prevent the denominator from being zero, represents the coefficient of the vibration signal part at the th frequency component, represents the coefficient of the environmental acoustic signal part at the th frequency component, and respectively represent the variances of the vibration signal and the environmental acoustic signal; Obtain the reconstructed vibration signal .

[0055] By constructing an environmental noise feature library for hydropower plants, including typical noises under different water flow rates and unit loads, noise decomposition is achieved by quickly identifying interference components through transfer learning, and the training model constructs basic feature extraction capabilities on a public acoustic dataset.

[0056] S4. Preprocess, extract features and fuse the collected data, and identify anomalies based on the real-time working condition information of the hydropower plant.

[0057] Specifically, in S4, the steps of preprocessing, feature extraction and fusion of the collected data are as follows: Preprocess the collected data, and the preprocessing includes denoising, sharpening, contrast enhancement, correction, filtering and time alignment processing; For the preprocessed data, analyze the video frames through a convolutional neural network to extract video data features ; Extract infrared temperature data features by statistically calculating the temperature mean, standard deviation and change rate ; Extract the vibration signal features through time-domain and frequency-domain analysis ; Extract the acoustic signal features through Mel-frequency cepstral coefficients, loudness, and frequency ; Perform weighted fusion on each modal feature to obtain the fusion feature , and the formula is as follows: ; Among them, , , and represent the weights of each modal data feature, which can be set through training or based on data quality.

[0058] Based on the fusion feature , combined with the operating condition information of the hydropower plant at the corresponding time, identify equipment abnormalities, and the steps are as follows: Obtain the operating condition information at the time of inspection data collection, including equipment load, water level, ambient temperature, etc.; According to historical data, obtain the normal features under the operating condition information corresponding to the inspection data collection time ; Calculate the cosine similarity between the fusion feature and the normal feature . When the cosine similarity is less than the threshold , it is determined that the data is abnormal.

[0059] In another solution, calculate the anomaly score based on the fusion feature and the normal feature . When the anomaly score is greater than or equal to the adaptive threshold , it is determined that the data is abnormal.

[0060] Specifically, the anomaly score is calculated by the formula: ; ; Among them, represents the absolute deviation between the fusion feature and the normal feature, represents the wavelet packet coherence coefficient between the fusion feature and the normal feature; and respectively represent the wavelet packet coefficients of the fusion feature and the normal feature, s represents the scale, j represents the position; Calculate the adaptive threshold : Obtain the abnormal score sequence within the most recent observation period, calculate the first quartile and the third quartile, and obtain the interquartile range ; Calculate the threshold according to the interquartile range : ; represents the 95th percentile in the abnormal score sequence, represents the risk coefficient.

[0061] S5. Record the regular inspection report, emergency inspection report, and abnormal report, and send them to the terminal device.

[0062] Specifically, in S5, the regular inspection report includes inspection route information, device data characteristics, the deviation value between the optimal inspection time and the actual inspection time of each device, and resource consumption statistics; The emergency inspection report includes the working condition change index value, specific working condition parameters exceeding the threshold, the list of key devices, and the details of the emergency inspection path; The abnormal report includes the abnormal device identifier, abnormal data characteristics, abnormal determination information, and associated working condition information; Push reports based on the differentiation of terminal role permissions. The mobile terminal of the operation and maintenance personnel receives real-time abnormal alarms, the management terminal obtains the summary report, and the monitoring large screen displays the inspection path map and device abnormal conditions in real time.

[0063] The present invention establishes a device data generation frequency model by analyzing the historical data characteristics of the hydropower plant equipment, combines the device location information, and uses an adaptive particle swarm genetic hybrid algorithm to dynamically plan the inspection path. With the dual goals of minimizing the total path length and the deviation between the actual inspection time and the optimal time, a globally optimal path is generated through constraint conditions. By encoding to represent the inspection sequence and time, and iteratively optimizing the path, the defect that the traditional fixed inspection mode cannot adapt to the dynamic changes of device data is solved, the inspection efficiency is improved, the missed inspection problem caused by time deviation is reduced, and the inspection cost is lowered.

[0064] Based on the real-time monitoring of the working condition parameters of the hydropower plant, the present invention quantifies the importance of the device through a dynamic evaluation model, and preferentially generates an emergency inspection path covering key devices when the working condition mutation exceeds the threshold. The hybrid algorithm is used to quickly solve the objective function to ensure that high-risk devices are preferentially covered under extreme conditions such as high head and large flow. This mechanism improves the response speed of emergency tasks, reduces the risk of missed inspections, effectively avoids the missed inspection of device abnormalities caused by sudden working conditions, and significantly enhances the operation safety of the hydropower plant.

[0065] For the different requirements of regular and emergency inspections, the present invention designs multi-objective optimization functions respectively. During regular inspections, by balancing the weight coefficients of path length and time deviation, the optimal matching of resource utilization and timeliness is achieved; during emergency inspections, with the core objective of maximizing the coverage rate of key equipment, a path is quickly generated by combining a hybrid algorithm. This avoids the one-sidedness of single-objective optimization and improves resource utilization at the same time.

[0066] Regarding the dynamic parameter optimization mechanism of the particle swarm genetic hybrid algorithm, the present invention adaptively adjusts the inertia weight and mutation probability by real-time feedback of the iteration state and individual fitness, improving the path planning efficiency and global optimality. The inertia weight dynamically decays according to the iteration progress; the mutation probability is dynamically adjusted based on the individual fitness difference; this can increase the population diversity, retain excellent genes, and avoid falling into local optimal solutions.

[0067] Aiming at the interference of the high-noise environment in hydropower plants on vibration signals, the present invention constructs an over-complete dictionary containing vibration characteristics and environmental acoustic characteristics, and separates the target signal and background noise through a sparse representation model. The ADMM algorithm is used to dynamically adjust the weight parameters, and the solution is iteratively obtained based on the time-frequency characteristics of the signal. Finally, a high-precision vibration signal is reconstructed, improving the signal-to-noise ratio of the vibration signal, increasing the accuracy of anomaly detection, especially applicable to scenarios with rapid water flow or high-speed operation of equipment, and solving the misjudgment problem caused by noise interference in traditional methods.

[0068] The present invention integrates multi-modal data of video, infrared temperature, vibration, and acoustics. Video features are extracted through a convolutional neural network, temperature changes are analyzed by statistical methods, vibration characteristics are captured by time-frequency analysis, and acoustic features are analyzed by Mel frequency cepstral coefficients. A dynamic weighting strategy is used to fuse and generate a comprehensive feature vector. Through the complementarity of multi-source data, the coverage rate of anomaly detection is improved, the false alarm rate is reduced, the limitations of a single data source are overcome, and at the same time, weights are assigned based on data quality to strengthen the contribution of high-reliability modalities and improve the overall detection robustness.

[0069] The present invention dynamically establishes a normal feature reference library by combining real-time operating condition parameters and historical data, identifies anomalies by calculating the cosine similarity between the features of real-time data and reference data, and sets an adaptive threshold to distinguish between operating condition fluctuations and real faults. For example, normal data changes under high load during the flood season will not be misjudged as anomalies, while feature deviations caused by equipment failures can be accurately captured. This can reduce the false alarm rate and support the long-term adaptive update of the reference library to adapt to the evolution of the operating conditions of hydropower plants, ensuring the accuracy of anomaly identification and the sustainability of the system.

[0070] Although the specific embodiments of the present invention have been described above, those skilled in the art of this technology should understand that the specific embodiments we described are illustrative only and not used to limit the scope of the present invention. Effective modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should all be covered by the scope protected by the claims of the present invention.

Claims

1. A multi-modal data fusion intelligent inspection method for hydropower plants, characterized in that, It includes the following steps: S1. Determine the equipment inspection sequence with the shortest path based on the location data of the internal equipment in the hydropower plant to obtain the initial inspection route; S2. When the inspection times reach the path update period, plan the inspection path according to the frequency of the key data characteristics of the internal equipment in the hydropower plant and the specific positions between the equipment, and update the inspection route information; among them, the planning of the inspection path includes the regular inspection path planning and the emergency inspection path planning, and the regular inspection route information and the emergency inspection route information are obtained respectively; S3. Obtain the inspection point information on the inspection route during the inspection process, establish an inspection point list, and collect the multimodal data of the equipment. The multimodal data includes the equipment operation video data, the equipment infrared temperature data, the environmental acoustic signal, and the multi-point vibration signal. During the collection process, decompose the multi-point vibration signal for underwater acoustic interference; S4. Preprocess, extract features and fuse the collected data, and identify anomalies based on the real-time working condition information of the hydropower plant; S5. Record the regular inspection report, the emergency inspection report and the anomaly report, and send them to the terminal device.

2. The intelligent inspection method for multimodal data fusion in a hydropower plant according to claim 1, wherein, In S1, use the inspection robot to perform inspections according to the initial inspection route information in the inspection route resource library, and collect the equipment location data of the hydropower plant.

3. A multimodal data fusion intelligent inspection method for hydropower plants according to claim 1, characterized in that, In S2, the regular inspection route information includes the equipment inspection sequence and the inspection start time; the emergency inspection route information includes the key equipment selection and the equipment inspection sequence.

4. A multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1, characterized in that The regular inspection path planning is as follows: 1) Establish a device data generation frequency feature model by analyzing the historical data of each device within the time window corresponding to the most recent path update cycle : ； Wherein: represents the fluctuation range of the basic data of the device , represents the time variable and respectively represent the peak time point and the degree of dispersion of data change , and respectively represent the amplitude, frequency and phase of the periodic data change represents the baseline value of the device data; 2) Calculate the optimal inspection time for each device according to the data generation frequency feature model, and establish the objective function and , which is used to minimize the total path length, and which is used to minimize the deviation between the actual inspection time and the optimal inspection time. The objective function formula is as follows: ； ； Wherein: represents the number of devices, represents the device to the device distance, is a decision variable indicating whether to go directly from device to device ; represents the actual inspection time of device The actual inspection time is calculated by the inspection start time, the time required for the passing path, and the time required for data collection. represents the optimal inspection time of device calculated according to the data generation frequency model; 3) Set the constraint conditions: The constraint conditions include that each equipment is inspected at most once, the flow conservation constraint, and the sub-loop elimination constraint; Each device is inspected at most once: ; Flow conservation constraint: ; Sub-circuit elimination constraint: ; where and represent the sequence numbers of device and device in the device inspection sequence respectively; 4) For the objective function and , solve under the constraint conditions through an adaptive particle swarm genetic hybrid algorithm to obtain the information of the regular inspection route.

5. A multi-modal data fusion intelligent inspection method for hydropower plants according to claim 4, characterized in that, The emergency inspection path planning is as follows: 1) Real-time collect the working condition information of the hydropower plant, define the working condition change index, and trigger an emergency inspection when the working condition change index exceeds the preset threshold ; The operating condition information includes operating condition parameters; the operating condition change index The calculation formula is as follows: ； Among them, represents the nth operating condition parameter, and respectively represent the maximum and minimum values of the nth operating condition parameter; 2) Establish a dynamic evaluation model for equipment importance according to the real-time collected working condition information of the hydropower plant: ； Among them, represents the operating condition at time of the device importance, represents the set of device attributes, represents the th evaluation function, represents the th weight of the evaluation function, represents the total number of evaluation functions; the evaluation functions include the relevance function of the device and safe operation, the device failure probability function, and the device current health status function; 3) Establish the objective function according to the dynamic evaluation model of equipment importance , which is used to minimize the negative value of the total importance of the inspected equipment. The formula is as follows: ； Among them, is a decision variable representing whether the device is selected; 4) For the objective function and , solve it under the constraint conditions through an adaptive particle swarm genetic hybrid algorithm to obtain the emergency inspection route information.

6. A multimodal data fusion intelligent inspection method for hydropower plants according to claim 5, characterized in that Particle coding is carried out using binary coding, and each particle is represented as a -dimensional vector, where each vector component takes a value of 0 or 1, representing whether the corresponding device is selected or not, and the constraint conditions also include that devices with an importance threshold exceeding must be selected.

7. A multi-modal data fusion intelligent inspection method for hydropower plants according to claim 4 or 5, characterized in that, The solution process of the adaptive particle swarm genetic hybrid algorithm is as follows: Particle coding is carried out using a real number coding method, and each particle is represented as a dimensional vector , where represents the start time of the inspection tour, represents the device sequence number in the device inspection sequence; Construct the fitness function ; among them, and respectively represent the weights of the two objectives, , and respectively represent the maximum values of the two objectives; According to the fitness function, update the velocity and position of each particle through the standard update rule of the particle swarm algorithm, perform selection, crossover and mutation operations on the particles in each generation through the genetic algorithm, and generate new solutions; When the fitness change of the particle reaches the preset stop condition, stop the optimization process, output the optimal equipment inspection sequence and the optimal inspection start time, and obtain the regular inspection route information; During the optimization process of the adaptive particle swarm genetic hybrid algorithm, adaptively adjust the inertia weight of the particle swarm algorithm and the mutation probability of the genetic algorithm; Inertia weight Adaptive adjustment formula: ； Mutation probability Adaptive adjustment formula: ； Among them, and represent the maximum and minimum values of the inertia weight respectively, represents the current iteration number, represents the maximum iteration number; and represent the maximum and minimum values of the mutation probability respectively, represents the fitness of the individual ; and represent the maximum fitness and the minimum fitness of the current population respectively.

8. A multimodal data fusion intelligent inspection method for hydropower plants according to claim 1, characterized in that, In S3, decompose the environmental sound interference for the multi-point vibration signal as follows: 1) Establish an over-complete dictionary of the device vibration signal and the environmental acoustic signal , where and represent the vibration feature dictionary and the environmental acoustic feature dictionary respectively; 2) Establish a sparse representation model based on the over-complete dictionary ; Among them, represents the vibration signal under environmental sound interference, and represents the sparse coefficient, represents the additive noise; 3) Solve , and iteratively solve the sparse coefficients and ; where , and respectively represent the weight matrices of the sparse coefficients of the vibration and ambient acoustic signals, and represent the regularization parameters; Automatically adjust the elements of the weight matrix according to the time-frequency characteristics of the signal and : ； ； wherein, represents a small positive number used to prevent the denominator from being zero, represents the coefficient of the vibration signal part at the -th frequency component, represents the coefficient of the environmental acoustic signal part at the -th frequency component, and respectively represent the variances of the vibration signal and the environmental acoustic signal; 4) Obtain the reconstructed vibration signal based on the solution results .

9. A multimodal data fusion intelligent inspection method for hydropower plants according to claim 1, characterized in that, In S4, the steps of preprocessing, feature extraction and fusion of the collected data are as follows: Preprocess the collected data, and the preprocessing includes denoising, sharpening, contrast enhancement, calibration, filtering and time alignment processing; For the preprocessed data, analyze the video frames through a convolutional neural network to extract video data features ; Extract infrared temperature data features by statistically analyzing the mean temperature, standard deviation, and rate of change ; Extract the vibration signal characteristics through time-domain and frequency-domain analysis ; Extract acoustic signal features through Mel-frequency cepstral coefficients, loudness, and frequency ; The modal features are weighted and fused to obtain the fused features , and the formula is as follows: ； Among them, , , and represent the weights of the feature data of each modality; Based on the fusion features , combined with the operation condition information of the hydropower plant at the corresponding time, identify equipment anomalies, and the steps are as follows: Obtain the working condition information during the inspection data collection, including the equipment load, water level, and environmental temperature; Obtain the normal features under the corresponding working condition information at the inspection data collection moment according to the historical data ; Calculate the fused feature and the normal feature to calculate the cosine similarity. When the cosine similarity is less than the threshold , it is determined that the data is abnormal; or, calculate the anomaly score based on the fused feature and the normal feature . When the anomaly score is greater than or equal to the adaptive threshold , it is determined that the data is abnormal.

10. A multimodal data fusion intelligent inspection method for a hydropower plant according to claim 1, characterized in that, In S5, the regular inspection report includes the inspection route information, the equipment data characteristics, the deviation value between the optimal inspection time and the actual inspection time of each equipment, and the resource consumption statistics; The emergency inspection report includes the working condition change index value, the specific working condition parameters exceeding the threshold, the list of key equipment, and the details of the emergency inspection path; The anomaly report includes the anomaly equipment identification, the anomaly data characteristics, the anomaly determination information, and the associated working condition information; Push reports based on different terminal role permissions. The operation and maintenance personnel receive real-time anomaly alerts on the mobile device, the management terminal obtains the summary report, and the monitoring large screen displays the inspection path map and device anomaly status in real time.

Images