Hydropower station maintenance site safety guarantee system and method based on multi-modal fusion

Abstract

The present application relates to a hydropower station maintenance site safety guarantee system and method based on multi-modal fusion, through high-definition cameras, infrared sensors and pressure sensors to collect on-site images, personnel and object data, after data processing, using image recognition algorithms combined with attention mechanisms, cooperating with multi-modal fusion technology to analyze and judge the on-site state, if there are tool left, personnel not evacuated and other hidden dangers, through the early warning feedback module to issue an alarm in time. Through the cooperative collection of multi-source data by high-definition cameras, infrared sensors and pressure sensors, combined with image recognition and multi-modal fusion technology, the hidden dangers of tool left, personnel not evacuated, sign fence not removed and other hidden dangers are monitored in all directions and accurately, which greatly improves the hidden danger detection accuracy and reduces the subjective omissions and visual blind area problems caused by manual inspection.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring technology for hydropower station equipment maintenance, specifically involving a safety assurance system and method for hydropower station maintenance site based on multimodal fusion. Background Technology

[0002] In the operation and maintenance management of hydropower stations, the cleanup of the site and the monitoring of personnel status after the maintenance of electromechanical equipment are crucial to ensuring the safe conduct of subsequent switching operations. Switching operations involve the state switching of the core electrical equipment of the hydropower station. If there are any problems with the tools left at the maintenance site, they can easily cause electrical faults such as short circuits and discharges during the energization of the equipment, resulting in damage to expensive electromechanical equipment. If warning signs and safety fences are not removed in time, they will interfere with equipment identification and on-site safety protection during the switching operation, significantly increasing the safety risk of misoperation.

[0003] Currently, hydropower station maintenance primarily relies on manual inspections to confirm site clearance and personnel evacuation status. This method has significant technical drawbacks: manual inspections are susceptible to blind spots and worker negligence, making it difficult to avoid missed inspections due to leftover tools and equipment. Furthermore, it cannot accurately and in real-time determine whether all personnel have evacuated, making it unsuitable for the current intelligent and efficient operation and maintenance management needs of hydropower stations. With the rapid development of IoT and AI technologies, there is an urgent need to introduce advanced technologies such as image recognition and multimodal fusion to build a more reliable on-site safety assurance mechanism after hydropower station maintenance. This would fill the gaps in traditional manual management and lay a solid technical foundation for the safe implementation of switching operations. Summary of the Invention

[0004] The technical problem of this invention is that, in the prior art, the manual confirmation of the completion of work and the evacuation of personnel after the maintenance of electromechanical equipment in hydropower stations is prone to oversights, and the traditional management method is difficult to adapt to the needs of intelligent operation and maintenance, resulting in potential hazards such as equipment damage and personal safety. This invention proposes a system and method for ensuring the completion of work and the evacuation of personnel after the maintenance of electromechanical equipment in hydropower stations.

[0005] The purpose of this invention is to solve the above-mentioned problems and propose a multimodal fusion-based on-site safety assurance system for hydropower station maintenance, comprising:

[0006] The data acquisition module is used to collect data on images, personnel presence, and object residue at the maintenance site. The data processing module is used for preprocessing and feature extraction of the collected data; The analysis and judgment module introduces a spatial and channel dual attention mechanism and multimodal fusion to improve the YOLOv8 image recognition algorithm, which is used to analyze and judge feature vectors and generate early warning information. The early warning feedback module is used to receive and transmit early warning information. When the analysis and judgment module performs multimodal fusion, it uses a feature-level fusion formula: F = α·F1 + β·F2 + γ·F3; In the formula, F represents the fused feature vector, F1 represents the image feature vector, F2 represents the infrared sensor feature vector, F3 represents the pressure sensor feature vector, and α, β, and γ represent the weighting coefficients, with α+β+γ=1. The YOLOv8 algorithm is improved by embedding a joint mechanism of spatial attention and channel attention. Anti-metal reflection preprocessing and small target enhancement detection structure for tools are configured for hydropower station maintenance scenarios. This is used to accurately identify areas where tools are easily left behind, such as workbenches, equipment corners, and tool racks.

[0007] Furthermore, the data acquisition module includes high-definition cameras, infrared sensors, and pressure sensors deployed at the maintenance site. The high-definition cameras are used to provide comprehensive coverage of the maintenance work surface, tool area, and entrances and exits without blind spots. The infrared sensors are used to monitor the thermal radiation of personnel entering and exiting, and the pressure sensors are used to monitor the placement status of workbenches and tool racks.

[0008] Furthermore, the analysis and judgment module includes using decision-level fusion to calculate the final decision result by weighting the image recognition decision result, the infrared sensor decision result, and the pressure sensor decision result. The calculation formula is as follows: D = ω1·D1 + ω2·D2 + ω3·D3; In the formula, D represents the final decision result, D1 represents the image recognition decision result, D2 represents the infrared sensor decision result, D3 represents the pressure sensor decision result, and ω1, ω2, and ω3 represent the decision weight coefficients, respectively.

[0009] Preferably, the attention mechanism includes a spatial attention mechanism and a channel attention mechanism; the spatial attention mechanism is used to locate key areas where tools and equipment are easily left behind; the channel attention mechanism is used to enhance the feature weights of tools, signs, fences and other hydropower station-specific detection objects.

[0010] Preferably, the data processing module performs Gaussian filtering, histogram equalization, size normalization and anti-metal reflection suppression preprocessing on the image data, and extracts the shape, outline and position features of tools, signs, fences and people.

[0011] Furthermore, the early warning feedback module adopts a three-level early warning mode, including on-site audible and visual alarms, information push from the operation and maintenance terminal, and linkage early warning with the switching operation control system. If the hidden danger is not dealt with within the preset time limit, the system will automatically send a signal to the switching operation control system to prohibit switching.

[0012] For example, the multimodal fusion-based hydropower station maintenance site safety assurance method includes the following steps: S1: After the maintenance work is completed, collect multi-source data such as images, infrared, and pressure from the maintenance site; S2: Preprocess the collected multi-source data, perform anti-reflective interference processing and feature extraction to generate various feature vectors; S3: The YOLOv8 algorithm is improved by adopting a spatial and channel dual attention mechanism; the feature vector is input into the improved YOLOv8 model, and the feature vector is comprehensively analyzed and judged through feature-level and decision-level two-level multimodal fusion to obtain the on-site state decision result; S4: Based on the decision results, if a safety hazard is identified, an early warning message is generated, triggering the on-site audible and visual alarm device; if the on-site condition is determined to meet the requirements, the system issues a normal signal. S5: Conduct on-site inspections and handle any potential hazards detected, such as leftover tools, unremoved signs, or personnel not evacuated. After handling, repeat steps S1-S4 until the on-site condition meets the safety standards for switching operations, at which point the process ends.

[0013] Preferably, in step S3, the secondary multimodal fusion integrates the features and decision results of the three types of data—image, infrared, and pressure—in a weighted manner, following the order of feature-level fusion followed by decision-level fusion, thereby improving the accuracy and robustness of hazard assessment.

[0014] Furthermore, step S4 also includes triple verification of personnel evacuation: establishing a facial recognition check-in and check-out system for maintenance personnel, and comparing the check-in and check-out records with infrared sensor personnel monitoring data and improved YOLOv8 algorithm personnel identification data to achieve double confirmation of the complete evacuation status of personnel.

[0015] Furthermore, in step S3, the YOLOv8 algorithm is improved and optimized specifically for hydropower station maintenance scenarios. It can achieve dedicated object classification and detection of tools, grounding wires, safety fences, and warning signs, accurately adapting to the dedicated inspection requirements for post-maintenance site cleanup of hydropower station electromechanical equipment.

[0016] Compared with the prior art, the beneficial effects of the present invention include: 1) The hydropower station maintenance site safety assurance system proposed in this invention uses high-definition cameras, infrared sensors, and pressure sensors to collect multi-source data in a coordinated manner. Combined with image recognition and multi-modal fusion technology, it can monitor potential hazards such as leftover tools, personnel not evacuated, and signs and fences not removed in a comprehensive and accurate manner, which can greatly improve the accuracy of hazard detection and reduce subjective omissions and blind spots caused by manual inspection.

[0017] 2) This invention adopts an improved YOLOv8 image recognition algorithm, introduces a spatial and channel dual attention joint mechanism, and combines it with anti-metal reflection preprocessing and small target enhancement detection structure for tools and implements. This effectively solves the recognition interference problem caused by strong metal reflection and complex lighting at the hydropower station maintenance site, and improves the detection accuracy of small tools and implements such as bolts and pins.

[0018] 3) This invention constructs a multi-verification mechanism for personnel evacuation, which deeply integrates the facial recognition check-in and check-out system of maintenance personnel with the monitoring data of the system's infrared sensors and the personnel detection data of image recognition, forming a triple comparison and double confirmation mechanism for personnel evacuation, thus completely avoiding the safety hazards of personnel staying on site.

[0019] 4) This invention constructs a full-process safety assurance system from data collection, analysis and judgment, early warning push to hidden danger handling and closed-loop verification. At the same time, it establishes a system reliability assurance mechanism with multiple verifications, regular calibration and maintenance and data backup, which greatly improves the stability and reliability of the system and provides a solid technical guarantee for the equipment safety and personal safety of subsequent switching operations after the hydropower station is overhauled. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Figure 1 This is a schematic diagram of the module architecture of the hydropower station maintenance site safety assurance system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the on-site safety assurance method for hydropower station maintenance according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the improved YOLOv8 network structure with integrated spatial attention mechanism according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the principle of the multimodal fusion technology according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating the personnel management and system collaboration process according to an embodiment of the present invention; Detailed Implementation The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the hydropower station maintenance site safety assurance system based on multimodal fusion includes: The data acquisition module is used to collect data on images, personnel presence, and object residue at the maintenance site. The data processing module is used for preprocessing and feature extraction of the collected data; The analysis and judgment module introduces a spatial and channel dual attention mechanism and multimodal fusion to improve the YOLOv8 image recognition algorithm, which is used to analyze and judge feature vectors and generate early warning information. The early warning feedback module is used to receive and transmit early warning information. When the analysis and judgment module performs multimodal fusion, it uses a feature-level fusion formula: F = α·F1 + β·F2 + γ·F3; In the formula, F represents the fused feature vector, F1 represents the image feature vector, F2 represents the infrared sensor feature vector, F3 represents the pressure sensor feature vector, and α, β, and γ represent the weighting coefficients, with α+β+γ=1. The YOLOv8 algorithm is improved by embedding a joint mechanism of spatial attention and channel attention. Anti-metal reflection preprocessing and small target enhancement detection structure for tools are configured for hydropower station maintenance scenarios. This is used to accurately identify areas where tools are easily left behind, such as workbenches, equipment corners, and tool racks.

[0024] The data acquisition module includes a high-definition camera, an infrared sensor, and a pressure sensor deployed at the maintenance site. The high-definition camera is used to provide full coverage of the maintenance work surface, tool area, and entrances and exits. The infrared sensor is used to monitor the thermal radiation of personnel entering and exiting. The pressure sensor is used to monitor the placement status of the workbench and tool rack.

[0025] The analysis and judgment module includes using decision-level fusion to calculate the final decision result by weighting the image recognition decision result, the infrared sensor decision result, and the pressure sensor decision result. The calculation formula is as follows: D = ω1·D1 + ω2·D2 + ω3·D3; In the formula, D represents the final decision result, D1 represents the image recognition decision result, D2 represents the infrared sensor decision result, D3 represents the pressure sensor decision result, and ω1, ω2, and ω3 represent the decision weight coefficients, respectively.

[0026] Attention mechanisms include spatial attention mechanisms and channel attention mechanisms; spatial attention mechanisms are used to locate key areas where tools and equipment are easily left behind; channel attention mechanisms are used to enhance the feature weights of tools, signs, fences, and other hydropower station-specific detection objects.

[0027] The data processing module performs Gaussian filtering, histogram equalization, size normalization, and anti-metal reflection suppression preprocessing on the image data, and extracts the shape, outline, and position features of tools, signs, fences, and people.

[0028] The early warning feedback module adopts a three-level early warning mode, including on-site audible and visual alarms, information push from operation and maintenance terminals, and linkage early warning with the switching operation control system. If the hidden danger is not dealt with within the preset time limit, the system will automatically send a prohibition signal to the switching operation control system.

[0029] For example, a multimodal fusion-based hydropower station maintenance site safety assurance system, or a multimodal fusion-based hydropower station maintenance site safety assurance method. Figure 2 As shown, it includes the following steps: S1: After the maintenance work is completed, collect multi-source data such as images, infrared, and pressure from the maintenance site.

[0030] S2: Preprocess the collected multi-source data, perform anti-reflective interference processing and feature extraction to generate various feature vectors; S3: The YOLOv8 algorithm is improved by adopting a spatial and channel dual attention mechanism; the feature vector is input into the improved YOLOv8 model, and the feature vector is comprehensively analyzed and judged through feature-level and decision-level two-level multimodal fusion to obtain the on-site state decision result; S4: Based on the decision results, if a safety hazard is identified, an early warning message is generated, triggering the on-site audible and visual alarm device; if the on-site condition is determined to meet the requirements, the system issues a normal signal. S5: Conduct on-site inspections and handle any potential hazards detected, such as leftover tools, unremoved signs, or personnel not evacuated. After handling, repeat steps S1-S4 until the on-site condition meets the safety standards for switching operations, at which point the process ends.

[0031] In step S3, the second-level multimodal fusion, following the order of feature-level fusion followed by decision-level fusion, integrates the features and decision results of the three types of data—image, infrared, and pressure—layer by layer with weights to improve the accuracy and robustness of hazard assessment.

[0032] Step S4 also includes triple verification of personnel evacuation: establish a facial recognition check-in and check-out system for maintenance personnel, and perform triple comparison between the check-in and check-out records and the personnel monitoring data from the infrared sensor and the personnel identification data from the improved YOLOv8 algorithm to achieve double confirmation of the complete evacuation status of personnel.

[0033] In step S3, the YOLOv8 algorithm is improved and optimized specifically for hydropower station maintenance scenarios. It can achieve dedicated object classification and detection of tools, grounding wires, safety fences, and warning signs, accurately adapting to the dedicated inspection requirements of post-maintenance site cleanup of hydropower station electromechanical equipment.

[0034] Specifically, system hardware deployment and multi-source data acquisition: This invention was implemented in the maintenance area of ​​the No. 3 generator at a large hydropower station. The maintenance area has a working area of ​​80㎡ and includes three functional zones: the main unit maintenance area, the tool storage area, and the temporary passage.

[0035] The data acquisition module is deployed as follows: 1) Deployment of high-definition cameras: Six 4-megapixel high-definition cameras are installed at four points on the top of the perimeter of the maintenance area and two points above the passageway. They use 120° wide-angle lenses to cover the entire maintenance work surface, tool storage area and the only entrance and exit from a 30° downward angle. The protection level is IP67, and it supports automatic white balance adjustment and adaptive focus. The adaptive focus range is 2.8-12mm, ensuring that there are no blind spots in areas where tools are easily left behind, such as corners of equipment and gaps in tool racks. The image acquisition frame rate is set to 2 frames / second and the resolution is 1080P.

[0036] 2) Infrared sensor deployment: Four HC-SR501 enhanced infrared sensors are installed on both sides of the entrance and exit door frame of the maintenance area and at the junction of the tool storage area and the main unit maintenance area. The detection angle is 150° and the detection distance is 8 meters. The installation height is adjusted to 1.2m and the sensing sensitivity is adjusted to medium level. It only responds to human body heat radiation of 36-38℃, effectively eliminating the interference of the operating temperature of electromechanical equipment. The false alarm rate is controlled within 2%, and it can output the switch signal of personnel entry and exit in real time.

[0037] 3) Pressure Sensor Deployment: Six high-precision strain gauge pressure sensors are embedded in the layered surfaces of two tool racks and one temporary workbench. The sensor surfaces are covered with a 2mm thick anti-slip, wear-resistant rubber layer and secured with shock-absorbing pads and insulating tape to prevent direct contact with mechanical equipment. The pressure measurement accuracy is ±0.2%FS, with a sampling frequency of 2Hz, used for accurate detection of the placement / residual status of tools such as wrenches, screwdrivers, and bolts. The baseline calibration cycle is set to once every hour to effectively eliminate zero drift error. The tool racks have six layered surfaces, and the strain gauge pressure sensors are model HBMPW10AC3 / 10kg.

[0038] 4) Auxiliary Equipment Deployment: Install one ZKTECO ZKBioFace702 facial recognition check-in / check-out terminal at the entrance and exit of the maintenance area. The terminal has a recognition accuracy of ≥99.9% and a response time of ≤1 second, accurately recording personnel names, entry / exit times, work zones, and other information. Install one audible and visual alarm each at the main unit maintenance area and the entrance / exit. The alarm volume is ≥100dB, supporting a dual warning mode of flashing red light + buzzer. The audible and visual alarm model is TBJ-150.

[0039] Data Acquisition Process: After the maintenance work is completed, the system automatically starts the data acquisition process when the staff submits the maintenance completion application; the high-definition camera continuously collects on-site image data, the infrared sensor monitors personnel entry and exit signals in real time, and the pressure sensor periodically uploads pressure data; all collected multi-source data are transmitted to the local edge computing gateway through industrial Ethernet with a transmission delay of ≤50ms, providing high-quality basic data for subsequent data processing and analysis.

[0040] Data preprocessing and hydropower station-specific feature extraction: Based on the acquired multi-source data including images, infrared, and pressure, targeted preprocessing and feature extraction operations are performed in the data processing module to complete data cleaning, optimization, and feature quantization, generating standardized feature vectors. This includes the following steps: 1) Image data preprocessing: The OpenCV library is used to preprocess the images captured by the high-definition camera. Gaussian filtering is performed using a 5×5 convolution kernel to remove Gaussian noise caused by dust and light refraction in the industrial environment. For the core problem of strong metal reflection in the hydropower station scene, adaptive reflection suppression processing is performed. Reflective areas are accurately identified by grayscale threshold segmentation, and the neighbor pixel mean replacement method is used to weaken reflection interference. At the same time, histogram equalization is performed to enhance image contrast. Finally, the size of the preprocessed image is normalized to 640×640 pixels and converted to RGB format to adapt to the input requirements of the improved YOLOv8 model.

[0041] 2) Sensor data preprocessing: Infrared sensor data: The output analog signal is converted into a 10-bit A / D converter through the Arduino Uno R3 development board, and then filtered out by a digital low-pass filter to remove interference from ambient temperature fluctuations. Finally, the binary data of "personnel present" / "no personnel present" is output. Pressure sensor data: The raw data collected is processed by a baseline calibration algorithm, and the current environmental pressure baseline value is subtracted. If the processed pressure value exceeds 0.5 kg, it is determined that "tools are placed"; otherwise, it is determined that "no tools are placed".

[0042] 3) Feature extraction: Image feature extraction: The improved YOLOv8n model is used to extract features from the preprocessed images. This model is trained on a dedicated dataset for hydropower stations, which includes 12 types of tools, grounding wires, safety fences, warning signs, and personnel, totaling 16 target categories. The dataset contains 000 samples, covering complex scene samples such as strong reflections, shadows, and occlusions. It can accurately extract the shape, color, and position features of the targets, and finally outputs a 512-dimensional image feature vector F1. Infrared sensor feature extraction: Extract key features such as personnel entry and exit timestamps, dwell time, and entry and exit frequency, and quantify them into a 64-dimensional feature vector F2; Pressure sensor feature extraction: Calculate the maximum, minimum and fluctuation variance of the pressure values ​​of each sensor, and combine them with the partition information of the tool rack / workbench to form a 32-dimensional feature vector F3; F1, F2, and F3 are concatenated according to their dimensions to form a 608-dimensional pre-fusion feature matrix, which serves as the input data for the analysis and judgment module.

[0043] Specifically, we improved the YOLOv8 model and dual attention mechanism for target recognition and multimodal fusion decision-making.

[0044] In the analysis and judgment module, based on the 608-dimensional pre-fusion feature matrix, target recognition and multimodal fusion decision-making are performed to achieve accurate judgment of the maintenance site status, including the following steps: 1) Improve target recognition using the YOLOv8 model like Figure 3 As shown, targeted structural improvements were made to the YOLOv8n model to adapt it to the detection requirements of hydropower station maintenance scenarios: Embedded dual attention mechanism: A joint mechanism of spatial attention (SA) and channel attention (CA) is embedded after the C2f module of YOLOv8n. The SA module generates a spatial attention map with a weight range of 0-1 to enhance the features of areas where tools are easily left behind, such as tool rack corners (weight 0.9) and equipment bases (weight 0.85). The CA module adaptively adjusts the feature channel weights, assigning high weights of 0.7-0.9 to the feature channels of hydropower station-specific targets such as tools, grounding wires, and safety fences, and low weights of 0.1-0.3 to the background interference channels. Add a small target detection branch: Add a dedicated small target detection branch to the model detection head. For micro tools ≤5cm, enhance the feature representation of small targets through feature pyramid downsampling, thereby improving the detection accuracy of micro tools to 96%; Output recognition results: After the model completes target recognition, it outputs the target category, bounding box coordinates and confidence score. The confidence score threshold is set to 0.8. If the score is lower than the threshold, it is determined that there is no target. Finally, the image recognition decision result D1 is formed.

[0045] 2) Multimodal fusion decision like Figure 4 As shown, a two-level multimodal fusion approach, employing feature-level and decision-level methods, is used to achieve deep integration of multi-source data and improve the accuracy of decision-making results. Feature-level fusion: According to the feature-level fusion formula F=α F1+β F2+γ F3, after multiple on-site tests and optimizations at hydropower stations, sets the weight coefficients for image features to α=0.6, infrared sensor features to β=0.25, and pressure sensor features to γ=0.15, satisfying α+β+γ=1. A 608-dimensional fused feature vector F is obtained by weighted summation. Decision-level fusion: Based on the decision-level fusion formula D=ω1 D1+ω2 D2+ω3 D3 sets decision weights ω1=0.7, ω2=0.2, and ω3=0.1 based on the reliability test results of each module; where D2 is the decision result of the infrared sensor and D3 is the decision result of the pressure sensor; the decision threshold is set to 0.7. When D≥0.7, it is judged as "there is a safety hazard", otherwise it is judged as "the site is safe".

[0046] 3) Typical Decision-Making Examples If the image recognition detects a bolt at the corner of the tool rack (D1=1, confidence level 0.92), the infrared sensor does not detect personnel on site (D2=0), and the pressure sensor detects pressure at the corresponding tool rack layer (D3=1), then the fusion decision result is D=0.7×1+0.2×0+0.1×1=0.8≥0.7, and the system determines it as "a hidden danger left in the tools".

[0047] Early warning feedback and coordinated evacuation 1) Implementation of Level 3 Early Warning Feedback When the early warning feedback module receives the "potential safety hazard" signal from the analysis and judgment module, it activates a three-level early warning mechanism within one second to achieve rapid transmission of hazard information and coordinated control: Local warning: The audible and visual alarms at the entrance and exit of the maintenance area and the main unit maintenance area are activated simultaneously. The red light flashes at a frequency of 2Hz, and the buzzer emits a rapid "beep" alarm at a 0.5-second interval. Terminal warning: Send precise SMS notifications to the mobile phones of maintenance supervisors and on-site safety officers, including the area of ​​the hazard, the type of hazard, and the time limit for handling; at the same time, pop-up warning windows appear on the web and APP of the hydropower station operation and maintenance management system, with attached on-site image screenshots and precise marking of the hazard location; Linked early warning: If the hidden danger is not dealt with within 10 minutes, the system will automatically send a prohibition signal to the switching operation control system to block the subsequent switching operation process until the hidden danger is eliminated.

[0048] 2) Triple confirmation of personnel evacuation like Figure 5 As shown, to ensure the complete evacuation of personnel from the maintenance site, a triple confirmation mechanism for personnel evacuation, consisting of facial recognition and dual-sensor detection, is constructed, including: First step: After completing their work, maintenance personnel sign out using the on-site facial recognition terminal, and the system automatically records the personnel's sign-out information; The second layer: Infrared sensors monitor the entrances and exits of the maintenance area in real time. If no personnel entry or exit signal is detected within 3 minutes after personnel sign out, it is determined that no personnel remain on the infrared level. The third step: Improve the YOLOv8 model to continuously identify personnel in the full range of images of the maintenance area. If no personnel targets are detected for 5 consecutive frames, it is determined that there are no personnel staying in the image. The system will only determine that personnel have been completely evacuated when all three of the above confirmation conditions are met; if any one of the confirmation conditions is not met, the system will immediately trigger a personnel lingering warning and notify the on-site supervisor to verify and handle the situation.

[0049] Hazard management and safety closed-loop verification Upon receiving the early warning information, on-site staff proceed to the affected area to investigate and address the issue. They then submit a hazard report via the hydropower station's operation and maintenance app, initiating a safety closed-loop verification process until the hazard is confirmed to be completely eliminated, including: 1) The data acquisition module restarts and synchronously acquires multi-source data, including images, infrared, and pressure, across the entire maintenance area for 30 seconds. 2) The analysis and judgment module performs a full-process analysis and judgment on the newly collected multi-source data: The improved YOLOv8 model did not detect any tools or equipment left behind, the pressure value of the pressure sensor corresponding to the hidden danger area was <0.5kg, and the infrared sensor did not detect any personnel signals; 3) The multimodal fusion decision result D=0.7×0+0.2×0+0.1×0=0<0.7, indicating that the system has determined that the hidden danger has been completely eliminated; 4) The early warning feedback module immediately stops the on-site audible and visual alarms, sends a notification to the maintenance supervisor, on-site safety officer and other relevant personnel that the hazard has been dealt with, and sends a switching operation control system permission signal to complete the closed loop of the entire process of on-site safety assurance after this maintenance.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A safety guarantee system for a hydropower station overhaul site based on multi-modal fusion, characterized in that, include: The data acquisition module is used to collect data on images, personnel presence, and object residue at the maintenance site. The data processing module is used for preprocessing and feature extraction of the collected data; The analysis and judgment module introduces a spatial and channel dual attention mechanism and multimodal fusion to improve the YOLOv8 image recognition algorithm, which is used to analyze and judge feature vectors and generate early warning information. The early warning feedback module is used to receive and transmit early warning information. When performing multimodal fusion, the analysis and judgment module uses a feature-level fusion formula: F = α·F1 + β·F2 + γ·F3; In the formula, F represents the fused feature vector, F1 represents the image feature vector, F2 represents the infrared sensor feature vector, F3 represents the pressure sensor feature vector, and α, β, and γ represent the weighting coefficients, with α+β+γ=1. The improved YOLOv8 algorithm incorporates a joint mechanism of spatial attention and channel attention, and is configured with anti-metal reflection preprocessing and small target enhancement detection structure for hydropower station maintenance scenarios, which is used to accurately identify areas where tools are easily left behind, such as workbenches, equipment corners, and tool racks.

2. The multimodal fusion-based safety guarantee system for hydropower station maintenance sites according to claim 1, characterized in that, The data acquisition module includes a high-definition camera, an infrared sensor, and a pressure sensor deployed at the maintenance site. The high-definition camera is used to provide full coverage of the maintenance work surface, tool area, and entrances and exits. The infrared sensor is used to monitor the thermal radiation of personnel entering and exiting. The pressure sensor is used to monitor the placement status of the workbench and tool rack.

3. The multi-modal fusion-based safety guarantee system for hydropower station maintenance site according to claim 1, characterized in that, The analysis and judgment module includes using decision-level fusion to calculate the final decision result by weighting the image recognition decision result, the infrared sensor decision result, and the pressure sensor decision result. The calculation formula is as follows: D = ω1·D1 + ω2·D2 + ω3·D3; In the formula, D represents the final decision result, D1 represents the image recognition decision result, D2 represents the infrared sensor decision result, D3 represents the pressure sensor decision result, and ω1, ω2, and ω3 represent the decision weight coefficients, respectively.

4. The multi-modal fusion-based safety guarantee system for hydropower station maintenance sites according to claim 1, characterized in that, The attention mechanism includes a spatial attention mechanism and a channel attention mechanism; the spatial attention mechanism is used to locate key areas where tools and equipment are easily left behind; the channel attention mechanism is used to enhance the feature weights of tools, signs, fences and other hydropower station-specific detection objects.

5. The hydropower station maintenance site safety assurance system based on multimodal fusion according to claim 1, characterized in that, The data processing module performs Gaussian filtering, histogram equalization, size normalization, and anti-metal reflection suppression preprocessing on the image data, and extracts the shape, outline, and position features of tools, signs, fences, and people.

6. The hydropower station maintenance site safety assurance system based on multimodal fusion according to claim 1, characterized in that, The early warning feedback module adopts a three-level early warning mode, including on-site audible and visual alarms, information push from operation and maintenance terminals, and linkage early warning with the switching operation control system. If the hidden danger is not dealt with within the preset time limit, the system will automatically send a signal to the switching operation control system to prohibit switching.

7. The method for ensuring safety at hydropower station maintenance sites based on multimodal fusion, as described in claims 1-6, is characterized in that... Includes the following steps: S1: After the maintenance work is completed, collect multi-source data such as images, infrared, and pressure from the maintenance site; S2: Preprocess the collected multi-source data, perform anti-reflective interference processing and feature extraction to generate various feature vectors; S3: The YOLOv8 algorithm is improved by adopting a spatial and channel dual attention mechanism; the feature vector is input into the improved YOLOv8 model, and the feature vector is comprehensively analyzed and judged through feature-level and decision-level two-level multimodal fusion to obtain the on-site state decision result; S4: Based on the decision results, if a safety hazard is determined to exist, an early warning message will be generated, triggering the on-site audible and visual alarm device; If the on-site condition is determined to meet the requirements, the system will issue a normal signal; S5: Conduct on-site inspections and handle any potential hazards detected, such as leftover tools, unremoved signs, or personnel not evacuated. After handling, repeat steps S1-S4 until the on-site condition meets the safety standards for switching operations.

8. The method for ensuring on-site safety during hydropower station maintenance based on multimodal fusion according to claim 7, characterized in that, In step S3, the secondary multimodal fusion integrates the features and decision results of the three types of data—image, infrared, and pressure—in a weighted manner, following the order of feature-level fusion followed by decision-level fusion, thereby improving the accuracy and robustness of hazard assessment.

9. The method for ensuring on-site safety during hydropower station maintenance based on multimodal fusion according to claim 7, characterized in that, Step S4 also includes triple verification of personnel evacuation: establishing a facial recognition check-in and check-out system for maintenance personnel, and comparing the check-in and check-out records with infrared sensor personnel monitoring data and improved YOLOv8 algorithm personnel identification data to achieve double confirmation of the complete evacuation status of personnel.

10. The method for ensuring on-site safety during hydropower station maintenance based on multimodal fusion according to claim 7, characterized in that, In step S3, the improved YOLOv8 algorithm is specifically optimized for hydropower station maintenance scenarios, enabling the classification and detection of specific objects such as tools, grounding wires, safety fences, and warning signs, accurately meeting the specific inspection requirements for post-maintenance site cleanup of hydropower station electromechanical equipment.

Images