Audio and video fused electric power intelligent inspection system and audio and video fused electric power intelligent inspection method
Through the integrated power intelligent inspection system of audio and video, deep learning algorithms are used to analyze audio and video data, the problem of insufficient inspection efficiency and quality in the existing technology is solved, automatic continuous inspection of power equipment is realized, and industrial vision is broken. The limitations of the abnormal states are identified and solved, and the inspection efficiency and quality are improved.
Patent Information
- Application Number
- CN202411379333.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing intelligent power inspection system has shortcomings in patrol efficiency and quality, and it is particularly difficult to identify abnormal states of large rotating equipment and power equipment, and it is impossible to effectively monitor operating sounds.
Using an intelligent power inspection system with audio and video fusion, it collects audio and video data through front-end sensing devices, combines edge computing all-in-one machine and intelligent inspection center service software, and uses deep learning algorithms to analyze and process the instrument reading and switching state. , temperature measurement, leakage and other abnormal states.
It improves the efficiency and quality of inspections, realizes 24-hour automated continuous inspection of power equipment, breaks through the limitations of industrial vision, can effectively identify and resolve abnormal states, reduces personnel workload, and shortens the inspection cycle.
Smart Images

Figure CN118917836A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of electric power intelligent inspection, and specifically discloses an audio-video integrated electric power intelligent inspection system and method. Background Art
[0002] As the scale of my country's power system continues to expand, installed capacity continues to increase, and digital equipment for power production is rapidly updated and developed, the complexity increases. The scope of daily inspections of power generation companies has expanded, the inspection content has become more cumbersome, and inspectors with different skill levels find different problems during the inspection process, resulting in a lack of objectivity in the inspection results. Manual inspection methods have the problems of low inspection efficiency and uneven inspection quality. Digital and intelligent technical means are one direction to solve the problem of manual inspection.
[0003] Intelligent inspection is an important area of digital transformation and development for power companies. It uses artificial intelligence, edge computing, the Internet of Things and other technologies to build an intelligent inspection system based on automatic inspection to replace the traditional manual inspection method. The front-end camera, robot and other devices are used to collect on-site video images. The data is processed by an intelligent recognition algorithm based on deep learning to identify the status of the equipment, find problems in time, improve the management level of power production and operation, and improve the economic benefits of power generation companies. At present, the power inspection method based on industrial vision has certain limitations. It is impossible to identify the abnormal status of large rotating equipment such as feed pumps and turbines, and it is difficult to monitor the abnormal operation sound of power equipment such as main transformers and switch cabinets.
[0004] In view of this, it is necessary to improve the problems in the prior art to solve the above problems. Summary of the invention
[0005] Purpose of the invention: The present invention discloses an audio and video integrated electric power intelligent inspection system and method, the system includes a front-end sensing device, an edge computing all-in-one machine and an intelligent inspection center service software, wherein the edge computing all-in-one machine is respectively connected to the front-end sensing device and the intelligent inspection center service software; the intelligent inspection method is adopted to replace the manual inspection method, which not only solves the problems of low efficiency and uneven inspection quality of manual inspection, but also breaks through the limitations of the electric power inspection method based on industrial vision based on the intelligent recognition algorithm processing based on deep learning, and can discover and solve many difficult problems of abnormal conditions such as instrument readings, switch status, temperature measurement, leakage, etc.
[0006] To achieve the above object, the present invention provides the following technical solutions: An audio and video integrated power intelligent inspection system, including front-end sensing equipment, edge computing all-in-one machine and intelligent inspection center service software; The edge computing integrated machine is respectively connected to the front-end sensing device and the intelligent inspection center service software; The front-end sensing device is used to collect audio and video data representing the operating status and index parameters of the power equipment and send them to the edge computing integrated machine. The edge computing integrated machine is used to analyze and process the data collected by the front-end sensing device, form inspection results, and upload the inspection results to the intelligent inspection center service software; After receiving the inspection results, the intelligent inspection center service software stores the inspection results in a database for viewing and downloading.
[0007] Furthermore, the front-end sensing equipment includes a high-definition camera, an infrared dual-spectrum camera, an audio sensor, a vibration sensor, a patrol robot and a patrol drone.
[0008] Furthermore, the high-definition camera and infrared dual-spectrum camera are used to collect equipment status data of main transformers, circuit breakers, switches, mutual inductors in the station, as well as the nacelle and tower base of the wind turbine, and equipment status data in the main plant and electronic room of the thermal power plant.
[0009] Furthermore, the audio sensor and vibration sensor are used to collect status data of various pumps in the pump room of a gas turbine power plant, coal mills in a thermal power plant, and status data of turbines in a turbine room of a hydropower station.
[0010] Furthermore, the inspection robots and inspection drones are used to collect status data of switches and measurement and control equipment in switch rooms and relay protection rooms.
[0011] Furthermore, the edge computing all-in-one machine is a computing device deployed on the edge side for data collection and intelligent analysis based on domestic GPU / NPU chips, integrating video data collection components, audio data collection software, audio and video data feature extraction software, as well as instrument readings, switch status, temperature measurement and leakage intelligent identification algorithms.
[0012] Furthermore, the intelligent inspection center service software has four application modules: inspection tasks, alarm center, inspection management, and system management; The inspection task module is used for regular inspections, inspection task details viewing, inspection task management, and supports downloading inspection result reports to the local computer; The alarm center module is used to display inspection alarms, view the details of inspection alarms, and issue alarm processing instructions based on the details of inspection alarms; The inspection management module is used to add, edit or delete the task bar in the inspection task module; The system management module is used to add, edit or delete relevant information of the front-end sensing device, and can remotely view the real-time image of the camera.
[0013] Furthermore, the system management module has permission management, user organization, process engine basic services and components; The inspection task, alarm center and inspection management function module respectively integrate relational database, object storage library and middleware, and are used to store system parameters, inspection results and cache data respectively.
[0014] An audio and video integrated power intelligent inspection method includes the following inspection steps: S1. Configure regular inspection tasks in the intelligent inspection center service software, and send inspection task instructions to the edge computing all-in-one machine via the TCP-based remote transmission interface; S2. After receiving the inspection task instruction, the edge computing machine reads the front-end perception data of the camera, audio sensor and vibration sensor according to the inspection points. The front-end perception data flows to the intelligent algorithm software inside the edge computing machine, and the intelligent algorithm recognizes the audio and video data after data preprocessing. S3. The result data after the intelligent algorithm recognition is completed is uploaded to the intelligent inspection center service software through the TCP-based remote transmission channel. The algorithm recognition result is based on the JSON format, including the inspection point, result image, and voiceprint feature value information; S4. After receiving the returned inspection results, the intelligent inspection center service saves the image and audio result data in the object repository, generates a storage path link and saves the inspection result information in the relational database. The inspection result data can be viewed through the inspection task viewing page, and supports PDF report download.
[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses an audio and video fusion intelligent inspection system and method, the system includes a front-end sensing device, an edge computing integrated machine and an intelligent inspection center service software, wherein the edge computing integrated machine is respectively connected to the front-end sensing device and the intelligent inspection center service software; the system uses front-end sensing devices such as high-definition cameras, infrared dual-spectrum cameras, audio sensors, etc. to collect audio and video multimodal data at the power production site, and the data is uploaded to the edge computing integrated machine. The data is analyzed and processed by the built-in intelligent algorithm of the edge computing integrated machine to form an inspection result, and the inspection result is uploaded to the intelligent inspection center service software. After receiving the inspection result, the intelligent inspection center service software stores the inspection result in an object-based database for retrieval, viewing and downloading at any time. By using audio and video sensing technology, intelligent recognition technology based on deep learning, and developing an intelligent inspection center service based on a microservice framework, the automation of daily power inspections is realized, replacing the traditional manual inspection method.
[0016] In the present invention, video data is processed by the intelligent recognition algorithm based on deep learning to identify the equipment status and environmental status, audio data is processed based on voiceprint feature extraction and AI recognition technology, and the edge computing all-in-one machine integrates domestic GPU / NPU chips to provide an efficient computing power foundation. The present invention realizes 24-hour uninterrupted automated continuous inspection of power production equipment and its operating environment, breaking through the limitations of the power inspection method based on industrial vision, and can discover and solve many problems of abnormal conditions such as instrument readings, switch status, temperature measurement, leakage, etc. It improves the inspection efficiency and quality, reduces the workload of personnel, greatly shortens the inspection cycle of power generation enterprises, and systematically solves the pain points of manual inspection methods of power generation enterprises. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Audio and video fusion power intelligent inspection system architecture; Figure 2 Audio and video fusion intelligent inspection process; Figure 3 Meter reading model. DETAILED DESCRIPTION
[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0019] See also Figure 1-2 The present invention provides an audio and video integrated power intelligent inspection system and method, the intelligent inspection system architecture is as follows Figure 1 As shown, the intelligent inspection system described in the present invention is mainly aimed at the field of power inspection, integrating technologies such as the Internet of Things, edge computing, artificial intelligence, and microservices to create an edge computing all-in-one machine and build an intelligent inspection software system. By configuring front-end sensing devices such as high-definition cameras, infrared thermal imaging cameras, audio sensors, robots, and drones, the collected audio and video multimodal data is analyzed by the intelligent recognition algorithm deployed in the edge computing all-in-one device to automatically identify the power generation equipment and environmental status, and the inspection results are uploaded to the intelligent inspection center application for storage and display, realizing intelligent inspection instead of manual inspection.
[0020] The system comprises: F1. Front-end sensing equipment, mainly including audio and video sensing terminals such as visible light cameras, infrared thermal imaging cameras, sound sensors, vibration sensors, etc.; introduce IoT sensing technology, use front-end sensing terminals such as high-definition visible light cameras, infrared thermal imaging cameras, audio sensors, vibration sensors, inspection robots and inspection drones to develop audio and video multi-modal data collection and data processing software; The high-definition camera and infrared dual-spectrum camera are used to collect equipment status data of the main transformer, circuit breaker, wind turbine cabin, tower base in the booster station, and equipment status data in the main plant of the thermal power plant; The audio sensor and vibration sensor are used to collect status data of various pumps in the pump room of the gas turbine power plant and status data of the turbines in the turbine room of the hydropower station; The inspection robots and inspection drones are used to collect status data of switches and measurement and control equipment in switch rooms and relay protection rooms.
[0021] F2, edge computing all-in-one machine, is a computing device for edge-side data collection and intelligent analysis based on domestic GPU / NPU chips. It supports AI frameworks such as pytorch and tensorflow, provides basic support for the operation of intelligent recognition algorithms, and integrates intelligent recognition algorithm modules for audio and video fusion. The video data acquisition component is developed based on the computer vision library OpenCV, and the audio data acquisition software is developed based on the audio processing library Soundfile. It integrates intelligent algorithms such as instrument readings, switch status, temperature measurement, leakage, etc.
[0022] Among them, various intelligent algorithms such as instrument readings, switch status, temperature measurement, leakage, etc. integrate the classification algorithms of the yolo framework and resnet. Among them, for the instrument reading analysis algorithm based on visible light, the target detection algorithm and the dial key point prediction algorithm are combined to realize the automatic recognition and reading of the instrument. First, collect on-site video data, and establish a pointer instrument target detection data set and a pointer instrument key point prediction data set; then, perform target detection training and key point prediction training respectively; then, first perform target detection to locate the dial, and predict the key points to obtain the scale and pointer key point coordinates; finally, calculate the readings by the angle method. The instrument reading model based on visible light is as follows: Figure 3 As shown; For the temperature measurement algorithm of the high-voltage side joint of the main transformer, the detection method is to continuously collect infrared dual-spectrum camera video images of key areas and key points of the equipment, and obtain temperature measurement data such as the maximum, minimum, and average values of the infrared thermal imaging temperature of key areas and key points.
[0023] Among them, the instrument reading analysis algorithm based on visible light is combined with the target detection algorithm and the dial key point prediction algorithm to realize the automatic recognition and reading of the instrument. The algorithm training and reasoning process is as follows: first, collect the on-site video data, and establish the pointer instrument target detection data set and the pointer instrument key point prediction data set; then, perform target detection training and key point prediction training respectively; then, first target detection locates the dial, and key point prediction obtains the scale and pointer key point coordinates; finally, calculate the reading by the angle method.
[0024] The switch status analysis algorithm based on visible light is based on the ResNet classification algorithm and adapted to domestic GPU / NPU chips. Equipment status models such as substation switches and switch status lights are constructed to achieve efficient identification of equipment status.
[0025] Based on the small sample leakage algorithm, the damage of the mechanical structure of the equipment can directly affect the normal operation of the unit. By using a small amount of field data samples, supplemented by small sample data processing and training, water (oil) leakage detection of key equipment can be achieved. The steps are: Data collection: Collect as many device leakage images as possible through on-site photography and Internet data crawling, and manually annotate them; Data augmentation: augment the annotated water (oil) leakage images through traditional image geometric transformation methods (including but not limited to rotation, flipping, scaling, cropping, adding noise, color space conversion, etc.); Data generation: Generate fake water leakage and oil leakage images through adversarial generative networks to further expand the training data set; 4) Algorithm training: Use all collected, amplified, and generated data to train the target detection algorithm so that it has the ability to accurately locate water (oil) leaks.
[0026] F3, intelligent inspection center service software, developed based on the Springboot framework, has four application functions: inspection tasks, alarm center, inspection management, and system management. It has basic services and components such as permission management, user organization, and process engine. It integrates relational database, object repository, and middleware to store system parameters, inspection results, cache and other data.
[0027] This intelligent inspection center service software builds an intelligent inspection basic platform based on the microservice architecture, with redis as the middleware, postgressql as the relational database, and minio as the object repository. It develops basic services such as permission management and user management, introduces activiti as the process engine, and Quartz as the inspection task scheduling management engine, providing a basis for the smooth operation of each process of intelligent inspection; it develops intelligent inspection application functions, which have four application functions: inspection tasks, alarm center, inspection management, and system management; Among them, the inspection task function supports the retrieval and details viewing of regular inspection tasks, and supports the downloading of inspection reports to the local computer. The alarm center function supports the detection, details viewing, and alarm processing of inspection alarms, and provides an API interface to support the push of alarms to other systems. The inspection management function supports the addition, editing, and deletion of inspection tasks, and the system management function supports the addition, editing, and deletion of sensing devices such as front-end cameras, and can remotely view the real-time images of cameras.
[0028] Table 1 List of intelligent inspection application function pages ; Based on the above inspection system, an audio and video fusion power intelligent inspection method is developed, which includes the following steps: S1. Configure regular inspection tasks in the intelligent inspection center service software, and send inspection task instructions to the edge computing all-in-one machine via the TCP-based remote transmission interface; S2. After receiving the inspection task instruction, the edge computing machine reads the front-end perception data of the camera, audio sensor and vibration sensor according to the inspection points. The front-end perception data flows to the intelligent algorithm software inside the edge computing machine, and the intelligent algorithm recognizes the audio and video data after data preprocessing. S3. The result data after the intelligent algorithm recognition is completed is uploaded to the intelligent inspection center service software through the TCP-based remote transmission channel. The algorithm recognition result is based on the JSON format, including the inspection point, result image, and voiceprint feature value information; S4. After receiving the returned inspection results, the intelligent inspection center service saves the image and audio result data in the object repository, generates a storage path link and saves the inspection result information in the relational database. The inspection result data can be viewed through the inspection task viewing page, and supports PDF report download.
[0029] After the edge computing all-in-one machine described in the inspection method steps of this embodiment receives the inspection task instruction, it reads the front-end perception data of the camera, audio sensor, vibration sensor, etc. according to the inspection point, and the front-end perception data flows from the edge computing all-in-one machine to the intelligent algorithm software. The collection of its audio and video data mainly uses the Internet of Things technology to convert the analog signals obtained by the audio sensor and vibration sensor into digital signals and transmit them to the edge computing all-in-one machine, where the audio data is in wav format. The camera video data is pushed to the edge computing all-in-one machine based on protocols such as GB / T28181. The data acquisition software integrated in the edge computing all-in-one machine collects pictures based on OpenCV, and the pictures are sent to the intelligent algorithm for recognition.
[0030] The present invention provides an audio and video fusion intelligent inspection system and method for power generation enterprises, which mainly utilizes high-definition cameras, infrared dual-spectrum cameras, audio sensors, vibration sensors and other intelligent sensing terminals to collect data such as the operating status and index parameters of power equipment, and timely discovers equipment abnormalities and safety hazards through data processing means such as intelligent analysis and numerical calculation, and realizes an intelligent inspection mode mainly supported by intelligent equipment and software systems, which is applied to the daily inspection business of power generation enterprises such as gas turbine power plants, hydropower plants, and coal-fired power plants.
[0031] Compared with the prior art, the beneficial effects of the audio and video integrated power intelligent inspection system and method of the present invention are: introducing Internet of Things perception technology, introducing voiceprint perception technology on the basis of traditional industrial vision, and applying it to power intelligent inspection, creating an edge computing all-in-one machine based on domestic GPU / NPU chips, completing the domestic software and hardware adaptation of yolo and restnet algorithms, integrating intelligent algorithms such as instrument readings, switch status, temperature measurement, leakage, etc.; and constructing an audio and video integrated intelligent inspection application with four main functions: inspection tasks, alarm center, inspection management, and system management; solving the limitations of traditional intelligent inspections that rely only on industrial vision, and expanding the voiceprint-based monitoring methods for rotating equipment and primary power equipment of power generation enterprises.
[0032] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. An audio and video integrated power intelligent inspection system, characterized in that: Including front-end sensing equipment, edge computing all-in-one machine and intelligent inspection center service software; The edge computing integrated machine is respectively connected to the front-end sensing device and the intelligent inspection center service software; The front-end sensing device is used to collect audio and video data representing the operating status and index parameters of the power equipment and send them to the edge computing integrated machine. The edge computing integrated machine is used to analyze and process the data collected by the front-end sensing device, form inspection results, and upload the inspection results to the intelligent inspection center service software; After receiving the inspection results, the intelligent inspection center service software stores the inspection results in a database for viewing and downloading.
2. According to the audio and video fusion power intelligent inspection system of claim 1, it is characterized by: The front-end sensing equipment includes a high-definition camera, an infrared dual-spectrum camera, an audio sensor, a vibration sensor, a patrol robot and a patrol drone.
3. According to claim 2, the audio and video fusion power intelligent inspection system is characterized in that: The high-definition camera and the infrared dual-spectrum camera are used to collect equipment status data of the main transformer, circuit breaker, nacelle of the wind turbine generator, tower base in the booster station, and equipment status data in the main plant of the thermal power plant.
4. According to the audio and video fusion power intelligent inspection system of claim 2, it is characterized in that: The audio sensor and the vibration sensor are used to collect the status data of various pumps in the pump room of the gas turbine power plant and the status data of the turbines in the turbine room of the hydropower station.
5. The power intelligent inspection system with audio and video fusion according to claim 2 is characterized in that: The inspection robots and inspection drones are used to collect status data of switches and measurement and control equipment in switch rooms and relay protection rooms.
6. The power intelligent inspection system with audio and video fusion according to claim 1 is characterized in that: The edge computing all-in-one machine is a computing device deployed on the edge side for data collection and intelligent analysis based on domestic GPU / NPU chips. It integrates video data collection components, audio data collection software, and intelligent recognition algorithms for instrument readings, switch status, temperature measurement, and leakage.
7. The power intelligent inspection system with audio and video fusion according to claim 1 is characterized in that: The intelligent inspection center service software has four application modules: inspection task, alarm center, inspection management, and system management; The inspection task module is used for regular inspections, inspection task details viewing, inspection task management, and supports downloading inspection result reports to the local computer; The alarm center module is used to display inspection alarms, view the details of inspection alarms, and issue alarm processing instructions based on the details of inspection alarms; The inspection management module is used to add, edit or delete the task bar in the inspection task module; The system management module is used to add, edit or delete relevant information of the front-end sensing device, and can remotely view the real-time image of the camera.
8. The power intelligent inspection system with audio and video integration according to claim 7 is characterized in that: The system management module has authority management, user organization, process engine basic services and components; The inspection task module, the alarm center module, and the inspection management module respectively integrate a relational database, an object storage library, and middleware, and are respectively used to store system parameters, inspection results, and cache data.
9. An audio and video fusion power intelligent inspection method, characterized in that: The inspection steps include the following: S1. Configure regular inspection tasks in the intelligent inspection center service software, and send inspection task instructions to the edge computing all-in-one machine via the TCP-based remote transmission interface; S2. After receiving the inspection task instruction, the edge computing machine reads the front-end perception data of the camera, audio sensor and vibration sensor according to the inspection points. The front-end perception data flows to the intelligent algorithm software inside the edge computing machine, and the intelligent algorithm recognizes the audio and video data after data preprocessing. S3. The result data after the intelligent algorithm recognition is completed is uploaded to the intelligent inspection center service software through the TCP-based remote transmission channel. The algorithm recognition result is based on the JSON format, including the inspection point, result image, and voiceprint feature value information; S4. After receiving the returned inspection results, the intelligent inspection center service saves the image and audio result data in the object repository, generates a storage path link and saves the inspection result information in the relational database. The inspection result data can be viewed through the inspection task viewing page, and supports PDF report download.
Citation Information
Patent Citations
Intelligent sensing inspection system and method for transformer substation
CN114665608A
Intelligent inspection system and method for transformer substation
CN115296193A
Edge detection system for power grid inspection and monitoring
CN116846059A
Data transmission method and system using combination of national cryptographic algorithm and DL / T476 protocol
CN118316696A
Substation inspection robot system based on artificial intelligence
WO2022037278A1
Cited By
Intelligent sensing system for power operation and maintenance
CN119298402A
Unmanned aerial vehicle multi-dimensional perception intelligent inspection early warning system and method based on edge calculation
CN119363820A
