Power equipment inspection method, device and equipment based on neural network model

Abstract

The invention relates to the technical field of intelligent inspection of electrical equipment, and discloses an electrical equipment inspection method, device and equipment based on a neural network model. The method achieves the high-precision detection and dynamic optimization scheduling of the defects of the key parts of the power equipment, improves the multi-source data fusion efficiency, improves the defect recognition accuracy, and reduces the operation and maintenance cost.

Description

Technical Field

[0001] This application relates to the field of intelligent inspection technology for power equipment, and in particular to a method, apparatus and equipment for power equipment inspection based on a neural network model. Background Technology

[0002] With the expansion of the power grid and the increasing complexity of its operating environment, critical components such as insulators in power equipment have become core links in ensuring power grid security. Traditional power equipment inspection mainly relies on manual inspection methods, which suffer from low efficiency and unstable detection accuracy. Although image recognition technology developed in recent years has achieved automated inspection to some extent, it still faces the following technical bottlenecks: First, existing systems lack the ability to parse multimodal control commands, making it difficult to accurately understand complex commands that include voice, text, and image interactions; second, multi-source heterogeneous data (such as visible light images, infrared thermal images, and acoustic data) lacks an effective spatiotemporal alignment and fusion mechanism; third, defect detection algorithms generally adopt a single model architecture, resulting in insufficient accuracy in locating minor defects in critical components of power equipment and failing to achieve cross-modal correlation analysis of defect features. In addition, existing solutions lack dynamic optimization capabilities in task scheduling, making it difficult to rationally allocate inspection resources based on equipment status and task priority. These technical deficiencies lead to prominent problems in existing power inspection systems, such as large fluctuations in recognition accuracy, high risks in high-risk environments, and high operation and maintenance costs. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a method, apparatus, and equipment for power equipment inspection based on a neural network model.

[0004] The first aspect of this application provides a power equipment inspection method based on a neural network model, applied to a power inspection system, characterized in that the power inspection system includes an instruction control device, multiple inspection devices, and a cloud server that are interconnected; the method includes: The system collects control information input by the user and performs semantic analysis on the control information through the command control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generates control commands that can be executed by the multiple inspection devices; the control information includes at least one of voice commands, text commands and image interaction commands. Based on the control command, at least one of the multiple inspection devices is controlled to inspect the target power equipment, and multi-angle image acquisition, environmental parameter monitoring and local sound wave detection are performed on the target power equipment to obtain image data and video data; The key components of the defect areas in the image data and video data are intelligently located and labeled using a deep reinforcement learning (DRL) model, generating labeled image data and labeled video data with bounding boxes, and then uploaded to the server. A preset neural network model is invoked to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and the extracted multimodal features are fused to obtain fused features. The neural network model is a neural network model fused with Transformer and YOLOX. The recognition and annotation results are generated based on the fusion features and added to the position of the annotation box for display.

[0005] Furthermore, this application also proposes that controlling at least one of a plurality of inspection devices to inspect the target power equipment based on the control command includes: Based on the coordinate information of the target power equipment, an inspection path for at least one inspection device is planned according to preset path planning constraints, and the path length of the inspection path is optimized using the A* algorithm to obtain the target inspection path. The movement strategy of the target inspection path is dynamically adjusted based on the task priority. The movement strategy includes directional movement, speed adjustment and obstacle avoidance. The shortest path priority strategy is adopted for tasks with high priority. By combining a PID control algorithm with a V2X communication protocol, the path trajectory of the inspection equipment is adjusted in real time to ensure that the inspection equipment accurately reaches the location of the target power equipment.

[0006] Furthermore, this application also proposes that the process of acquiring multi-angle images, monitoring environmental parameters, and detecting local acoustic waves of the target power equipment to obtain image data and video data includes: Establish communication connections between the inspection equipment and multispectral cameras, infrared thermal imagers, and high-definition video cameras deployed at different angles on the target power equipment; After establishing a communication connection, the inspection equipment emits detection sound waves of a preset frequency and simultaneously reads visible light images, infrared thermal images, and video data recorded by the multispectral camera, infrared thermal imager, and high-definition video camera. Spatiotemporal alignment of visible light images, infrared thermograms, and video data is performed using a time synchronization device (TSD). The visible light image and the infrared thermal image are stitched together using an image stitching algorithm to generate image data.

[0007] Furthermore, this application also proposes that the step of intelligently locating and labeling key components of defect regions in the image data and video data using a deep reinforcement learning (DRL) model to generate labeled image data and labeled video data with bounding boxes includes: Initialize the Q-network parameters of the deep reinforcement learning (DRL) model, define the action space as a bounding box, and determine the coordinates and size scaling of the bounding box based on the inspection parameters to obtain the target deep reinforcement learning (DRL) model. The image data and video data are sequentially input into the target deep reinforcement learning (DRL) model to identify defective regions of key components and add labeled boxes. The intersection-union ratio (IOU) between the bounding box and the defect region is evaluated based on the reward function, and the position of the bounding box is iteratively optimized by combining the IOU with the Q-learning algorithm until the IOU meets a preset threshold. The annotation box is superimposed on the image data or the video data as a semi-transparent rectangle to generate annotated image data and annotated video data with annotation boxes. The color of the frame line of the annotation box is dynamically adjusted according to the defect type during superposition.

[0008] Furthermore, this application also proposes that the step of calling a preset neural network model to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and fusing the extracted multimodal features to obtain fused features, including: The Transformer and YOLOX fusion model is invoked to extract the color features, texture features, and shape features of the labeled image data, as well as the temporal motion features of the labeled video data, and the region of interest for each feature is visualized through the Class Activation Graph (CAM). Weights of each feature are assigned through an attention mechanism to generate weighted fusion features, wherein the weights of each feature extracted from the image data are determined by the heatmap intensity of the class activation map. The spatial correlation between features is modeled using a graph convolutional network (GCN), and the feature sharing between the image data and the video data is added by combining a federated learning framework to generate the final fused features.

[0009] Furthermore, this application also proposes that the generation of recognition and annotation results based on the fused features includes: A classification decision tree is constructed based on the fused features, and the cross-entropy loss function is used to optimize each node in the classification decision tree; The CART algorithm is used to recursively divide the feature space in each annotation box based on the optimized classification decision tree, generate a probability distribution map of defect type and severity, and associate the probability distribution map with the annotation box to obtain the final identification and annotation results of the inspection data.

[0010] Furthermore, this application also proposes that the method further includes: Collect historical inspection data and manually corrected records to build an incremental training dataset; Deep reinforcement learning (DRL) models and neural network models are trained periodically based on the training dataset to fine-tune the model parameters.

[0011] A second aspect of this application provides a power equipment inspection device based on a neural network model, applied to a power inspection system. The power inspection system includes an instruction control device, multiple inspection devices, and a cloud server that are interconnected. The device includes: The system collects control information input by the user and performs semantic analysis on the control information through the command control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generates control commands that can be executed by the multiple inspection devices; the control information includes at least one of voice commands, text commands and image interaction commands. Based on the control command, at least one of the multiple inspection devices is controlled to inspect the target power equipment, and multi-angle image acquisition, environmental parameter monitoring and local sound wave detection are performed on the target power equipment to obtain image data and video data; The key components of the defect areas in the image data and video data are intelligently located and labeled using a deep reinforcement learning (DRL) model, generating labeled image data and labeled video data with bounding boxes, and then uploaded to the server. A preset neural network model is invoked to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and the extracted multimodal features are fused to obtain fused features. The neural network model is a neural network model fused with Transformer and YOLOX. The recognition and annotation results are generated based on the fusion features and added to the position of the annotation box for display.

[0012] A third aspect of this application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to execute the above-described power equipment inspection method based on a neural network model.

[0013] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned power equipment inspection method based on a neural network model.

[0014] The technical solution provided in this application involves collecting control information input by the user and performing semantic analysis on the control information through a command control device to extract the target power equipment to be inspected, inspection parameters, and task priorities, generating multiple control commands executable by the inspection equipment. The control information includes at least one of voice commands, text commands, and image interaction commands. Based on the control commands, at least one of the multiple inspection equipment is controlled to inspect the target power equipment, and multi-angle image acquisition, environmental parameter monitoring, and local acoustic wave detection are performed on the target power equipment to obtain image data and video data. A deep reinforcement learning (DRL) model is used to intelligently locate and label key components in the defect areas of the image data and video data, generating labeled image data and labeled video data with labeled boxes, which are then uploaded to the server. A preset neural network model is called to extract multimodal features from the labeled box positions of the labeled image data and labeled video data on the server, and the extracted multimodal features are fused to obtain fused features. The neural network model is a neural network model fused with Transformer and YOLOX. Based on the fused features, recognition and labeling results are generated and added to the labeled box positions for display. The method provided in this application obtains control commands by parsing the control information of the inspection equipment, and then controls the inspection equipment to collect image and video data of the target power equipment. By combining a deep learning model and a neural network model that integrates Transformer and YOLOX, the image and video data are labeled with annotation information that matches the inspection parameters in the control information. This method significantly improves the annotation efficiency and the accuracy of inspection identification, enhances the overall power inspection efficiency, reduces the missed detection rate, and provides intelligent technical support for the safe operation of the power grid. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of one embodiment of the power equipment inspection method based on a neural network model in this application. Figure 2 This is a schematic diagram of one embodiment of the power equipment inspection device based on a neural network model in this application. Figure 3 This is a schematic diagram of one embodiment of the electronic device described in this application. Detailed Implementation

[0016] To address the aforementioned issues, the power equipment inspection method proposed in this application implements weakly supervised annotation through deep reinforcement learning (DRL), optimizes feature extraction by integrating Transformer and YOLOX, and improves the NMS algorithm to enhance the accuracy of detection box screening. This solves the technical bottlenecks of traditional methods, such as strong reliance on manual annotation, difficulty in identifying defects in small samples, and missed detection of multiple targets.

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] In existing technologies, power equipment inspection mainly relies on manual operation or single sensor data acquisition. Traditional methods use fixed-path inspection equipment combined with basic image processing algorithms for defect identification. Due to the complex and widely distributed operating environment of power equipment, existing technologies struggle to achieve simultaneous multi-dimensional data acquisition and intelligent analysis, resulting in low inspection efficiency, high risks in high-risk environments, and an inability to meet the real-time requirements of power grid safety maintenance. For example, minute defects such as surface cracks or dirt accumulation on insulators are often missed or misjudged due to insufficient image resolution or environmental interference, further increasing time costs and safety hazards through manual verification.

[0019] To address the aforementioned issues, the inventors discovered that existing technologies suffer from drawbacks such as difficulties in multi-device collaboration, weak data fusion capabilities, and insufficient defect location accuracy. Analysis revealed that a single inspection device cannot cover the multi-angle inspection needs in complex environments, and traditional image processing algorithms lack dynamic optimization capabilities. Therefore, they proposed constructing a multi-device collaborative system that combines intelligent control command parsing with a deep learning model to achieve an automated closed-loop data acquisition and analysis process. Furthermore, they considered introducing a multi-modal feature fusion mechanism, using reinforcement learning to dynamically adjust the bounding box position to solve the problem of inaccurate positioning of key components in complex scenarios. Ultimately, a multi-layered data processing flow based on a neural network model was formed, covering the entire chain of optimization from command parsing to defect identification.

[0020] Therefore, this application proposes a power equipment inspection method based on a neural network model, applicable to a power inspection system comprising a command control device, multiple inspection devices, and a cloud server. The method includes collecting user-input control information and generating control commands through semantic analysis; controlling the inspection devices to perform multi-angle data acquisition; utilizing a deep reinforcement learning model for defect localization and annotation; calling a fusion neural network model to extract multimodal features and generate recognition results; and finally overlaying and displaying the annotation boxes.

[0021] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the power equipment inspection method based on a neural network model in this application includes: 101. Collect user-input control information, and perform semantic analysis on the control information through the command control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generate multiple control commands that can be executed by the inspection equipment; the control information includes at least one of voice commands, text commands and image interaction commands.

[0022] The command control device here can be understood as a command generation model, or as an intelligent processing module capable of parsing voice, text, and image interaction commands. Specifically, it can be implemented using a pre-trained language model combined with a rule engine to convert user input into structured control commands. After the user inputs control commands via voice or text, the system first performs noise filtering and semantic parsing to generate standardized commands containing device coordinates and detection parameters.

[0023] The inspection parameters can be understood as the content that needs to be monitored, such as the damage status of various core components in power equipment, such as insulators. The task priority can be understood as the order in which different target power equipment are inspected.

[0024] 102. Based on control commands, control at least one of multiple inspection devices to inspect the target power equipment, and perform multi-angle image acquisition, environmental parameter monitoring and local acoustic wave detection on the target power equipment to obtain image data and video data.

[0025] The inspection equipment here can be drones, ground robots, and handheld terminals. After receiving control commands, based on the specific number and location of the target power equipment, the corresponding number of inspection equipment is selected, and then the inspection path is planned. The inspection is controlled based on the planned path to obtain image data and video data of each target power equipment.

[0026] 103. Using a deep reinforcement learning (DRL) model, key components of defect areas in image and video data are intelligently located and labeled, generating labeled image and video data with bounding boxes, and then uploaded to the server.

[0027] It should be noted that this deep reinforcement learning model refers to a dynamic optimization algorithm based on Q-networks and reward mechanisms. Specifically, it can be implemented using Q-learning combined with intersection-union ratio (IU) evaluation, and used to iteratively adjust the positional accuracy of defect annotation boxes.

[0028] In this embodiment, the targets for identification and labeling, such as core components like insulators, are first determined based on the inspection parameters obtained through semantic analysis in step 101. Based on the targets, corresponding features are matched from a pre-set identification feature library. The corresponding features, image data, and video data are then fed into the DRL model for identification and labeling. During identification, candidate regions are first identified using the model. Then, image features within the candidate regions are identified based on the corresponding features to determine the targets and their locations, and bounding boxes are generated for labeling.

[0029] 104. Call the preset neural network model to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and fuse the extracted multimodal features to obtain fused features.

[0030] The neural network model is a fusion of Transformer and YOLOX. The fusion of Transformer and YOLOX refers to a feature extractor that combines an attention mechanism with an object detection framework. Specifically, it can be implemented by using a multi-head attention module connected in parallel with the YOLOX backbone network to simultaneously extract static features of images and temporal features of videos.

[0031] Multimodal feature fusion refers to a feature processing method that integrates color, texture, and motion information. Specifically, it can be implemented using graph convolutional networks combined with federated learning frameworks to enhance the feature representation capabilities of complex defects.

[0032] In practical applications, the construction steps of this Transformer and YOLOX fusion model include: Replace the YOLOX CSPDarkNet backbone network with a Transformer encoder to extract global features through sequence embedding transformation; A cross-stage partial layer (CSPL) is introduced into the feature pyramid and fused with the Transformer decoder to enhance multi-scale feature representation; A Global Attention Guidance Module (GAGM) is added to guide the model on low-detail information using information from high-level feature outputs, thereby improving the model's ability to identify minute defects such as insulator cracks.

[0033] 105. Generate recognition and annotation results based on fusion features, and add them to the annotation box for display.

[0034] By implementing the above steps and using the fusion of Transformer and YOLOX models, effective correlation and complementarity of multimodal features are achieved, solving the problem of identifying subtle defects in complex environments. At the same time, the automation level and detection accuracy of power equipment inspection are effectively improved. The need for manual intervention is reduced through multi-device collaborative operation, the rate of missed detection and false judgment is reduced by using intelligent annotation algorithms, and the ability to identify complex defects is enhanced by combining multimodal feature fusion. Ultimately, efficient, safe and accurate power equipment condition monitoring is achieved.

[0035] For step 101, in one implementation example, the step of performing semantic analysis on the control information through the command control device to extract the target power equipment to be inspected, inspection parameters, and task priorities, and generating control commands executable by the multiple inspection devices, includes: The command control device performs noise filtering and format standardization on the control information to generate pre-processed control information. The pre-trained BERT model is combined with multiple instance learning (MIL) technology to perform intent recognition on the pre-processed control information and determine the type of user needs; wherein, the multiple instance learning divides the control information into multiple instance packages, each package containing at least one positive example instruction; Based on the user demand type, a combination of rule matching and semantic parsing is used to extract the target power equipment, inspection parameters, and task priorities from the control information, and generate structured control instructions.

[0036] It should be noted that this noise filtering refers to eliminating environmental noise or irrelevant characters in the control information. Specifically, it can be achieved by using digital filtering algorithms combined with wavelet transform technology to improve the clarity of voice or image commands.

[0037] Standardization processing refers to unifying instructions from different sources into a structured text format, which can be achieved using XML or JSON templates to eliminate differences in multimodal data input.

[0038] This multi-instance learning technique refers to dividing control information into sample packets containing positive instructions for training. Specifically, an attention mechanism can be used to dynamically allocate weights to distinguish the core requirements in composite instructions.

[0039] The combination of rule matching and semantic parsing refers to quickly locating key entities through regular expressions and then using a semantic model to deepen understanding. Specifically, it can be achieved by combining dependency parsing with a domain dictionary to improve the accuracy of entity extraction.

[0040] In practical applications, the preprocessing stage uses digital filtering algorithms to remove background noise from voice commands, such as equipment operating sounds or wind noise. Simultaneously, handwritten text in image interaction commands is converted to standard text format using OCR. In the intent recognition stage, the standardized text is input into a pre-trained BERT model. Combined with multi-instance learning (MIM) techniques, long texts are segmented into multiple instance packets. For example, the compound command "Prioritize checking the transformer oil temperature of substation A and recording the ambient humidity" is split into two sub-packets: equipment inspection and parameter recording. An attention mechanism identifies the higher-priority sub-packet as the user's request type. In the entity extraction stage, dynamically generated regular expressions are used to match equipment names and parameter units. For example, the pattern "[\u4e00-\u9fa5]{2,5}substation" is used to match Chinese equipment names. Then, the optimized BERT model parses the semantic association between parameter ranges and priority encodings, ultimately generating a structured control command containing equipment ID, temperature threshold, and urgency level labels. Multi-instance learning effectively distinguishes primary and secondary requirements in compound commands. For instance, in commands that simultaneously include equipment location and environmental detection, the higher-priority equipment location requirement is accurately identified. Meanwhile, dynamically generated regular expression rules can adapt to the differences in device naming in different regions compared to fixed patterns, such as being compatible with both "XX Substation" and "XX Power Station" naming methods.

[0041] Furthermore, based on the user demand type, the target power equipment to be inspected, inspection parameters, and task priorities are extracted from the control information using a combination of rule matching and semantic parsing, and structured control instructions are generated, including: Based on the user requirement type, regular expression rules are used to perform pattern matching on multiple example packages divided from the control information to extract key entities. These regular expression rules are dynamically generated based on historical inspection data. For example, the control information is divided into multiple example packages according to the user requirement type, and each package contains at least one positive example instruction. For instance, a positive example instruction for a transformer inspection task might include voltage parameters and infrared detection requirements. The dynamically generated regular expressions scan the example packages to extract key entities such as device IDs and parameter ranges.

[0042] The parameters of the BERT model are optimized using the cross-entropy loss function, and the optimized BERT model is used to perform semantic parsing on the key entities to generate structured control instructions. These control instructions include at least the device ID, parameter range, and priority code. For example, the optimized BERT model performs contextual analysis on the extracted entities, parsing "prioritize processing substation A" into priority code P1, and combining this with the parameter range to generate structured instructions. Thus, the combination of rule matching and semantic parsing can cover both explicit patterns and implicit semantics, ensuring the completeness and accuracy of the instruction parameters.

[0043] The regular expression rules here are dynamically generated based on historical inspection data. Specifically, they are automatically adjusted and matched based on high-frequency entities and patterns in historical commands. This can be achieved by using a sliding window to count the frequency of device names and parameter combinations in historical commands, and then generating dynamic regular expression templates through clustering algorithms. This feature can adapt to changes in command patterns across different device types and task scenarios, avoiding the lack of generalization caused by fixed rules.

[0044] The cross-entropy loss function optimizes the parameters of the BERT model by calculating the difference between the model's predicted distribution and the true label distribution, and then adjusting the BERT model's weights accordingly. This can be achieved using a stochastic gradient descent algorithm combined with a learning rate decay strategy. This feature can improve the model's semantic parsing accuracy for electrical industry terminology and reduce errors in key entity extraction.

[0045] In practical applications, the dynamic generation of regular expression rules is specifically achieved by collecting historical inspection command data and labeling key entity types; mining frequently occurring entity combinations based on the Apriori algorithm to generate an initial regular expression template; and optimizing the parameters of the regular expression template using a genetic algorithm to obtain the final regular expression rule. The objective function of this optimization is to maximize the F1 score.

[0046] The semantic parsing of the key entities using the optimized BERT model is specifically achieved by adding a fully connected layer to the output layer of the BERT model to map the semantic vectors to a preset intent category space; using Class Activation Graph (CAM) to visualize key semantic features and generating heatmaps to assist manual verification; and combining a federated learning framework to achieve BERT model parameter sharing among multiple inspection systems, thereby improving the parsing accuracy in small sample scenarios.

[0047] The generation of structured control instructions is specifically achieved by designing an instruction template library, which includes equipment type, parameter thresholds, and priority fields. The templates are defined in XML format. A dynamic priority adjustment algorithm is used to reorder the instruction execution order according to equipment load and task urgency. The instructions are signed using the SHA-256 hash algorithm to ensure that they are not tampered with during transmission, and the signature value is transmitted synchronously to the inspection equipment along with the instructions.

[0048] The dynamic priority adjustment algorithm specifically involves constructing a priority evaluation function with input variables including remaining battery power, task deadline, and defect severity level; using a fuzzy logic reasoning system to determine priority weights, where the membership function is generated through training on historical data; and using a particle swarm optimization (PSO) algorithm to solve for the optimal execution order, with the optimization objective being to minimize the total task completion time.

[0049] This application's embodiments adapt to changes in instruction patterns through dynamic rules, while optimizing model parameters to enhance domain semantic understanding. This solves the problems of low efficiency and poor accuracy in traditional methods. Furthermore, it can efficiently extract key information such as target devices, parameters, and priorities from multimodal control instructions, generating structured instructions containing complete parameter ranges and codes. For example, for complex instructions mixing speech and text, it can accurately identify device IDs and associate them with corresponding detection parameters, avoiding delays and errors caused by manual intervention. The generation of structured instructions further provides reliable input for task scheduling, supporting priority-encoded dynamic path planning and resource allocation.

[0050] For step 102, controlling at least one of the multiple inspection devices to inspect the target power equipment based on the control command includes: Based on the coordinate information of the target power equipment, an inspection path for at least one inspection device is planned according to preset path planning constraints, and the path length of the inspection path is optimized using the A* algorithm to obtain the target inspection path. The movement strategy of the target inspection path is dynamically adjusted based on the task priority. The movement strategy includes directional movement, speed adjustment and obstacle avoidance. The shortest path priority strategy is adopted for tasks with high priority. By combining a PID control algorithm with a V2X communication protocol, the path trajectory of the inspection equipment is adjusted in real time to ensure that the inspection equipment accurately reaches the location of the target power equipment.

[0051] It should be noted that the A* algorithm is a heuristic search algorithm, specifically implemented using a grid map model combined with Manhattan distance as a heuristic function. It is used to quickly calculate the shortest path and optimize the movement distance of inspection equipment in complex environments. Dynamic task priority adjustment refers to real-time correction of the inspection path based on a preset urgency level. This can be implemented using a time window-based scheduling algorithm to ensure that high-priority tasks are allocated path resources first. The PID control algorithm refers to a proportional-integral-derivative controller, specifically implemented using a closed-loop feedback mechanism combined with position error calculation, used to precisely adjust the movement trajectory of the inspection equipment. The V2X communication protocol refers to vehicle-to-everything (V2X) communication technology, specifically implemented using 5G networks and edge computing nodes to transmit location data in real time, used to enhance the collaborative obstacle avoidance capabilities of inspection equipment and environmental facilities.

[0052] For example, in the path planning phase, an initial inspection route is first generated based on the coordinate information of the target equipment. The A* algorithm is used to iteratively optimize the path nodes, eliminating redundant turning points to shorten the travel distance. When multiple inspection tasks are concurrent, the system dynamically adjusts the movement strategy according to preset priority parameters. For example, high-priority tasks are assigned straight paths and their movement speed is increased, while obstacles are avoided using obstacle avoidance algorithms. During equipment movement, the PID controller continuously monitors the deviation between the actual position and the planned path, correcting trajectory deviations by adjusting motor speed or steering angle. The V2X communication protocol shares real-time location data with surrounding equipment, triggering emergency braking or path replanning when a potential collision risk is detected. This solves the problem of low path planning efficiency in power equipment inspection, and also optimizes the path length through optimization algorithms, shortening equipment movement time; improves the collision risk caused by insufficient equipment movement accuracy, achieving accurate positioning in complex environments; and optimizes the task priority processing mechanism to ensure timely handling of critical defects. This solution significantly reduces the risk of high-risk environment operations while ensuring inspection efficiency, forming a complete closed-loop control system.

[0053] In another embodiment, before controlling at least one of the plurality of inspection devices to inspect the target power equipment based on the control command, the method further includes: Based on the task priorities and the actual load of each inspection device, a greedy algorithm is used to allocate inspection tasks to determine the target inspection devices for control inspection. Among them, tasks with higher priorities are assigned to inspection devices with lower loads.

[0054] In step 102, after the inspection equipment is moved to the location of the target power equipment, the inspection equipment is controlled to collect image data and video data. Specifically, the process of acquiring multi-angle images, monitoring environmental parameters, and detecting local acoustic waves on the target power equipment to obtain image data and video data includes: Establish communication connections between the inspection equipment and multispectral cameras, infrared thermal imagers, and high-definition video cameras deployed at different angles on the target power equipment; After establishing a communication connection, the inspection equipment emits detection sound waves of a preset frequency and simultaneously reads visible light images, infrared thermal images, and video data recorded by the multispectral camera, infrared thermal imager, and high-definition video camera. Spatiotemporal alignment of visible light images, infrared thermograms, and video data is performed using a time synchronization device (TSD). The visible light image and the infrared thermal image are stitched together using an image stitching algorithm to generate image data.

[0055] Multispectral cameras are devices capable of capturing images across multiple spectral bands beyond the visible light spectrum. They can be implemented using sensors with integrated filter arrays to acquire surface state information of target equipment under different spectra. Infrared thermal imagers generate thermal distribution maps by detecting the infrared energy radiated by objects. They can be implemented using uncooled microbolometer arrays to identify areas of localized overheating or abnormal temperatures within equipment. High-definition video cameras support high-resolution dynamic image recording. They can be implemented using CMOS sensors combined with optical image stabilization modules to capture continuous footage of equipment operation. Preset-frequency detection acoustic waves are acoustic signals used to trigger synchronous acquisition by multiple devices. They can be implemented using piezoelectric ceramic transducers to generate ultrasonic waves of a specific frequency, ensuring the synchronization of acquisition actions by different devices through an acoustic triggering mechanism. Time synchronization devices are modules used to align timestamps from multiple data sources. They can be implemented using GPS timing modules or network time protocols, eliminating time discrepancies between devices through a unified clock source. Image stitching algorithms refer to computational methods that fuse multiple images into a single view. Specifically, they can be implemented using affine transformation algorithms based on feature point matching, which eliminate viewpoint differences and retain complementary information through coordinate mapping.

[0056] Specifically, after the inspection equipment establishes communication connections with the multispectral camera, infrared thermal imager, and high-definition camera, the inspection equipment emits a preset frequency acoustic signal to trigger each sensor to synchronously start data acquisition. The multispectral camera acquires visible light images to identify surface cracks or dirt, the infrared thermal imager records a thermal distribution map to detect areas of abnormal temperature rise, and the high-definition camera captures dynamic video to observe the equipment's operating status. A time synchronization device aligns the data from each sensor using a unified timestamp, eliminating timing misalignments caused by equipment response delays or differences in transmission rates. Subsequently, an image stitching algorithm spatially registers and fuses the visible light image and the infrared thermal image. For example, after establishing a correspondence by extracting SIFT feature points, a weighted average method is used to generate a fused image, allowing the same location to simultaneously present optical and thermodynamic features, providing a multidimensional data foundation for subsequent defect detection.

[0057] This application's embodiments achieve high-precision synchronous acquisition of multi-angle images and environmental parameters, solving the fusion difficulties caused by data asynchrony in traditional methods. The use of acoustic triggering and time synchronization devices ensures the spatiotemporal consistency of multi-source data, reducing the computational complexity of data alignment in subsequent processing. The image stitching algorithm fuses visible light and infrared images into a unified view, enabling simultaneous detection of surface defects and temperature anomaly areas, thus improving the comprehensiveness and accuracy of power equipment defect identification.

[0058] In another embodiment, the step of performing multi-angle image acquisition, environmental parameter monitoring, and local acoustic wave detection on the target power equipment to obtain image data and video data further includes: Vibration signals from the equipment are collected using piezoelectric sensors and converted into time-domain acoustic data, with a sampling frequency of not less than 44.1 kHz. Short-time Fourier transform (STFT) is used to generate a spectrogram, extract frequency domain features, and classify sound wave patterns using support vector machine (SVM). By combining Class Activation Map (CAM) to locate areas of mechanical loosening or abnormal discharge, an acoustic detection report is generated and displayed in conjunction with the image annotation results.

[0059] For step 103, the step of intelligently locating and labeling key components of the defect region in the image data and the video data using a deep reinforcement learning (DRL) model to generate labeled image data and labeled video data with bounding boxes includes: Initialize the Q-network parameters of the deep reinforcement learning (DRL) model, define the action space as a bounding box, and determine the coordinates and size scaling of the bounding box based on the inspection parameters to obtain the target deep reinforcement learning (DRL) model. The image data and video data are sequentially input into the target deep reinforcement learning (DRL) model to identify defective regions of key components and add labeled boxes. The intersection-union ratio (IOU) between the bounding box and the defect region is evaluated based on the reward function, and the position of the bounding box is iteratively optimized by combining the IOU with the Q-learning algorithm until the IOU meets a preset threshold. The annotation box is superimposed on the image data or the video data as a semi-transparent rectangle to generate annotated image data and annotated video data with annotation boxes. The color of the frame line of the annotation box is dynamically adjusted according to the defect type during superposition.

[0060] Specifically, a Q-network structure is used to implement the model's basic architecture. The model's basic architecture is built through Q-network parameter initialization, and the action space is defined using rectangle coordinates and scaling parameters to configure the operation range of the annotation box.

[0061] The intersection-union ratio (IUGR) is an evaluation metric for the degree of overlap between the bounding box and the actual defect area. Specifically, it can be calculated as the ratio of the intersection area to the union area of ​​the two regions, used to quantify the accuracy of the bounding box positioning. The Q-learning algorithm is a reinforcement learning algorithm based on value functions. Specifically, it can use temporal difference updates to optimize the Q-value table, iteratively adjusting the bounding box position to maximize the reward function.

[0062] During initialization, the Q-network parameters are set to random values, the action space is defined as the coordinate range and scaling factor of the bounding boxes, and inspection parameters such as equipment type and environmental conditions are input to determine the initial bounding box size. Image and video data are input into a trained deep reinforcement learning model. The model analyzes data features to identify key component regions that may have defects and generates initial bounding boxes. The reward function is based on the Cross-Union Ratio (CUI). When the CUI is lower than a preset threshold, the Q-learning algorithm updates the Q-network parameters according to the current state and reward value, adjusting the position and size of the bounding boxes, iterating until the CUI reaches the required level. The optimized bounding boxes are overlaid on the original data in a semi-transparent form, and the box line color is selected from a predefined color table according to the defect type; for example, crack defects are displayed in red, and dirt defects are displayed in yellow.

[0063] In another implementation, intelligent localization using the DRL model specifically includes: extracting image / video features based on a Transformer encoder to generate multi-scale feature maps, wherein the encoder uses sequence embedding transformation to enhance global feature representation; generating candidate defect regions through a Region Proposal Network (RPN), with candidate box parameters initially filtered by non-maximum suppression (NMS); the DRL model uses candidate boxes as the initial state, defines the action space as coordinate offset and size scaling ratio, iteratively optimizes the position of the labeled boxes through a Q-learning algorithm, and the reward function combines the IOU increment and defect category confidence.

[0064] The iterative optimization of the bounding box position using the Q-learning algorithm includes: The state space is constructed using the current bounding box parameters, and the action space consists of the horizontal / vertical offset and the width and height adjustment amounts. The reward function is designed as follows: a positive reward is given when the IOU increases, and a negative penalty is given when it decreases; the reward for the termination action is linked to the product of the defect category probability. An experience replay mechanism is used to store historical adjustment trajectories, and a priority sampling strategy is used to focus on training high-reward actions to improve the model's ability to locate complex backgrounds.

[0065] Furthermore, the generation of labeled image data and labeled video data specifically includes: The optimized annotation box parameters are converted to generate an XML format annotation file, which includes coordinates, dimensions, and confidence scores. Multiple instance learning (MIL) is used to validate the annotation results. The bounding boxes are divided into positive and negative example boxes, and the annotation accuracy is optimized by using the cross-entropy loss function. The annotation boxes are overlaid on the original image / video as semi-transparent rectangles, and the frame color is dynamically adjusted according to the defect type to generate annotated visual data. The video stream is then compressed using H.265 encoding.

[0066] For step 104, the preset neural network model is invoked to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and the extracted multimodal features are fused to obtain fused features, including: The Transformer and YOLOX fusion model is invoked to extract the color features, texture features, and shape features of the labeled image data, as well as the temporal motion features of the labeled video data, and the region of interest for each feature is visualized through the Class Activation Graph (CAM). Weights of each feature are assigned through an attention mechanism to generate weighted fusion features, wherein the weights of each feature extracted from the image data are determined by the heatmap intensity of the class activation map. The spatial correlation between features is modeled using a graph convolutional network (GCN), and the feature sharing between the image data and the video data is added by combining a federated learning framework to generate the final fused features.

[0067] It should be noted that the neural network model that integrates Transformer and YOLOX is a joint model that shares parameters between the Transformer structure based on the self-attention mechanism and the single-stage object detection framework YOLOX. Specifically, it can be implemented by sharing weights in the encoder layer, which is used to simultaneously extract global semantic information of the image and temporal detection features of the video.

[0068] Class activation maps are a visualization method that generates feature heatmaps through backpropagation gradients. Specifically, they can be implemented using gradient-weighted class activation mapping algorithms to verify the effectiveness of feature extraction and provide a basis for weight allocation.

[0069] Specifically, in the image data feature extraction stage, the Transformer module performs global context modeling on the bounding box region, capturing the correlation between color and texture, while the YOLOX module locates the shape contour through a cross-stage feature pyramid. After extracting the temporal motion trajectory from the video data using YOLOX, temporal motion features are generated through temporal pooling. A class activation map visualizes and compares the thermal distribution of the two types of features, selecting high-response regions as effective features. An attention mechanism assigns a weight range of 0.6 to 0.8 to static image features and a weight range of 0.2 to 0.4 to dynamic video features based on heatmap intensity, forming a weighted fusion feature. A graph convolutional network maps multimodal features to graph-structured data, models the spatial relationships between feature nodes through an adjacency matrix, and a federated learning framework establishes a feature sharing matrix in the cloud, enabling the correlation analysis of image and video features uploaded by different inspection devices under encrypted conditions, ultimately generating a fused feature vector containing surface defects and changes in operating status of the equipment.

[0070] This approach achieves joint extraction of cross-modal features through the collaborative work of Transformer and YOLOX, improving the ability to capture complex defects compared to a single model. The dynamic weight allocation mechanism overcomes the adaptability limitations of traditional fixed weights when dealing with different levels of equipment aging. The combination of graph convolutional networks and federated learning is the first to achieve spatial correlation modeling of cross-equipment features in the field of power equipment inspection, solving the problem of missing feature correlations caused by data isolation in traditional methods. This effectively improves the accuracy of identifying defects in key components of power equipment, accurately distinguishing between surface cracks and complex defects caused by operational anomalies. It adapts to the detection needs under different lighting conditions and equipment operating states, enabling collaborative analysis of multi-source heterogeneous data in the complex electromagnetic environment of substations, reducing the false detection rate caused by insufficient feature fusion, and providing highly discriminative feature representations for subsequent defect classification.

[0071] For step 105, generating recognition and annotation results based on the fused features includes: A classification decision tree is constructed based on the fused features, and the cross-entropy loss function is used to optimize each node in the classification decision tree; The CART algorithm is used to recursively divide the feature space in each annotation box based on the optimized classification decision tree, generate a probability distribution map of defect type and severity, and associate the probability distribution map with the annotation box to obtain the final identification and annotation results of the inspection data.

[0072] It should be noted that a classification decision tree is a tree-like classification model that uses feature attributes to divide the data into levels. Specifically, information gain or Gini coefficient can be used as the splitting criterion. Its function is to achieve multi-level classification of defect types by judging feature differences layer by layer.

[0073] CART stands for Classification and Regression Tree Algorithm. Specifically, it can be implemented by recursively dividing the feature space using a binary tree structure. Its purpose is to reduce the loss of details caused by global feature analysis through the fine division of local regions.

[0074] A probability distribution map is a visual map that reflects the probability of different defect types and their severity appearing within the labeled box. Specifically, it can be implemented by calculating the conditional probability using Bayes' theorem and mapping it to the coordinate system of the labeled box. Its function is to establish a direct correlation between the quantitative evaluation results and the spatial location.

[0075] Specifically, firstly, a classification decision tree model is constructed based on fused features. The splitting threshold of each node in the decision tree is iteratively optimized using the cross-entropy loss function, allowing the classification boundary to adaptively adjust to different defect features. Then, the CART algorithm is used to perform recursive feature space partitioning within the local area covered by the bounding boxes. By progressively discriminating the differences in texture, color, and temporal motion features, a distribution map containing probability values ​​of defect type and severity is generated. Finally, the probability values ​​of each region in the probability distribution map are mapped to the coordinate positions of the corresponding bounding boxes, forming an integrated output of defect localization and classification results. By combining recursive local feature partitioning with a cross-entropy optimized decision tree structure, the classification boundary can be dynamically adjusted while preserving detailed features, thereby improving the spatial resolution and classification accuracy of defect recognition.

[0076] In this embodiment of the application, before constructing a classification decision tree based on the fused features and optimizing each node in the classification decision tree using the cross-entropy loss function, the method further includes: Stochastic gradient descent (SGD) combined with a step size decay strategy is used to perform power equipment aging decay processing on the fused features to obtain new fused features for subsequent generation of identification and labeling results.

[0077] Stochastic gradient descent is used to correct the fused features and suppress the overexpression of aging. At the same time, exponential decay or cosine annealing algorithm in the step decay strategy is used to dynamically adjust the correction magnitude.

[0078] Specifically, the fused features generated during the feature fusion stage include both key defect features of the current detection target and aging features accumulated over long-term operation of the equipment. When updating the parameters of the fused features using the stochastic gradient descent algorithm, the gradient direction corresponding to the aging features is dynamically adjusted, thereby gradually reducing their weight in the iteration process. A step size decay strategy is simultaneously applied to the optimization process, gradually reducing the parameter update step size as the training rounds increase, making the correction process of aging features tend to stabilize. This joint optimization mechanism treats the equipment aging features as noise components for attenuation, making the final output of the new fused features more focused on the real defect information within the current detection cycle, thus avoiding interference from historical aging data in the construction of the classification decision tree.

[0079] In some specific implementations, the initial learning rate of the step-decay strategy can be set to 0.01, and the decay coefficient can be configured to decrease to 0.5 times the original value every 10 training epochs. The batch size of the stochastic gradient descent can be set to 32, and the momentum parameter can be set to 0.9. For the aging and degradation processing of power equipment, a feature space dimensionality reduction method can be used.

[0080] In this embodiment, by introducing a step size decay strategy and a joint optimization mechanism of stochastic gradient descent, a systematic decay processing of aging features is achieved, solving the problem of model misjudgment caused by aging data interference.

[0081] For example, in insulator crack detection, this solution can accurately identify new cracks caused by external impacts, while eliminating false positives due to surface oxide spots caused by material aging. In transformer oil temperature anomaly detection, it can eliminate the impact of baseline temperature drift caused by long-term operation on the anomaly detection threshold, improving the timeliness and accuracy of defect identification.

[0082] In this embodiment, to improve the accuracy of subsequent identification and labeling, the method further includes: collecting historical inspection data and manually corrected records to construct an incremental training dataset; and periodically training a deep reinforcement learning (DRL) model and a neural network model based on the training dataset to fine-tune the model parameters.

[0083] Specifically, historical inspection data and manually corrected records are integrated into an incremental training dataset. The manually corrected records correct erroneous samples in the automatic annotation and supplement data on abnormal states caused by equipment aging or environmental changes. The incremental training dataset is spatiotemporally aligned with timestamps and equipment status labels to ensure the continuity and consistency of data distribution. During periodic training, the deep reinforcement learning model and neural network model employ an incremental learning strategy to extract features and update weights on newly added data, while freezing some underlying network parameters to prevent catastrophic forgetting. During training, the model calculates the gradient of the loss function using the backpropagation algorithm, fine-tuning only the parameters of the top-level classifier and feature fusion module, achieving continuous optimization of model performance with limited computing resources. Thus, the model can adapt to the characteristic changes of progressive defects such as surface contamination accumulation and loosening of mechanical structures in power equipment insulators, while effectively identifying equipment anomalies caused by sudden environmental factors.

[0084] By employing an incremental training mechanism, local parameter optimization is performed only on newly added data and corrected samples, achieving dynamic model updates while reducing training resource consumption. Existing technologies rely on manual periodic adjustments of model thresholds and feature weights, lacking an automated feedback mechanism. This solution, however, integrates manual correction records with automated data collection to form a closed-loop optimization process, improving the model's environmental adaptability.

[0085] In this embodiment, after identification and labeling are completed, the following steps are also included: Set a threshold for the severity of defects. When the annotation result exceeds the threshold, a multi-level warning is triggered. The warning level is determined by the difference between the IOU and the threshold. The system notifies maintenance personnel through multiple channels, including SMS, email, and app push notifications, while simultaneously generating maintenance work orders and automatically planning the optimal maintenance path. Dynamic Bayesian Networks (DBNs) are used to predict defect development trends, provide early warnings of potential faults, and shorten response times.

[0086] The implementation scheme provided in this application realizes intelligent annotation of power grid insulator images through the DRL model, which significantly reduces the annotation error rate and reduces labor costs; it enhances the multi-scale feature extraction capability by integrating the Transformer and YOLOX models, and optimizes the detection box selection by combining the improved α-IOU NMS algorithm, which effectively improves the recognition accuracy of small sample defects (such as micro cracks) and the detection completeness in multi-target scenarios; at the same time, it realizes real-time processing and decision feedback of inspection data through multi-modal data fusion and edge computing deployment, which improves the overall efficiency of power inspection and reduces the missed detection rate.

[0087] The above describes the power equipment inspection method based on a neural network model in the embodiments of this application. The following describes the power equipment inspection device based on a neural network model in the embodiments of this application. This power equipment inspection device is applied to a power inspection system, characterized in that the power inspection system includes an instruction control device, multiple inspection devices, and a cloud server that are interconnected. Please refer to [link to relevant documentation]. Figure 2 The power equipment inspection device based on a neural network model provided in this application embodiment includes: The instruction generation module 210 is used to collect control information input by the user, and perform semantic analysis on the control information through the instruction control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generate control commands that can be executed by the multiple inspection devices; the control information includes at least one of voice instructions, text instructions and image interaction instructions. The acquisition module 220 is used to control at least one of the multiple inspection devices to inspect the target power equipment based on the control command, and to perform multi-angle image acquisition, environmental parameter monitoring and local sound wave detection on the target power equipment to obtain image data and video data. The annotation module 230 is used to intelligently locate and annotate key components of defect areas in the image data and video data respectively through a deep reinforcement learning (DRL) model, generate annotated image data and annotated video data with annotation boxes, and upload them to the server. The extraction module 240 is used to call a preset neural network model to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and to fuse the extracted multimodal features to obtain fused features. The neural network model is a neural network model fused by Transformer and YOLOX. The display module 240 is used to generate recognition and annotation results based on the fusion features and add them to the position of the annotation box for display.

[0088] In an exemplary embodiment, the acquisition module 220 includes: a planning control unit 221, configured to: Based on the coordinate information of the target power equipment, an inspection path for at least one inspection device is planned according to preset path planning constraints, and the path length of the inspection path is optimized using the A* algorithm to obtain the target inspection path. The movement strategy of the target inspection path is dynamically adjusted based on the task priority. The movement strategy includes directional movement, speed adjustment and obstacle avoidance. The shortest path priority strategy is adopted for tasks with high priority. By combining a PID control algorithm with a V2X communication protocol, the path trajectory of the inspection equipment is adjusted in real time to ensure that the inspection equipment accurately reaches the location of the target power equipment.

[0089] In an exemplary embodiment, the acquisition module 220 includes: an acquisition unit 222, configured to: Establish communication connections between the inspection equipment and multispectral cameras, infrared thermal imagers, and high-definition video cameras deployed at different angles on the target power equipment; After establishing a communication connection, the inspection equipment emits detection sound waves of a preset frequency and simultaneously reads visible light images, infrared thermal images, and video data recorded by the multispectral camera, infrared thermal imager, and high-definition video camera. Spatiotemporal alignment of visible light images, infrared thermograms, and video data is performed using a time synchronization device (TSD). The visible light image and the infrared thermal image are stitched together using an image stitching algorithm to generate image data.

[0090] In one exemplary embodiment, the annotation module 230 includes: Model building unit 231 is used to initialize the Q network parameters of the deep reinforcement learning (DRL) model, define the action space as a bounding box, and determine the coordinates and size scaling of the bounding box based on the inspection parameters to obtain the target deep reinforcement learning (DRL) model. The recognition unit 232 is used to sequentially input the image data and the video data into the target deep reinforcement learning (DRL) model to identify the defective areas of key components and add annotation boxes; Optimization unit 233 is used to evaluate the intersection-union ratio (IOU) between the bounding box and the defect region based on the reward function, and iteratively optimize the position of the bounding box by combining the IOU with the Q-learning algorithm until the IOU meets a preset threshold. The annotation unit 234 is used to overlay the annotation box as a semi-transparent rectangle onto the image data or the video data to generate annotated image data and annotated video data with annotation boxes. The color of the frame line of the annotation box is dynamically adjusted according to the defect type during the overlay.

[0091] In one exemplary embodiment, the extraction module 240 includes: Extraction unit 241 is used to call the Transformer and YOLOX fusion model to extract the color features, texture features and shape features of the labeled image data, as well as the temporal motion features of the labeled video data, and visualize the region of interest of each feature through class activation graph (CAM); The fusion unit 242 is used to allocate the weights of each feature through an attention mechanism to generate weighted fusion features, wherein the weights of each feature extracted from the image data are determined by the heatmap intensity of the class activation map; a graph convolutional network (GCN) is used to model the spatial correlation between features, and a federated learning framework is used to add feature sharing between the image data and the video data to generate the final fusion features.

[0092] In one exemplary embodiment, the display module 250 is specifically used for: A classification decision tree is constructed based on the fused features, and the cross-entropy loss function is used to optimize each node in the classification decision tree; The CART algorithm is used to recursively divide the feature space in each annotation box based on the optimized classification decision tree, generate a probability distribution map of defect type and severity, and associate the probability distribution map with the annotation box to obtain the final identification and annotation results of the inspection data.

[0093] In one exemplary embodiment, the device further includes an optimization module 260, specifically used for: Collect historical inspection data and manually corrected records to build an incremental training dataset; Deep reinforcement learning (DRL) models and neural network models are trained periodically based on the training dataset to fine-tune the model parameters.

[0094] In this embodiment, control commands are obtained by parsing the control information of the inspection equipment. Then, the inspection equipment is controlled to collect image and video data of the target power equipment. The image and video data are labeled with annotation information that matches the inspection parameters in the control information by combining a deep learning model and a neural network model that integrates Transformer and YOLOX. This method significantly improves the annotation efficiency and the accuracy of inspection identification, improves the overall power inspection efficiency, reduces the missed detection rate, and provides intelligent technical support for the safe operation of the power grid.

[0095] above Figure 2 The power equipment inspection device based on the neural network model in this application embodiment is described in detail from the perspective of modular functional entities. The electronic equipment in this application embodiment is described in detail from the perspective of hardware processing.

[0096] See Figure 3 As shown, the electronic device includes a processor 300 and a memory 301. The memory 301 stores machine-executable instructions that can be executed by the processor 300. The processor 300 executes the machine-executable instructions to implement the above-mentioned power equipment inspection method based on a neural network model.

[0097] Furthermore, Figure 3 The electronic device shown also includes a bus 302 and a communication interface 303. The processor 300, the communication interface 303 and the memory 301 are connected via the bus 302.

[0098] The memory 301 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 302 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0099] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the method steps of the aforementioned embodiment.

[0100] This application also provides an electronic device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the power equipment inspection method based on the neural network model in the above embodiments.

[0101] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the power equipment inspection method based on the neural network model.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A power equipment inspection method based on a neural network model, applied to a power inspection system, characterized in that, The power inspection system includes interconnected command and control devices, multiple inspection devices, and a cloud server; the method includes: The system collects control information input by the user and performs semantic analysis on the control information through the command control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generates control commands that can be executed by the multiple inspection devices; the control information includes at least one of voice commands, text commands and image interaction commands. Based on the control command, at least one of the multiple inspection devices is controlled to inspect the target power equipment, and multi-angle image acquisition, environmental parameter monitoring and local sound wave detection are performed on the target power equipment to obtain image data and video data; The key components of the defect areas in the image data and video data are intelligently located and labeled using a deep reinforcement learning (DRL) model, generating labeled image data and labeled video data with bounding boxes, and then uploaded to the server. A preset neural network model is invoked to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and the extracted multimodal features are fused to obtain fused features. The neural network model is a neural network model fused with Transformer and YOLOX. The recognition and annotation results are generated based on the fusion features and added to the position of the annotation box for display.

2. The power equipment inspection method based on a neural network model according to claim 1, characterized in that, The step of controlling at least one of multiple inspection devices to inspect the target power equipment based on the control command includes: Based on the coordinate information of the target power equipment, an inspection path for at least one inspection device is planned according to preset path planning constraints, and the path length of the inspection path is optimized using the A* algorithm to obtain the target inspection path. The movement strategy of the target inspection path is dynamically adjusted based on the task priority. The movement strategy includes directional movement, speed adjustment and obstacle avoidance. The shortest path priority strategy is adopted for tasks with high priority. By combining a PID control algorithm with a V2X communication protocol, the path trajectory of the inspection equipment is adjusted in real time to ensure that the inspection equipment accurately reaches the location of the target power equipment.

3. The power equipment inspection method based on a neural network model according to claim 2, characterized in that, The process of acquiring multi-angle images, monitoring environmental parameters, and detecting local acoustic waves of the target power equipment to obtain image data and video data includes: Establish communication connections between the inspection equipment and multispectral cameras, infrared thermal imagers, and high-definition video cameras deployed at different angles on the target power equipment; After establishing a communication connection, the inspection equipment emits detection sound waves of a preset frequency and simultaneously reads visible light images, infrared thermal images, and video data recorded by the multispectral camera, infrared thermal imager, and high-definition video camera. Spatiotemporal alignment of visible light images, infrared thermograms, and video data is performed using a time synchronization device (TSD). The visible light image and the infrared thermal image are stitched together using an image stitching algorithm to generate image data.

4. The power equipment inspection method based on a neural network model according to any one of claims 1-3, characterized in that, The process of intelligently locating and labeling key components of defect regions in the image data and video data using a deep reinforcement learning (DRL) model to generate labeled image data and labeled video data with bounding boxes includes: Initialize the Q-network parameters of the deep reinforcement learning (DRL) model, define the action space as a bounding box, and determine the coordinates and size scaling of the bounding box based on the inspection parameters to obtain the target deep reinforcement learning (DRL) model. The image data and video data are sequentially input into the target deep reinforcement learning (DRL) model to identify defective regions of key components and add labeled boxes. The intersection-union ratio (IOU) between the bounding box and the defect region is evaluated based on the reward function, and the position of the bounding box is iteratively optimized by combining the IOU with the Q-learning algorithm until the IOU meets a preset threshold. The annotation box is superimposed on the image data or the video data as a semi-transparent rectangle to generate annotated image data and annotated video data with annotation boxes. The color of the frame line of the annotation box is dynamically adjusted according to the defect type during superposition.

5. The power equipment inspection method based on a neural network model according to any one of claims 1-3, characterized in that, The process involves calling a preset neural network model to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and then fusing the extracted multimodal features to obtain fused features, including: The Transformer and YOLOX fusion model is invoked to extract the color features, texture features, and shape features of the labeled image data, as well as the temporal motion features of the labeled video data, and the region of interest for each feature is visualized through the Class Activation Graph (CAM). Weights of each feature are assigned through an attention mechanism to generate weighted fusion features, wherein the weights of each feature extracted from the image data are determined by the heatmap intensity of the class activation map. The spatial correlation between features is modeled using a graph convolutional network (GCN), and the feature sharing between the image data and the video data is added by combining a federated learning framework to generate the final fused features.

6. The power equipment inspection method based on a neural network model according to any one of claims 1-3, characterized in that, The generation of recognition and annotation results based on the fused features includes: A classification decision tree is constructed based on the fused features, and the cross-entropy loss function is used to optimize each node in the classification decision tree. The CART algorithm is used to recursively divide the feature space in each annotation box based on the optimized classification decision tree, generate a probability distribution map of defect type and severity, and associate the probability distribution map with the annotation box to obtain the final identification and annotation results of the inspection data.

7. The power equipment inspection method based on a neural network model according to claim 6, characterized in that, The method further includes: Collect historical inspection data and manually corrected records to build an incremental training dataset; The deep reinforcement learning (DRL) model and neural network model are trained periodically based on the training dataset to fine-tune the model parameters.

8. A power equipment inspection device based on a neural network model, applied to a power inspection system, characterized in that, The power inspection system includes an interconnected command and control device, multiple inspection devices, and a cloud server; the device includes: The instruction generation module is used to collect control information input by the user, and perform semantic analysis on the control information through the instruction control device to extract the target power equipment to be inspected, inspection parameters and task priorities, and generate control commands that can be executed by the multiple inspection devices; the control information includes at least one of voice instructions, text instructions and image interaction instructions. The acquisition module is used to control at least one of the multiple inspection devices to inspect the target power equipment based on the control command, and to perform multi-angle image acquisition, environmental parameter monitoring and local sound wave detection on the target power equipment to obtain image data and video data. The annotation module is used to intelligently locate and annotate key components of defect areas in the image data and video data using a deep reinforcement learning (DRL) model, generate annotated image data and annotated video data with annotation boxes, and upload them to the server. The extraction module is used to call a preset neural network model to extract multimodal features from the bounding box positions of the labeled image data and labeled video data in the server, and to fuse the extracted multimodal features to obtain fused features. The neural network model is a neural network model fused by Transformer and YOLOX. The display module is used to generate recognition and annotation results based on the fused features and add them to the position of the annotation box for display.

9. An electronic device, characterized in that, The electronic device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to execute the power equipment inspection method based on a neural network model as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the power equipment inspection method based on the neural network model as described in any one of claims 1-7.

Images