A power equipment autonomous identification method, device, equipment and medium

Abstract

The application discloses a kind of power equipment autonomous identification method, device, equipment and medium, it is related to power equipment intelligent identification technical field, utilize large language model and generate instruction package to the instruction that sample of user input is generated deconstruction as structured parameter, drive simulation platform response instruction package and carry out multimodal simulation, and generate three-dimensional sample point cloud in combination with back projection;Dynamic gated multimodal recognition network is constructed, and the network is trained, and initial equipment recognition network is obtained, and then the initial equipment recognition network is used to iteratively optimize the image and point cloud to be measured, obtain target equipment recognition network and output power equipment identification result, through simulation and big model, it realizes the data automatic production of almost zero cost, unlimited quantity, accurate annotation, to solve data bottleneck problem;Meanwhile, through confidence screening plus active learning constitutes closed loop, get rid of the limitation of static model, with autonomous evolution ability, realize more intelligent autonomous intelligence.

Description

Technical Field

[0001] This invention relates to the field of intelligent identification technology for power equipment, and in particular to a method, device, equipment, and medium for autonomous identification of power equipment. Background Technology

[0002] With the continuous advancement of new power system construction, unmanned aerial vehicle (UAV) inspection and mobile mapping technologies have become core tools for power grid operation and maintenance. The power industry has accumulated massive amounts of power equipment images and point cloud data during routine inspections. Intelligent identification of power equipment based on multimodal data is a key technological support for achieving automatic fault detection in the power grid, refined equipment status assessment, and comprehensive improvement of operation and maintenance efficiency. It is of great significance for ensuring the safe and stable operation of the power system and has therefore become a key research direction in the field of intelligent power operation and maintenance.

[0003] Currently, deep learning technology, with its powerful feature extraction capabilities, has been widely applied to power equipment identification tasks. Multimodal data fusion is also considered a core path to improve the recognition performance of complex scenes. Related research and applications are gradually developing from single image or point cloud recognition to multi-source data collaborative analysis, so as to make full use of the information advantages of different modal data.

[0004] However, current mainstream power equipment identification solutions still heavily rely on manually labeled static samples for model training. This results in high labeling costs and difficulty in obtaining rare category samples such as faulty equipment, which can easily lead to imbalanced training data. At the same time, traditional multimodal data fusion methods mostly use fixed weights, which cannot be dynamically adjusted according to the reliability of different modal data in the scenario. In complex inspection scenarios such as occlusion and light and shadow interference, the generalization ability is insufficient and the fine recognition accuracy is difficult to meet the actual operation and maintenance needs, thus restricting the large-scale application of multimodal recognition technology in power inspection scenarios. Summary of the Invention

[0005] This invention provides a method, device, equipment, and medium for autonomous identification of power equipment, which solves the technical problems of high cost of manual annotation, difficulty in obtaining rare samples, and weak model generalization ability and insufficient precision of fine identification caused by insufficient multimodal data fusion in the prior art.

[0006] The first aspect of this invention provides a method for autonomous identification of power equipment, comprising:

[0007] The large language model is used to deconstruct user input samples into structured parameters and generate instruction packages;

[0008] The simulation platform responds to the instruction packet and performs multimodal simulation, and uses the multimodal simulation results for back projection to generate a three-dimensional sample point cloud.

[0009] A dynamic gated multimodal recognition network is constructed, and the dynamic gated multimodal recognition network is trained using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial device recognition network;

[0010] The initial device recognition network is used to iteratively optimize the image under test and the point cloud to obtain the target device recognition network, and the target device recognition network is used for recognition to obtain the recognition result.

[0011] Optionally, the structured parameters include device category, attribute modification, and associated attributes obtained by context association based on the attribute modification, wherein the attribute modification is used to characterize the attribute change information of the device; the texture corresponding to the attribute modification includes material texture and / or fault texture, and the instruction package includes modification instructions corresponding to the attribute modification and size parameter instructions corresponding to the attribute modification.

[0012] Optionally, the process of obtaining the associated attributes specifically includes:

[0013] Obtain a pre-constructed knowledge graph of power equipment, wherein the nodes of the knowledge graph include power equipment, equipment components, equipment status, and environmental conditions, and the edges represent the relationships between the nodes;

[0014] Based on the attribute modification, a search is performed in the power equipment knowledge graph to determine that the node corresponding to the attribute modification has an edge connecting to the first node;

[0015] Based on the first node, physical attribute reasoning is performed to obtain the reasoned attributes;

[0016] Based on the inference attribute, a search is performed in the power equipment knowledge graph to determine that the node corresponding to the inference attribute has an edge connecting to the second node;

[0017] The associated attributes are determined based on the first node and the second node.

[0018] Optionally, the step of responding to the instruction packet through a simulation platform and performing multimodal simulation, and then using the multimodal simulation results for back projection to generate a three-dimensional sample point cloud, includes:

[0019] The simulation platform responds to the instruction packet by extracting the target 3D model from the device resource library and at least one texture corresponding to the attribute modification and / or the associated attribute, and randomly selecting at least one occlusion model from the occlusion resource library.

[0020] Rendering is performed from multiple preset perspectives based on the target 3D model, at least one texture, and at least one occlusion model to obtain multimodal simulation results;

[0021] The multimodal simulation results include RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps;

[0022] Using the depth sample map and camera intrinsic parameters, a 3D sample point cloud that is registered with the depth sample map at the pixel level is generated in real time through back projection.

[0023] Optionally, the step of generating a 3D sample point cloud registered at the pixel level with the depth sample map in real time through back projection using the depth sample map and camera intrinsic parameters includes:

[0024] Using camera intrinsic parameters, the coordinates of each pixel in the depth sample map are transformed to the camera standard coordinate system;

[0025] Each pixel in the camera's standard coordinate system is back-projected using its corresponding depth value to obtain a point cloud in three-dimensional space.

[0026] The point cloud in the three-dimensional space is transformed to the world coordinate system to obtain a three-dimensional sample point cloud.

[0027] Optionally, the step of constructing a dynamic gated multimodal recognition network and training the dynamic gated multimodal recognition network using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial device recognition network includes:

[0028] A dynamic gated multimodal recognition network is constructed, which includes an image encoder, a sparse voxel encoder, multiple dynamic fusion gate modules, and a single decoder.

[0029] The different level outputs of the image encoder are respectively connected to the first input of each of the dynamic fusion gating modules; the different level outputs of the sparse voxel encoder are respectively connected to the second input of each of the dynamic fusion gating modules; the outputs of each of the dynamic fusion gating modules are connected to the input of the single decoder.

[0030] Acquire training data consisting of the RGB sample image, the depth sample map, the 3D sample point cloud, the instance segmentation map, and the semantic segmentation map;

[0031] The RGB sample images are preprocessed by standardization and size scaling, and then input into the image encoder to output multi-scale image feature maps;

[0032] The three-dimensional sample point cloud is preprocessed with dynamic sparse voxel representation and input into the sparse voxel encoder to output a multi-scale point cloud feature map that corresponds one-to-one with the scale of the multi-scale image feature map.

[0033] Through the dynamic fusion gating modules, feature fusion is performed on image feature maps and point cloud feature maps of the same scale to obtain multi-scale fused features;

[0034] The multi-scale fused features are input into the single decoder, and semantic labels and instance embedding results are output.

[0035] The initial device recognition network is obtained by supervised training based on the instance segmentation map, the semantic segmentation map, the semantic labels, and the instance embedding results.

[0036] Optionally, the step of iteratively optimizing the initial device recognition network on the image to be tested and the point cloud to obtain a target device recognition network, and then using the target device recognition network for recognition to obtain a recognition result, includes:

[0037] The initial device recognition network is used to process the image and point cloud under test to obtain the device segmentation result and the corresponding confidence level.

[0038] Forward the test image and point cloud with a confidence level lower than the preset threshold to the annotation terminal;

[0039] The initial device identification network is trained and its parameters are updated based on the annotation results from the annotation end.

[0040] Jump to the step of processing the image and point cloud under test using the initial device recognition network to obtain the device segmentation result and the corresponding confidence level, until the recognition accuracy of the iteratively updated initial device recognition network reaches the preset accuracy requirement, and the target device recognition network is obtained.

[0041] The target device recognition network is used to process the image and point cloud to output the recognition result of the power equipment.

[0042] A second aspect of the present invention provides an autonomous identification device for power equipment, comprising:

[0043] The deconstruction module is used to deconstruct user-input samples into structured parameters using a large language model and generate instruction packages.

[0044] The processing module is used to respond to the instruction packet through the simulation platform and perform multimodal simulation, and use the multimodal simulation results to perform back projection to generate a three-dimensional sample point cloud;

[0045] An initial training module is used to construct a dynamic gated multimodal recognition network and train the dynamic gated multimodal recognition network using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial device recognition network.

[0046] The deep training module is used to iteratively optimize the initial device recognition network for the test image and point cloud to obtain the target device recognition network, and then use the target device recognition network to perform recognition to obtain the recognition result.

[0047] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the power equipment autonomous identification method described above.

[0048] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the power equipment autonomous identification method as described above.

[0049] As can be seen from the above technical solutions, the present invention has the following advantages:

[0050] This invention provides a method, apparatus, device, and medium for autonomous identification of power equipment. It utilizes a large language model to deconstruct user-input sample generation commands into structured parameters and generate command packets. This drives a simulation platform to respond to the command packets and perform multimodal simulations. A three-dimensional sample point cloud is generated using back projection. A dynamically gated multimodal recognition network is constructed. This network is trained using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial equipment identification network. This initial network is then iteratively optimized using the test image and point cloud to obtain the target equipment identification network, which outputs the power equipment identification result. Users only need to provide sample generation commands; no manual data annotation is required. The large language model deconstructs these commands into structured parameters and generates command packets, driving the simulation platform to perform multimodal simulations. A three-dimensional sample point cloud is then generated through back projection. Throughout the process, the multimodal data and corresponding 3D point clouds generated by the simulation naturally possess complete scene and equipment information, and can be directly used as training samples without additional manual annotation. At the same time, the simulation process can flexibly generate samples in various scenarios according to needs, without being limited by the difficulty of obtaining rare data in reality. This avoids the high cost of manual annotation and breaks through the bottleneck of the difficulty in obtaining rare samples, providing sufficient and suitable multimodal data for model training. Furthermore, the efficient fusion of multimodal data is achieved through a dynamic gated multimodal recognition network, overcoming the shortcomings of traditional multimodal recognition schemes, such as weak model generalization ability and insufficient fine recognition accuracy caused by insufficient data fusion. This provides an efficient and reliable technical path for the autonomous identification of power equipment. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the steps of an autonomous identification method for power equipment provided in Embodiment 1 of the present invention.

[0053] Figure 2 This is a flowchart illustrating the construction process of a three-dimensional model of a power equipment provided in Embodiment 2 of the present invention.

[0054] Figure 3 This is a schematic diagram of the device identification network provided in Embodiment 2 of the present invention;

[0055] Figure 4 This is a structural block diagram of an autonomous identification device for power equipment provided in Embodiment 3 of the present invention;

[0056] Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0057] This invention provides a method, apparatus, device, and medium for autonomous identification of power equipment, which addresses the technical problems in the prior art, such as high cost of manual annotation, difficulty in obtaining rare samples, and weak model generalization ability and insufficient precision in refined identification due to insufficient multimodal data fusion.

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Currently, power line inspections widely utilize drones and mobile mapping systems, accumulating massive amounts of point cloud and image data. However, limitations such as high annotation costs, imbalanced sample categories (e.g., scarcity of data on faulty equipment), and insufficient multimodal data fusion in complex environments hinder the generalization ability and precision of model identification. Therefore, how to abandon the traditional paradigm of relying on large amounts of static, manually annotated data and instead construct a solution capable of equipment identification from a small amount of seed data is an urgent problem to be solved.

[0060] Example 1:

[0061] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of an autonomous identification method for power equipment provided in Embodiment 1 of the present invention.

[0062] This invention provides a method for autonomous identification of power equipment, comprising:

[0063] Step 101: Use a large language model to deconstruct the user-input sample generation instructions into structured parameters and generate an instruction package.

[0064] In this embodiment of the invention, the large language model performs semantic recognition and information extraction on the sample generation instructions input by the user to obtain the key parameters required for sample generation. Then, the key parameters are organized into a structured parameter form and packaged in a format recognizable by the simulation platform to form a corresponding instruction package.

[0065] Step 102: Respond to the command packet through the simulation platform and perform multimodal simulation, and use the multimodal simulation results to perform back projection to generate a three-dimensional sample point cloud.

[0066] In this embodiment of the invention, the simulation platform receives and parses the instruction packet, performs multimodal scene simulation based on the structured parameters therein, and obtains the corresponding multimodal simulation results; then, based on the depth-related data in the multimodal simulation results, and combined with the camera parameters, back projection calculation is performed to convert the two-dimensional planar data into three-dimensional spatial data, and finally a three-dimensional sample point cloud matching the simulation results is generated.

[0067] Step 103: Construct a dynamic gated multimodal recognition network, and train the dynamic gated multimodal recognition network using 3D sample point cloud and multimodal simulation results to obtain the initial device recognition network.

[0068] In this embodiment of the invention, a dynamic gated multimodal recognition network is constructed, consisting of image encoding, point cloud encoding, multi-scale dynamic gating fusion and decoding modules. The three-dimensional sample point cloud and multimodal simulation results are used as training data to input the network, and the annotation information corresponding to the simulation results is used as supervision signals to train the network until the network converges, thus obtaining the initial device recognition network.

[0069] Step 104: Iteratively optimize the image and point cloud under test using the initial device recognition network to obtain the target device recognition network, and use the target device recognition network for recognition to obtain the recognition result.

[0070] In this embodiment of the invention, the image to be tested and the point cloud are input into the initial device recognition network for processing. The network is iteratively updated and optimized according to the processing results until the network performance meets the preset requirements, thus obtaining the target device recognition network. Then, the image to be tested and the point cloud are input into the target device recognition network to complete the power equipment recognition and output the corresponding recognition result.

[0071] Large Language Model: A model fine-tuned with knowledge from the power industry to parse natural language sample generation instructions and generate simulation instruction packages; Sample Generation Instructions: User-inputted instructions in natural language specifying simulation sample generation requirements; Structured Parameters: Standardized key parameters extracted from the instructions by the large language model, recognizable by the simulation platform; Instruction Package: A set of parameters containing all simulation execution instructions, encapsulated according to the simulation platform format; Simulation Platform: An automated simulation environment built with a high-fidelity game engine, capable of performing multimodal simulation and point cloud generation; Multimodal Simulation: The simulation process by which the simulation platform generates various types of data, including images, point clouds, and depth maps; Multimodal Simulation Results: A collection of multimodal data output from the multimodal simulation, including images, depth maps, and segmentation maps; Back Projection: The process of converting two-dimensional pixels into three-dimensional spatial point coordinates using depth sample maps and camera intrinsic parameters; 3D Sample point cloud: 3D point cloud data generated by back projection and pixel-level registered with RGB sample images; Dynamic gated multimodal recognition network: A multimodal recognition network composed of a dual-stream encoder, a dynamic fusion gating module, and a single decoder, capable of adaptively fusing image and point cloud features; Initial device recognition network: A dynamic gated multimodal recognition network with basic recognition capabilities after training with simulated samples; Image to be tested: RGB image data to be recognized collected in power inspection scenarios; Point cloud: 3D geometric point cloud data to be recognized collected in power inspection scenarios; Iterative optimization: The process of improving network recognition accuracy by forming a closed loop through low-confidence sample annotation and model retraining; Target device recognition network: A device recognition network with recognized accuracy achieved after iterative optimization and usable for actual recognition; Recognition result: Recognition information such as power equipment category, instance location, and contour boundary output by the target device recognition network.

[0072] This invention addresses the high cost of sample labeling in current power equipment identification methods by proposing an autonomous power equipment identification scheme. The technical solution of this invention achieves near-zero-cost, unlimited, and accurately labeled automatic data generation through simulation and large-scale models, fundamentally solving the data bottleneck problem. Furthermore, the technical solution of this invention forms a closed loop through confidence-based filtering and active learning, overcoming the limitations of static models and possessing autonomous evolution capabilities, achieving autonomous intelligence that "gets smarter with use."

[0073] Example 2:

[0074] This invention provides a method for autonomous identification of power equipment, comprising:

[0075] Step 201: Use a large language model to deconstruct the user-input sample generation instructions into structured parameters and generate an instruction package.

[0076] Furthermore, the structured parameters include device category, attribute modification, and associated attributes obtained by context association based on attribute modification. Attribute modification is used to characterize the attribute change information of the device. The textures corresponding to attribute modification include material textures and / or fault textures. The instruction package includes modification instructions corresponding to attribute modification and size parameter instructions corresponding to attribute modification.

[0077] It should be noted that the textures corresponding to attribute modifications include at least one of the following: material textures and fault textures.

[0078] Equipment Category: The type of power equipment specified in the structured parameters, used to match and call the corresponding 3D model; Attribute Modification: Parameters in the structured parameters that characterize the equipment's state or attribute changes, reflecting the equipment's non-original factory state; Associated Attributes: Equipment-related environmental or physical attributes obtained through contextual reasoning based on attribute modifications; Equipment Attribute Change Information: Deviation information such as equipment defects, aging, and environmental impacts characterized by attribute modifications; Material Map: Texture files simulating normal material or state changes on the equipment surface; Fault Map: Texture files simulating equipment defects and fault states; Modification Command: Commands in the command package corresponding to attribute modifications, used to control the simulation platform to apply textures or adjust parameters; Size Parameter Command: Commands in the command package corresponding to attribute modifications, used to control the size of the equipment model and parameters of defect areas.

[0079] Furthermore, the associated attributes obtained by context association based on attribute modifications include:

[0080] A11. Obtain a pre-built knowledge graph of power equipment. The nodes of the knowledge graph include power equipment, equipment components, equipment status, and environmental conditions. The edges represent the relationships between the nodes.

[0081] A12. Based on attribute modification, search the power equipment knowledge graph to determine the node corresponding to the attribute modification that has an edge connecting to the first node.

[0082] A13. Based on the first node, perform physical attribute reasoning to obtain the reasoned attributes;

[0083] A14. Based on the reasoning attributes, search the knowledge graph of power equipment to determine that the nodes corresponding to the reasoning attributes have edges connecting to the second node;

[0084] A15. Determine the associated attributes based on the first and second nodes.

[0085] Power Equipment Knowledge Graph: A pre-constructed structured graph storing prior knowledge in the power equipment domain; Node: A basic unit representing an entity or concept in the power equipment knowledge graph; Power Equipment: A node representing power system equipment in the knowledge graph; Equipment Component: A node representing the components of power equipment in the knowledge graph; Equipment Status: A node representing the operating or defective state of equipment in the knowledge graph; Environmental Conditions: A node representing the environment in which the equipment exists in the knowledge graph; Edge: A line segment connecting nodes in the knowledge graph, representing the relationships between nodes; Relationship: The causal, influence, or subordinate relationship between nodes in the knowledge graph; First Node: A node directly connected to the attribute modification node, retrieved from the knowledge graph based on attribute modification; Physical Attribute Reasoning: The process of deriving relevant attributes from the first node based on the physical laws of the power domain; Reasoning Attribute: The equipment-related physical or environmental attributes obtained through physical attribute reasoning; Second Node: A node directly connected to the reasoning attribute node, retrieved from the knowledge graph based on the reasoning attribute.

[0086] In this embodiment of the invention, semantically driven sample generation is implemented. Instructions described in natural language are used to invoke a large language model, providing complete instruction support for automatic modeling and rendering in the subsequent simulation environment. This lays the foundation for generating accurately registered multimodal data and pixel-level / point-level ground truth labels. This embodiment uses a finely tuned large language model as the "simulation commander." Taking the user-input natural language instruction "Generate a point cloud and image of an insulator surface with 5% contamination" as an example, the implementation process is explained in detail:

[0087] Upon receiving the user's sample generation instruction, the large language model first deconstructs it into a series of structured parameters, specifically including three categories: device category, attribute modification, and related attributes obtained by context association based on attribute modification. The deconstruction logic of each parameter is as follows:

[0088] Equipment Category (Object Recognition): Extract the equipment category as "insulator" from the instruction and call the 3D model of insulator in the equipment resource library. This model includes a variety of common types such as disc type and column type to cover the equipment form requirements in different scenarios.

[0089] Attribute Modification: The description "5% dirt on the surface" in the instruction is converted into a structured instruction to represent the attribute change information of the device. The texture corresponding to this attribute modification is a "dirt" material texture. Alternatively, a fault texture can be selected according to the type of device defect. In this embodiment, a dirt-type material texture is used. Specifically, it is converted into two instructions: ① Select the "dirt" material texture; ② Set the coverage area parameter of the texture to 5% of the model surface area to accurately restore the state description in the instruction.

[0090] The associated attributes are obtained by performing contextual association based on attribute modifications, specifically by following these steps:

[0091] A11. Obtain a pre-constructed knowledge graph of power equipment. Nodes in the knowledge graph include power equipment, equipment components, equipment status, and environmental conditions. Edges represent the relationships between nodes. A12. Based on attribute modification (pollution status), search the knowledge graph to identify the first node connected to the node corresponding to the attribute modification by an edge. For example, the node corresponding to the pollution status can be associated with the node "deterioration in equipment insulation performance". A13. Based on the first node (deterioration in equipment insulation performance), perform physical attribute reasoning to obtain the inferred attributes. For example, deterioration in insulation performance can infer "increased humidity" and "increased salt density," which are related to pollution. Strongly related physical attributes of the scene; A14. Based on the inferred attributes (increased humidity, increased salinity), search in the power equipment knowledge graph to determine the second node connected to the node corresponding to the inferred attribute by an edge. For example, "increased humidity" and "increased salinity" can be associated with the "humid coastal environment" node; A15. Based on the first node (decreased equipment insulation performance) and the second node (humid coastal environment), determine the associated attributes, namely humid coastal environment. At the same time, the physical attributes "increased humidity" and "increased salinity" obtained by inference are also included in the associated attribute category. The LLM has power knowledge and can automatically pass these parameters to the subsequent physics engine.

[0092] After extracting and associating all structured parameters, the large language model encapsulates various parameters into instruction packages according to the format specifications recognizable by the simulation platform. The instruction packages include modification instructions corresponding to attribute modifications (such as instructions to load dirt material textures and instructions to set the texture coverage area to 5% of the model surface area), size parameter instructions corresponding to attribute modifications (such as standard dimensions of insulators and structural parameter instructions), and physical attribute parameter instructions obtained from context association (such as instructions to set environmental humidity and salt density values), providing complete parametric instruction inputs for subsequent automatic modeling and rendering on the simulation platform.

[0093] Step 202: Respond to the command packet through the simulation platform and perform multimodal simulation, and use the multimodal simulation results to perform back projection to generate a three-dimensional sample point cloud.

[0094] The simulation platform is as follows:

[0095] Components: High-fidelity game engines / simulation platforms (such as Unity, Unreal Engine, NVIDIA Omniverse) and their script interfaces.

[0096] Process: Receives parameterized instruction packets from the upper layer. Automatic Modeling and Placement: Instantiates the corresponding insulator model from a preset, parameterized device resource library and applies the specified material texture (dirt). Automatic Rendering: Invokes the rendering pipeline to automatically render sample images from multiple preset perspectives: high-resolution RGB images, depth maps, surface normal maps, instance segmentation maps, etc. Point Cloud Generation: Utilizes the depth map and camera intrinsics to generate accurate, pixel-level registered 3D point clouds in real-time through back projection. Automatic Truth Mapping: Simultaneously outputs semantic segmentation maps and instance segmentation maps during the rendering process. The category (insulator, background, dirty area) and specific object instance of each pixel / point are automatically and accurately recorded without any manual intervention.

[0097] Furthermore, step 202 may include the following sub-steps:

[0098] S11. The simulation platform responds to the instruction packet by extracting the target 3D model from the device resource library and at least one texture corresponding to the attribute modification and / or associated attributes, and randomly selecting at least one occlusion model from the occlusion resource library.

[0099] S12. Render from multiple preset perspectives based on the target 3D model, at least one texture and at least one occlusion model to obtain multimodal simulation results;

[0100] The multimodal simulation results include RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps.

[0101] S13. Using the depth sample map and camera intrinsic parameters, generate a 3D sample point cloud that is registered with the depth sample map at the pixel level in real time through back projection.

[0102] In this embodiment of the invention, the simulation platform, in response to an instruction packet, extracts a target 3D model and at least one texture corresponding to attribute modification and / or associated attributes from a device resource library; and randomly selects at least one occlusion model from an occlusion resource library; renders from multiple preset perspectives based on the target 3D model, at least one texture, and at least one occlusion model to obtain RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps; using the depth sample maps and camera intrinsics, a 3D sample point cloud registered at the pixel level with the depth sample maps is generated in real time through back projection; the instance segmentation map is used to record the instance to which a pixel belongs; and the semantic segmentation map is used to record the category to which a pixel belongs.

[0103] In its implementation, after receiving the parameterized instruction package from the upper layer, the simulation platform first instantiates the corresponding target 3D model from the preset, parameterized device resource library (taking the insulator in the user instruction as an example, extracting the 3D model of the insulator), and applies the material map specified in the instruction (such as a dirt material map). At the same time, it randomly selects at least one occlusion model from the occlusion resource library to simulate the occlusion situation in the actual inspection scenario. The simulation platform calls the built-in rendering pipeline to automatically render and generate sample images from multiple preset perspectives, including high-definition RGB images, depth sample maps, surface normal maps, instance segmentation maps, etc. During the rendering process, automatic ground truth annotation is completed simultaneously. The output instance segmentation map is used to record the object instance to which the pixel belongs, and the semantic segmentation map is used to record the category to which the pixel belongs. The category (such as insulator, background, dirt area) and the specific object instance to which each pixel / point belongs are automatically and accurately recorded without any manual intervention. In the point cloud generation stage, the depth sample map and camera intrinsic parameters are used to generate accurate 3D point clouds that are pixel-level registered with the image in real time through back projection.

[0104] Equipment Resource Library: A database storing parametric 3D models of various power equipment; Target 3D Model: 3D models of power equipment in the equipment resource library that match the equipment category; Obstruction Resource Library: A database storing 3D models of common obstructions in power inspection scenarios; Obstruction Model: 3D models used to simulate equipment obstructions in real-world scenarios; Preset Multiple Viewpoints: Virtual camera rendering viewpoints pre-set by the simulation platform; Rendering: The process by which the simulation platform generates 2D image data based on the model, textures, and viewpoints; RGB Sample Image: A three-channel color image containing equipment texture and color information generated by the simulation; Depth Sample Image: A single-channel image generated by the simulation that records the distance between pixels and the virtual camera; Instance Segmentation Image: A ground truth image generated synchronously in the simulation, labeling the device instance to which a pixel belongs; Semantic Segmentation Image: A ground truth image generated synchronously in the simulation, labeling the device category to which a pixel belongs; Camera Intrinsic Parameters: Parameters describing the inherent attributes of the virtual camera, such as focal length and principal point coordinates; Pixel-Level Registration: The alignment status of multimodal data where the 3D sample point cloud and the RGB sample image pixel spatial positions correspond one-to-one.

[0105] like Figure 2 As shown, the 3D models of each power device stored in the above-mentioned equipment resource library are constructed according to the following steps:

[0106] Data is collected from power equipment using airborne cameras and airborne lidar to obtain images of the power equipment and aerial point clouds; ground-based lidar is used to collect data from the power equipment to obtain ground point clouds; the ground point clouds and aerial point clouds are registered and fused to obtain target point clouds; an initial 3D model of the power equipment is constructed based on the target point clouds; the target point clouds are registered with the images of the power equipment, and based on the registration results, the pixel information of the power equipment images is associated with the points in the initial 3D model to obtain the 3D model of the power equipment.

[0107] Furthermore, S13 may include the following sub-steps:

[0108] S131. Using camera intrinsic parameters, transform the coordinates of each pixel on the depth sample map to the camera standard coordinate system.

[0109] S132. Back-project each pixel in the camera's standard coordinate system using its corresponding depth value to obtain a point cloud in three-dimensional space.

[0110] S133. Transform the point cloud in three-dimensional space to the world coordinate system to obtain a three-dimensional sample point cloud.

[0111] Pixel: The smallest image unit in the depth sample map, carrying the corresponding depth value; Camera standard coordinate system: A three-dimensional intermediate coordinate system with the virtual camera optical center as the origin; Depth value: The distance value from the virtual camera to the scene point corresponding to the pixel in the depth sample map; Point cloud in three-dimensional space: Point cloud data generated in the camera standard coordinate system after back projection; World coordinate system: The global three-dimensional coordinate system of the simulation scene, which is the final coordinate system to which the three-dimensional sample point cloud belongs.

[0112] In this embodiment of the invention, when the simulation platform renders and generates a depth sample map, it simultaneously outputs the intrinsic parameter information of the corresponding virtual camera. These intrinsic parameters include inherent attributes such as the focal length and center position of the virtual camera. For each pixel in the depth sample map, based on its position in the image and combined with the camera intrinsic parameters, the planar position of the pixel in the camera standard coordinate system with the virtual camera lens as the origin is calculated. This is equivalent to mapping the pixel in the planar image to a virtual three-dimensional plane from the camera's perspective. Each pixel in the depth sample map has a corresponding depth value, which represents the distance from the virtual camera lens to the corresponding actual point in the scene. By combining the planar position in the camera coordinate system obtained in the previous step with the depth value corresponding to the pixel, the three-dimensional spatial position of the point in the camera's perspective can be calculated. By iterating through all pixels in the depth sample map and repeating the above calculation process, a complete set of 3D points from the camera's perspective can be obtained, which is the point cloud in the camera coordinate system. At this point, each 3D point corresponds one-to-one with a pixel in the RGB image, achieving pixel-level positional association. During simulation platform rendering, the extrinsic parameters of the virtual camera are also output synchronously. These extrinsic parameters record the actual position and orientation of the virtual camera in the entire simulation scene. Using these extrinsic parameters, all 3D points in the camera coordinate system are transformed into the global coordinate system (world coordinate system) of the simulation scene, resulting in a 3D sample point cloud that perfectly matches the position of the entire simulation scene. The final generated 3D sample point cloud is completely pixel-level / point-level registered with the RGB image, depth map, and segmentation map, requiring no additional registration operations, and can be directly used as 3D data for training multimodal models.

[0113] In the specific implementation, using depth sample maps and camera intrinsic parameters, a 3D sample point cloud registered with the depth sample map at the pixel level is generated in real time through back projection. The core principle of this step is: using the camera's pinhole imaging model, the 2D coordinates and depth value (Z) of each pixel in the depth map are used to calculate its coordinates (X, Y, Z) in the 3D camera coordinate system through the camera intrinsic parameter matrix. This is a precise and parallelizable geometric calculation process. Specifically, as follows:

[0114] I. Input:

[0115] Depth sample image: A single-channel image where each pixel value represents the vertical distance (Z value) from the camera's optical center to the object's surface. This image is typically in 16-bit or 32-bit floating-point format to maintain precision.

[0116] Camera intrinsic parameter matrix (K): A 3x3 matrix containing the camera's optical center parameters. Specifically, it can be represented by f... x and f y This represents the camera's focal length (in pixels) in the x and y directions; represented by c. x c y: Represents the coordinates (in pixels) of the camera principal point (the projection of the optical center onto the image).

[0117] II. Processing Steps

[0118] The entire process is performed independently for each pixel of the depth map, making it ideal for parallel computing (such as on a GPU).

[0119] For each pixel in the depth map, with coordinates (u, v) and depth value Z = depth(u, v), perform the following calculation:

[0120] Step 1: Transform the pixel coordinates to the camera-normalized coordinate system. The purpose is to eliminate the influence of camera intrinsic parameters and obtain the normalized coordinates of the point on the camera's imaging plane. At this point, a virtual, "ideal" image coordinate system without focal length and principal point offset is obtained. This point is located on the normalized imaging plane one unit in front of the camera.

[0121] Step 2: Apply depth values ​​for backprojection. The purpose is to project points on the normalized plane back into 3D space based on their true depth values. At this point, the coordinates (X, Y, X) of the corresponding 3D point in the camera coordinate system are obtained. camera Y camera Z camera ).

[0122] Origin: Camera optical center.

[0123] Z-axis: The direction of the camera's optical axis.

[0124] X-axis: The horizontal direction to the right of the image.

[0125] Y-axis: The vertical downward direction of the image.

[0126] Step 3: Transform to world coordinate system

[0127] The camera's pose in the virtual world (i.e., the extrinsic parameter matrix, which includes the rotation matrix R and the translation vector t) is used to transform the image into the world coordinate system. This step unifies the point clouds generated from different viewpoints into the same global coordinate system, facilitating subsequent fusion and analysis.

[0128] Step 4: Organize and output point cloud

[0129] All the calculated valid 3D points (usually excluding points with a depth of 0 or invalid values) are organized into a point cloud data structure.

[0130] Step 203: Construct a dynamic gated multimodal recognition network, and train the dynamic gated multimodal recognition network using 3D sample point cloud and multimodal simulation results to obtain the initial device recognition network.

[0131] Furthermore, step 203 may include the following sub-steps:

[0132] S21. Construct a dynamic gated multimodal recognition network, which includes an image encoder, a sparse voxel encoder, multiple dynamic fusion gate modules, and a single decoder.

[0133] The different levels of the image encoder's output terminals are connected to the first input terminals of each dynamic fusion gating module; the different levels of the sparse voxel encoder's output terminals are connected to the second input terminals of each dynamic fusion gating module; and the output terminals of each dynamic fusion gating module are connected to the input terminal of the single decoder.

[0134] In this embodiment of the invention, the specific structure and connection relationship of each module are as follows:

[0135] The image encoder includes multiple Transformer layers, which can capture the global contextual relationships within the image through a self-attention mechanism, and different layers can output multi-scale image feature maps.

[0136] The sparse voxel encoder includes multiple sparse 3D convolutional layers and sets a Transformer layer after the preset sparse 3D convolutional layers, which can output point cloud feature maps at multiple scales. The scale of the point cloud feature map corresponds one-to-one with the scale of the image feature map.

[0137] The dynamic fusion gating module is connected to specific layers of the image encoder and sparse voxel encoder. Each dynamic fusion gating module includes three types of sub-modules: feature alignment layer, gating weight generation layer, and dynamic weighted fusion layer, which are used to perform weighted fusion of point cloud feature maps and image feature maps of the same scale.

[0138] S22. Obtain training data consisting of RGB sample images, depth sample maps, 3D sample point clouds, instance segmentation maps, and semantic segmentation maps.

[0139] In this embodiment of the invention, the training data comes from the multimodal simulation results generated in step 202, wherein the RGB sample images, depth sample maps, and 3D sample point clouds are pixel-level / point-level registration relationships, and the instance segmentation map and semantic segmentation map are automatically generated ground truth labeled data, which can be used as input data and supervision labels for model training at the same time.

[0140] S23. Perform standardization and size scaling preprocessing on the RGB sample images, input them into the image encoder, and output multi-scale image feature maps.

[0141] It should be noted that the image encoder includes a block embedding layer and multiple Transformer layers connected in sequence. The image encoder receives RGB sample images that have been preprocessed by normalization and scaling. The block embedding layer performs block embedding processing on the image, and then captures the global context relationship of the image through the self-attention mechanism of each Transformer layer. The image encoder leads out multi-scale image feature map outputs from different downsampling levels and connects to the first input of each dynamic fusion gating module.

[0142] In this embodiment of the invention, the RGB sample images are first standardized to unify the distribution range of pixel values, and then scaled according to a preset size to make the image resolution adapt to the input requirements of the image encoder. The preprocessed image data is input into the image encoder, and through the self-attention calculation of multiple Transformer layers, image feature maps of different scales are extracted and output layer by layer. Each scale of image feature map corresponds to the input level of a dynamic fusion gating module.

[0143] S24. Perform dynamic sparse voxel characterization preprocessing on the 3D sample point cloud, input it into the sparse voxel encoder, and output a multi-scale point cloud feature map that corresponds one-to-one with the scale of the multi-scale image feature map.

[0144] It should be noted that the sparse voxel encoder includes multiple cascaded sparse 3D convolutional layers and at least one Transformer layer. The sparse voxel encoder receives a 3D sample point cloud preprocessed by dynamic sparsification voxel representation. Each sparse 3D convolutional layer downsamples and extracts features from the voxels. Then, through the Transformer layer connected to the output of the preset sparse 3D convolutional layer, sparse self-attention calculation is performed only on non-empty voxels. The sparse voxel encoder outputs multi-scale point cloud feature maps from different downsampling levels, which are respectively connected to the second input of each dynamic fusion gating module. The scales of the multi-scale point cloud feature maps and the multi-scale image feature maps correspond one-to-one.

[0145] In this embodiment of the invention, the three-dimensional sample point cloud is first subjected to dynamic sparse voxel characterization processing to convert the point cloud data into a sparse voxel mesh form, retaining the voxel units corresponding to the valid points; the preprocessed voxel data is input into the sparse voxel encoder, which first extracts the local spatial features of the point cloud through multiple sparse 3D convolutional layers, and then captures the long-distance dependencies of the point cloud through the subsequently set Transformer layer, and finally outputs multi-scale point cloud feature maps, and the point cloud feature map at each scale matches the size of the corresponding scale image feature map output by the image encoder.

[0146] S25. Through each dynamic fusion gating module, feature fusion is performed on the image feature map and point cloud feature map of the same scale to obtain multi-scale fused features.

[0147] It should be noted that each dynamic fusion gating module contains a feature alignment layer, a gating weight generation layer, and a dynamic weighted fusion layer. The feature alignment layer projects the point cloud feature map received from the second input end onto the image plane based on camera parameters, aligning its spatial dimension with the image feature map of the same scale received from the first input end. The gating weight generation layer concatenates the aligned point cloud features and image features along the feature dimension, and generates a single-channel gating weight map through a convolutional network and a sigmoid activation function. The dynamic weighted fusion layer, based on the gating weight map, dynamically weights and fuses the image feature map of the same scale with the aligned point cloud feature map, outputting multi-scale fused features.

[0148] In this embodiment of the invention, each dynamic fusion gating module fuses the image feature map and point cloud feature map at the corresponding scale. The specific process is as follows:

[0149] Feature alignment layer: Projects point cloud features onto the image plane, aligning the spatial dimension of the point cloud features with the image feature map of the same scale, thus achieving spatial position matching between the two features;

[0150] Gated weight generation layer: The aligned point cloud features are stitched together with the image features, and then the stitched features are processed by a convolutional network to generate a single-channel gated weight map. This weight map can reflect the importance of the two modal features at different spatial locations.

[0151] Dynamic weighted fusion layer: Based on the weights output by the gated weight generation layer, the aligned point cloud features and image features at the same scale are weighted and fused to obtain the fused features at that scale; through the above process, each dynamic fusion gate module outputs a fused feature at one scale, and finally the multi-scale fused features are obtained by combining them.

[0152] S26. Input the multi-scale fused features into the single decoder and output the semantic labels and instance embedding results.

[0153] It should be noted that the multi-scale fusion features output by each dynamic fusion gating module are all connected to the input of the single-unit decoder. The single-unit decoder receives the multi-scale fusion features output by each dynamic fusion gating module and uses a skip connection structure to perform step-by-step upsampling and progressive refinement of the fusion features. The output of the single-unit decoder outputs semantic labels and instance embedding results, which, together with the instance segmentation map and semantic segmentation map, serve as supervision signals to complete the training of the dynamic gating multimodal recognition network and obtain the initial device recognition network.

[0154] In this embodiment of the invention, the single decoder receives the multi-scale fusion features output by each dynamic fusion gating module, and gradually restores the spatial resolution of the feature map through operations such as upsampling and feature splicing, and finally outputs the semantic label and instance embedding vector corresponding to each pixel / point; wherein, the semantic label represents the device category to which the position belongs, and the instance embedding vector is used to distinguish different device instances.

[0155] S27. Supervised training is performed based on instance segmentation maps, semantic segmentation maps, semantic labels, and instance embedding results to obtain the initial device recognition network.

[0156] In this embodiment of the invention, the semantic labels output in step S26 are compared with the semantic segmentation map generated in step 202 to calculate the semantic segmentation loss; at the same time, the instance embedding results are compared with the instance segmentation map to calculate the instance discrimination loss; the two types of losses are combined to construct a joint loss function, and the parameters of the dynamic gated multimodal recognition network are updated through the backpropagation algorithm; the training iteration process is repeated until the recognition performance of the model on the validation set reaches the preset convergence condition, and finally the initial device recognition network is obtained.

[0157] Image Encoder: An encoder employing a VisionTransformer structure to extract multi-scale texture and color features from images; Sparse Voxel Encoder: An encoder employing a sparse voxel Transformer structure to extract multi-scale spatial geometric features from point clouds; Dynamic Fusion Gating Module: A core module with multi-level settings that adaptively adjusts the fusion weights of image and point cloud features; Individual Decoder: A decoding module employing a skip connection structure that outputs semantic and instance segmentation results from the device; Training Data: A multimodal training set consisting of RGB sample images, depth sample maps, 3D sample point clouds, instance segmentation maps, and semantic segmentation maps; Standardization and Resizing Preprocessing: Adapting to pixel normalization and resizing operations on the image input to the image encoder; Multi-scale Image Feature Map: Image... Examples of features include: multi-resolution image feature sets output from different levels of the encoder; dynamic sparse voxel representation preprocessing: a preprocessing method that calculates and stores only valid voxels to reduce point cloud processing overhead; multi-scale point cloud feature maps: multi-resolution point cloud feature sets output by the sparse voxel encoder that correspond to the scale of the image feature maps; feature fusion: a weighted fusion operation of image and point cloud features at the same scale by a dynamic fusion gating module; multi-scale fusion features: multi-modal complementary feature sets output by dynamic fusion gating modules at each level; semantic labels: results from the decoder representing the device category to which a pixel / point belongs; instance embedding results: high-dimensional vectors from the decoder used to distinguish different device instances; supervised training: a training method that uses simulated segmentation maps as ground truth labels and updates network parameters through backpropagation.

[0158] It is worth mentioning that the device recognition network can be a dynamic sparse point cloud-image Transformer fusion network, which has the following characteristics:

[0159] Asymmetric Feature Interaction: Design an asymmetric dual-stream encoder. The image stream uses the VisionTransformer, i.e., the image encoder described above extracts rich texture and color information; the point cloud stream uses the Sparse Voxel Transformer, i.e., the Sparse Voxel encoder described above efficiently processes large-scale scenes.

[0160] Dynamic fusion gating, also known as the aforementioned dynamic fusion gating module, introduces a learnable "fusion gating" module at multiple network levels. This module can dynamically decide whether to trust image features or point cloud features based on the characteristics of the current local area. For example, on textured device surfaces (such as nameplates), image features are more reliable, while in geometrically complex areas (such as cable intersections), point clouds are more reliable.

[0161] Individual Decoder: The decoder receives the fused multimodal features and outputs the final individual segmentation result through iterative upsampling and context aggregation. That is, each power device (such as transformer and circuit breaker) is not only identified in terms of category, but its precise three-dimensional contour and two-dimensional boundary are also fully delineated.

[0162] like Figure 3 The diagram shows the structure of the device identification network, and its data processing procedure is as follows:

[0163] I. Input and Preprocessing

[0164] Image stream input:

[0165] Data: RGB image of size H×W×3.

[0166] Preprocessing: Standardize and scale to a fixed size.

[0167] Point cloud flow input:

[0168] Data: N×3 original point cloud (X, Y, Z), possibly with intensity values.

[0169] Preprocessing: Dynamic sparsity voxel characterization.

[0170] Unlike traditional voxelization, which produces dense, mostly empty 3D meshes, this approach employs sparse voxel convolution. Only voxels containing points are computed and stored, forming a sparse voxel tensor where each non-empty voxel contains its 3D coordinates and pooled point features. This significantly reduces memory consumption and computational load, making it possible to process point clouds in large-scale power scene scenarios.

[0171] II. Dual-stream encoder

[0172] A. Image encoder: VisionTransformer

[0173] Structure: ViT or the more representative SwingTransformer variant can be used.

[0174] process:

[0175] Patching and embedding: The image is segmented into non-overlapping patches. Each patch is transformed into a feature vector through linear projection and then positional encoding is added.

[0176] Transformer layer: The feature vector sequence passes through multiple Transformer layers, and captures the global contextual relationships within the image through a self-attention mechanism (e.g., identifying the relationship between the tower body, insulator string, and conductor).

[0177] Multi-scale feature output: To better integrate with point cloud features, not only are the global features extracted from the last layer, but also multi-scale image feature maps F are extracted from intermediate layers (after sampling by 2x, 4x, and 8x). img .

[0178] B. Sparse Voxel Encoder: Sparse Voxel Transformer

[0179] Structure: Transformer architecture based on sparse voxels.

[0180] process:

[0181] Sparse voxel feature extraction: Several layers of sparse 3D convolution are used to downsample and extract features from sparse voxels, gradually expanding the receptive field.

[0182] Sparse self-attention: A Transformer layer is introduced at the critical downsampling level. Self-attention is computed only between non-empty voxels.

[0183] Each non-empty voxel is treated as a token. Query-key-value pairs are generated from voxel features.

[0184] Attention weights are enhanced by calculating the geometric relationships (distance, relative position) between voxels, making the network pay more attention to spatially adjacent parts.

[0185] Multi-scale feature output: Similarly, output multi-scale point cloud features F pc Its spatial scale and image features F img Correspondingly.

[0186] III. Dynamic Fusion Gating Module

[0187] This module is responsible for deciding "where and which modality to believe more".

[0188] Input: Image features F from the same scale img and point cloud features F pc .

[0189] Core component: Gated weight graph G. This example does not simply add or concatenate two features, but generates a spatially relevant, adaptive weight graph.

[0190] The implementation steps are as follows:

[0191] (1) Feature alignment: Using camera parameters, the point cloud features F are aligned. pc Projected onto the image plane, making its spatial dimensions the same as F img Alignment.

[0192] (2) Gating signal generation: The aligned F img and F pc The data is concatenated along the feature dimension and then passed through a small convolutional network (such as a 1x1 convolution with a sigmoid activation function) to generate a single-channel gated weight map G. Each pixel value of G is between 0 and 1.

[0193] (3) Dynamic weighted fusion:

[0194] F fused =G·F img +(1-G)·F pc

[0195] F fused The image-point cloud cross-modal fusion feature output after dynamic gating weighting, G, has the following physical meanings: G close to 1 indicates that the network highly trusts the image features. This typically occurs in areas with rich textures and vivid colors, such as equipment nameplates, the texture of insulator skirts, and the color differentiation of transformers. G close to 0 indicates that the network highly trusts the point cloud features. This typically occurs in areas with complex geometry or occlusion / light and shadow interference in the image, such as crisscrossing cables, dense forest backgrounds, and the back of equipment.

[0196] Fourth, the individual decoder upsamples the fused multimodal features and outputs the final, individualized segmentation result. It typically employs a skip connection structure similar to U-Net or FPN.

[0197] The process is as follows:

[0198] (1) Feature upsampling: Upsampling the deepest fused features by transposing convolution or interpolation.

[0199] (2) Skip connections introduce fused features of the same scale in the encoder through skip connections to supplement the details lost during the encoding process.

[0200] (3) Progressive refinement, upsampling step by step until the resolution of the original input image or point cloud is restored.

[0201] (4) The output header is followed by a convolutional layer to output two results for each pixel / point:

[0202] (5) Semantic label, which category the point belongs to (e.g., transformer, insulator).

[0203] (6) An instance is embedded in a high-dimensional vector so that all points of the same device instance have similar vectors, while the vectors of different instances differ significantly. Individualization can be achieved through subsequent clustering (such as DBSCAN), that is, distinguishing transformer A from transformer B.

[0204] The network structure in this embodiment has the following advantages:

[0205] The balance between efficiency and performance is achieved through the sparse voxel Transformer, which solves the computational bottleneck of directly processing large-scale point clouds.

[0206] Adaptive fusion and dynamic fusion gating mechanisms are no longer rigid "one-size-fits-all" fusion, but rather enable the network to learn to weigh the reliability of two modalities of information in different scenarios, thus achieving "intelligent" fusion.

[0207] Task-oriented decoding: The monolithic decoder not only completes semantic segmentation, but also directly outputs instance-level results through instance embedding, achieving the goal in one step and meeting the needs of precise positioning and management of individual devices in power scenarios.

[0208] This structure together forms a powerful and efficient platform that can make full use of the precise geometric information of point clouds and the rich texture information of images to achieve excellent segmentation performance in complex outdoor power inspection scenarios.

[0209] In some embodiments, in step S110, the associated attributes obtained from context association establish an integrated knowledge model of "equipment status - environmental conditions - physical laws". When a core attribute modification instruction is received (such as insulator pollution 5%), the device does not execute the instruction in isolation, but treats it as a state event, thereby activating a complex reasoning chain to automatically deduce and generate a series of associated attributes that are strongly related to it physically, electrically, or in operation and maintenance logic, thus achieving the comprehensiveness and authenticity of data generation. Specific implementation steps can be as follows:

[0210] The first layer: the knowledge layer—constructing a multimodal knowledge graph of power equipment, which is the brain that stores all the necessary prior knowledge.

[0211] Entity and attribute definition:

[0212] Equipment Library: Contains 3D models of all power equipment and their modifiable parameters (such as the pollution level, damage location, and material of insulators).

[0213] Environment library: Defines all environmental variables (such as humidity, temperature, light intensity, and wind speed).

[0214] Status library: Defines the operating status of the device (such as temperature, deformation, discharge intensity, salt density value).

[0215] Relationship building:

[0216] Establish weighted causal edges between the aforementioned entities. For example:

[0217] [Insulator Pollution Level ↑] -- Strong Correlation --> [Electrical Property Insulation Strength ↓]

[0218] [Insulator Pollution Level ↑] -- Strong Correlation --> [Physical Property Salt Density Value ↑]

[0219] [Insulator pollution level ↑] -- Strong correlation --> [Heat generation from thermal properties ↑] (due to increased leakage current)

[0220] Increased ambient humidity enhances the electrical effects of insulator contamination (wet and polluted conditions increase conductivity).

[0221] [Increased heat generation] leads to --> [decreased point cloud density] (thermal halo effect)

[0222] Second layer: Reasoning layer – rule-based causal chain and probability graph unfolding

[0223] When a core attribute modification instruction is received, the inference layer is activated.

[0224] Primary association (deterministic reasoning):

[0225] Direct causal retrieval: Starting with "increased pollution level of insulators" in the knowledge graph, all outgoing edges are directly retrieved to obtain the first-level associated attributes, such as "decreased insulation strength" and "increased salt density".

[0226] Application of physical constraints: Based on physical laws, the system automatically supplements associated attributes. For example, when the "salt density value" rises to a certain threshold, the system will associate it with the state attribute "increased probability of partial discharge".

[0227] Finally, the associated attributes are determined based on the attributes obtained from the two layer searches.

[0228] Step 204: Use the initial device recognition network to process the image and point cloud to obtain the device segmentation result and the corresponding confidence level.

[0229] In this embodiment of the invention, the RGB image to be tested and the 3D point cloud data are input into the initial device recognition network trained in step 203. Following the same preprocessing procedure as the training phase, the image is standardized and scaled, and the point cloud is dynamically sparsed voxel-represented. Then, the image encoder, sparse voxel encoder, and dynamic fusion gating module of the network are input. Finally, the device segmentation result corresponding to each pixel / point is output by the single decoder, including semantic labels and instance embedding vectors. At the same time, the network outputs the confidence value corresponding to each segmentation result. The confidence value reflects the network's certainty about the current recognition result and can be calculated by the class probability output by the decoder, instance embedding similarity, etc.

[0230] Step 205: Forward the test images and point clouds with confidence levels below the preset threshold to the annotation end.

[0231] In this embodiment of the invention, a confidence threshold (e.g., 0.7) is preset, and the test images and corresponding point clouds with confidence levels lower than the threshold in step 204 are filtered out. These samples are usually scenarios with high uncertainty in network recognition, such as power equipment data with missing textures, severe occlusion, or rare operating conditions, which need to be manually annotated. Therefore, they are forwarded to the annotation end, where the annotators manually annotate the images and point clouds to generate corresponding semantic segmentation maps and instance segmentation map ground truth labels.

[0232] Step 206: Train the initial device recognition network based on the annotation results from the annotation end and update the network parameters.

[0233] In this embodiment of the invention, the annotation results generated by the annotation end are combined with the corresponding test image and point cloud to form a new training sample, which is added to the training dataset. According to the training process in step 203, the new sample is input into the initial device recognition network, the semantic segmentation loss and instance discrimination loss are calculated, and the network parameters are updated through the backpropagation algorithm to improve the network's ability to recognize low-confidence scenes.

[0234] Step 207: Jump to execute the step of using the initial device recognition network to process the image to be tested and the point cloud to obtain the device segmentation result and the corresponding confidence score, until the recognition accuracy of the iteratively updated initial device recognition network reaches the preset accuracy requirement, and the target device recognition network is obtained.

[0235] In this embodiment of the invention, after the network parameters are updated in step 206, the process jumps back to step 204, and the updated device recognition network is used to continue processing new test images and point clouds, repeating the closed-loop process of "recognition and confidence assessment → low confidence sample screening and labeling → network retraining". After each iteration of training, the recognition accuracy of the device recognition network (such as average intersection-over-union ratio, instance segmentation accuracy, etc.) is evaluated through the validation set. When the recognition accuracy reaches the preset accuracy requirement, the iterative training is stopped, and the final target device recognition network is obtained.

[0236] Step 208: Use the target device recognition network to process the image and point cloud under test, and output the recognition results of the power equipment.

[0237] In this embodiment of the invention, the target device identification network obtained in step 207 has been optimized through multiple rounds of iterations, and its identification performance for various power inspection scenarios meets the preset requirements. By inputting the image to be tested and the point cloud into the target network, a high-confidence power device identification result can be output, including information such as the category, instance location, and contour boundary of each device, which can be directly used for power device inspection analysis and status assessment.

[0238] Device segmentation results: The semantic and instance segmentation results of the test data output by the device recognition network; Confidence: The network's assessment of the certainty of its own segmentation results; Preset threshold: The critical confidence value for determining whether a sample needs manual annotation; Annotation end: The annotation platform or system used for manual annotation of low-confidence samples; Annotation results: Manually annotated ground truth data of low-confidence samples output by the annotation end; Network parameters: The learnable weights and biases of the encoder, fusion module, and decoder in the recognition network; Recognition accuracy: A segmentation accuracy index that measures the performance of the device recognition network; Preset accuracy requirements: Pre-set performance judgment standards that the network needs to achieve.

[0239] It is worth mentioning that, firstly, the user-input sample generation instructions are deconstructed into structured parameters using a large language model; the structured parameters include device category, attribute modification, and associated attributes obtained by contextual association based on attribute modification; an instruction package is generated based on the structured parameters; then, the simulation platform responds to the instruction package by extracting the target 3D model from the device resource library and at least one texture corresponding to the attribute modification and / or associated attributes; and randomly selecting at least one occlusion model from the occlusion resource library; rendering is performed from multiple preset perspectives based on the target 3D model, at least one texture, and at least one occlusion model to obtain RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps; using the depth sample map and camera intrinsics, a 3D sample point cloud registered with the depth sample map at the pixel level is generated in real time through back projection; the instance segmentation map is used to record the instance to which a pixel belongs; the semantic segmentation map is used to record the category to which a pixel belongs; then, using the RGB sample image, depth sample map, 3D sample point cloud, The device recognition network is trained using instance segmentation maps and semantic segmentation maps to obtain an initial device recognition network. This network includes an image encoder, a sparse voxel encoder, multiple dynamic fusion gating modules, and a single decoder. The sparse voxel encoder comprises multiple sparse 3D convolutional layers, with a Transformer layer following the preset sparse 3D convolutional layers. The dynamic fusion gating modules are connected to specific layers of the image encoder and sparse voxel encoder, and include a feature alignment layer, a gating weight generation layer, and a dynamic weighted fusion layer. The dynamic weighted fusion layer fuses aligned point cloud features and image features at the same scale based on the weights generated by the gating weight generation layer. Finally, the initial device recognition network is used to process the test image and point cloud to obtain device segmentation results and confidence scores. Test images and point clouds with confidence scores below a preset threshold are forwarded to the annotation end. The device recognition network is trained based on the annotation results from the annotation end until a device recognition network meeting the preset accuracy is obtained. This invention achieves near-zero-cost, unlimited, and precisely labeled automatic data production through simulation and large-scale models, fundamentally solving the data bottleneck problem. At the same time, the technical solution disclosed in this invention forms a closed loop through confidence screening and active learning, breaking away from the limitations of static models, possessing autonomous evolution capabilities, and achieving autonomous intelligence that "gets smarter the more it is used."

[0240] In this invention, a large language model is used to deconstruct user-input sample generation instructions into structured parameters such as device category, attribute modification, and associated attributes, and generate instruction packages. This drives the simulation platform to respond to the instruction packages, extracting the target 3D model and corresponding textures from the device resource library and combining them with occlusion models for multi-view rendering. This generates multimodal simulation results including RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps. Then, based on the depth sample maps and camera intrinsic parameters, pixel-level registered 3D sample point clouds are generated through back projection. This allows for low-cost, large-scale generation of balanced training samples covering various device states, fault scenarios, and complex environments without relying on manual annotation, effectively solving the problems of high manual annotation costs and difficulty in obtaining rare category samples such as faulty devices in existing technologies. Simultaneously, a dynamic gated multimodal recognition network is constructed, including an image encoder, a sparse voxel encoder, multiple dynamic fusion gating modules, and a single decoder. The image encoder extracts multi-scale images... This invention extracts multi-scale point cloud features using a feature-based, sparse voxel encoder. A dynamic fusion gating module then sequentially aligns features at the same scale, generates gating weights, and dynamically weights and fuses them. This achieves adaptive weighting of image texture information and point cloud geometric information, avoiding the problem of modal information failure in complex inspection scenarios with missing textures or severe occlusion, as in traditional fixed-weight fusion methods. This significantly improves the model's adaptability to different scenarios. Based on this, the generated multi-modal simulation samples are used to supervise the training of the network to obtain an initial device recognition network. This initial network then processes the test image and point cloud, filters low-confidence data for annotation, and iteratively trains, achieving autonomous evolution and optimization of the model. Finally, a target device recognition network meeting the preset accuracy is obtained and used for power equipment recognition. This invention comprehensively improves the model's generalization ability and refined recognition accuracy, solving the technical defects of weak generalization ability and insufficient recognition accuracy caused by insufficient multi-modal data fusion in existing models.

[0241] Example 3:

[0242] Please see Figure 4 , Figure 4 This is a structural block diagram of an autonomous identification device for power equipment provided in Embodiment 3 of the present invention.

[0243] The present invention provides an autonomous identification device for power equipment, comprising:

[0244] Deconstruction module 401 is used to deconstruct user-input sample generation instructions into structured parameters using a large language model and generate instruction packages;

[0245] The processing module 402 is used to respond to the instruction packet through the simulation platform and perform multimodal simulation, and use the multimodal simulation results to perform back projection to generate a three-dimensional sample point cloud;

[0246] The initial training module 403 is used to construct a dynamic gated multimodal recognition network and train the dynamic gated multimodal recognition network using three-dimensional sample point clouds and multimodal simulation results to obtain the initial device recognition network.

[0247] The deep training module 404 is used to iteratively optimize the initial device recognition network for the test image and point cloud to obtain the target device recognition network, and then use the target device recognition network for recognition to obtain the recognition result.

[0248] Since the above is a device corresponding to a power equipment autonomous identification method, its implementation principle is the same as that of a power equipment autonomous identification method. For the sake of convenience and brevity, those skilled in the art can clearly understand that the specific working process of the device and module described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0249] Example 4:

[0250] Please see Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention.

[0251] An electronic device according to an embodiment of the present invention includes: a memory 501 and a processor 502. The memory 501 stores a computer program. When the computer program is executed by the processor 502, the processor 502 performs the power equipment autonomous identification method as described in the above embodiment.

[0252] Memory 501 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 501 has storage space 503 for program code 513 for performing any of the method steps described above. For example, storage space 503 for program code may include various program codes 513 for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When this code is run by a computing device, it causes the computing device to perform the various steps in the power equipment autonomous identification method described above.

[0253] Example 5:

[0254] Embodiment 5 of the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the power equipment autonomous identification method as described in the above embodiments.

[0255] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0256] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0257] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0258] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0259] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0260] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous identification of power equipment, characterized in that, include: The large language model is used to deconstruct user input samples into structured parameters and generate instruction packages; The simulation platform responds to the instruction packet and performs multimodal simulation, and uses the multimodal simulation results for back projection to generate a three-dimensional sample point cloud. A dynamic gated multimodal recognition network is constructed, and the dynamic gated multimodal recognition network is trained using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial device recognition network; The initial device recognition network is used to iteratively optimize the image under test and the point cloud to obtain the target device recognition network, and the target device recognition network is used for recognition to obtain the recognition result.

2. The method for autonomous identification of power equipment according to claim 1, characterized in that, The structured parameters include device category, attribute modification, and associated attributes obtained by context association based on the attribute modification. The attribute modification is used to characterize the attribute change information of the device. The texture corresponding to the attribute modification includes material texture and / or fault texture. The instruction package includes modification instructions corresponding to the attribute modification and size parameter instructions corresponding to the attribute modification.

3. The method for autonomous identification of power equipment according to claim 2, characterized in that, The process of obtaining the associated attributes is as follows: Obtain a pre-constructed knowledge graph of power equipment, wherein the nodes of the knowledge graph include power equipment, equipment components, equipment status, and environmental conditions, and the edges represent the relationships between the nodes; Based on the attribute modification, a search is performed in the power equipment knowledge graph to determine that the node corresponding to the attribute modification has an edge connecting to the first node; Based on the first node, physical attribute reasoning is performed to obtain the reasoned attributes; Based on the inference attribute, a search is performed in the power equipment knowledge graph to determine that the node corresponding to the inference attribute has an edge connecting to the second node; The associated attributes are determined based on the first node and the second node.

4. The method for autonomous identification of power equipment according to claim 2, characterized in that, The step of responding to the instruction packet through a simulation platform and performing multimodal simulation, and then using the multimodal simulation results for back projection to generate a three-dimensional sample point cloud, includes: The simulation platform responds to the instruction packet by extracting the target 3D model from the device resource library and at least one texture corresponding to the attribute modification and / or the associated attribute, and randomly selecting at least one occlusion model from the occlusion resource library. Rendering is performed from multiple preset perspectives based on the target 3D model, at least one texture, and at least one occlusion model to obtain multimodal simulation results; The multimodal simulation results include RGB sample images, depth sample maps, instance segmentation maps, and semantic segmentation maps; Using the depth sample map and camera intrinsic parameters, a 3D sample point cloud that is registered with the depth sample map at the pixel level is generated in real time through back projection.

5. The method for autonomous identification of power equipment according to claim 4, characterized in that, The step of generating a 3D sample point cloud that is registered at the pixel level with the depth sample map in real time through back projection using the depth sample map and camera intrinsic parameters includes: Using camera intrinsic parameters, the coordinates of each pixel in the depth sample map are transformed to the camera standard coordinate system; Each pixel in the camera's standard coordinate system is back-projected using its corresponding depth value to obtain a point cloud in three-dimensional space. The point cloud in the three-dimensional space is transformed to the world coordinate system to obtain a three-dimensional sample point cloud.

6. The method for autonomous identification of power equipment according to claim 4, characterized in that, The construction of a dynamic gated multimodal recognition network, and the training of the dynamic gated multimodal recognition network using the 3D sample point cloud and the multimodal simulation results to obtain an initial device recognition network, includes: A dynamic gated multimodal recognition network is constructed, which includes an image encoder, a sparse voxel encoder, multiple dynamic fusion gate modules, and a single decoder. The different level outputs of the image encoder are respectively connected to the first input of each of the dynamic fusion gating modules; the different level outputs of the sparse voxel encoder are respectively connected to the second input of each of the dynamic fusion gating modules; the outputs of each of the dynamic fusion gating modules are connected to the input of the single decoder. Acquire training data consisting of the RGB sample image, the depth sample map, the 3D sample point cloud, the instance segmentation map, and the semantic segmentation map; The RGB sample images are preprocessed by standardization and size scaling, and then input into the image encoder to output multi-scale image feature maps; The three-dimensional sample point cloud is preprocessed with dynamic sparse voxel representation and input into the sparse voxel encoder to output a multi-scale point cloud feature map that corresponds one-to-one with the scale of the multi-scale image feature map. Through the dynamic fusion gating modules, feature fusion is performed on image feature maps and point cloud feature maps of the same scale to obtain multi-scale fused features; The multi-scale fused features are input into the single decoder, and semantic labels and instance embedding results are output. The initial device recognition network is obtained by supervised training based on the instance segmentation map, the semantic segmentation map, the semantic labels, and the instance embedding results.

7. The method for autonomous identification of power equipment according to any one of claims 1-6, characterized in that, The initial device recognition network is used to iteratively optimize the image under test and the point cloud to obtain the target device recognition network, and the target device recognition network is used for recognition to obtain the recognition result, including: The initial device recognition network is used to process the image and point cloud under test to obtain the device segmentation result and the corresponding confidence level. Forward the test image and point cloud with a confidence level lower than the preset threshold to the annotation terminal; The initial device identification network is trained and its parameters are updated based on the annotation results from the annotation end. Jump to the step of processing the image and point cloud under test using the initial device recognition network to obtain the device segmentation result and the corresponding confidence level, until the recognition accuracy of the iteratively updated initial device recognition network reaches the preset accuracy requirement, and the target device recognition network is obtained. The target device recognition network is used to process the image and point cloud to output the recognition result of the power equipment.

8. A power equipment autonomous identification device, characterized in that, include: The deconstruction module is used to deconstruct user-input samples into structured parameters using a large language model and generate instruction packages. The processing module is used to respond to the instruction packet through the simulation platform and perform multimodal simulation, and use the multimodal simulation results to perform back projection to generate a three-dimensional sample point cloud; An initial training module is used to construct a dynamic gated multimodal recognition network and train the dynamic gated multimodal recognition network using the three-dimensional sample point cloud and the multimodal simulation results to obtain an initial device recognition network. The deep training module is used to iteratively optimize the initial device recognition network for the test image and point cloud to obtain the target device recognition network, and then use the target device recognition network to perform recognition to obtain the recognition result.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the power equipment autonomous identification method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the power equipment autonomous identification method as described in any one of claims 1-7.

Images