A power distribution network scene open vocabulary segmentation method and system fusing multi-fine-grained semantics, an electronic device, and a storage medium

Abstract

The application relates to the fields of power distribution network intelligent inspection, three-dimensional scene understanding and open vocabulary semantic segmentation, and discloses a power distribution network scene open vocabulary segmentation method and system fusing multiple fine-grained semantics, an electronic device and a storage medium, which comprise the following steps: acquiring multiple-view images to construct a three-dimensional scene expression; extracting semantic features from a two-dimensional candidate mask and aggregating the semantic features to a three-dimensional scene unit; constructing fine-grained candidates and a relation graph, forming multiple-level candidates through multi-level aggregation and recording parent-child attribution; parsing a query into a query quadruple of a target, a parent subject, an attribute and a spatial relation, scoring to obtain a target candidate under the constraint of the parent subject, performing cross-level re-search to obtain a revised candidate when the confidence is low, and outputting a two-dimensional segmentation mask and a three-dimensional target region. The application improves the granularity adaptation capability of power distribution network scene open vocabulary segmentation and the spatial attribution accuracy of local component segmentation, and simultaneously considers the two-dimensional mask boundary precision and the three-dimensional scene consistency.

Description

Technical Field

[0001] This invention relates to the fields of intelligent inspection of power distribution networks, 3D scene understanding, and open vocabulary semantic segmentation. Specifically, it is an open vocabulary segmentation method for power distribution network scenes that integrates multi-fine-grained semantics to perform fine segmentation of equipment targets, constituent areas, and local components in a power distribution network scene based on natural language queries input by users. Background Technology

[0002] With the development of visual language models, multi-view 3D reconstruction, and intelligent inspection technologies, open-vocabulary 3D segmentation technology has formed a relatively clear development path: First, 2D visual language models achieve text-driven target localization by mapping image regions and text to a unified semantic space; second, 3D language field methods map and aggregate 2D semantic features to 3D scene units, and improve the stability of multi-view semantic features through visibility perception aggregation; on this basis, fine-grained 3D candidate maps and hierarchical candidate methods further organize scene units into structured multi-level semantic regions.

[0003] In power distribution network inspection scenarios, switchgear and other power distribution equipment have a hierarchical structure of "overall equipment - sub-components - parts". For example, switchgear includes sub-components such as cabinet doors and operation panels, and operation panels include parts such as buttons, handles, and indicator lights. Open vocabulary segmentation tasks need to support queries at different semantic granularities, such as "switchgear" (equipment level), "operation panel" (sub-component level), and "red button" (part level).

[0004] The inventors of this application further analyzed the search results and concluded that the existing technology still has the following shortcomings when processing open vocabulary segmentation in the power distribution equipment scenario: (1) The candidate region has insufficient expressive power and it is difficult to cover multiple levels of targets such as complete equipment, sub-components and local parts at the same time; (2) Queries with different semantic granularities are not effectively distinguished, and the use of text similarity matching uniformly reduces the adaptability of the method to different query types; (3) There is a lack of clear spatial ownership constraints between local parts and their respective subjects, and a large number of similar parts (such as buttons and indicator lights on different equipment) in the power distribution scenario are prone to mismatch; (4) Small part targets are easily covered by complete equipment candidates in the three-dimensional semantic expression due to their imprecise boundaries or small area proportions. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an open vocabulary segmentation method for power distribution network scenes that can construct a multi-granularity candidate space that can simultaneously cover complete equipment, constituent regions and local components under the conditions of multi-view images and three-dimensional scene representation of power distribution networks, and select target candidates according to query quadruples and parent subject constraints, thereby stably outputting a three-dimensional target region consistent with the query semantics.

[0006] To address the aforementioned technical problems, this invention provides a method for open vocabulary segmentation in power distribution network scenarios that integrates multiple fine-grained semantics. The method includes: acquiring multi-view image sequences and camera parameters of a power distribution network scenario; constructing a three-dimensional scene representation based on the multi-view images and camera parameters, wherein the three-dimensional scene representation includes multiple three-dimensional scene units; generating two-dimensional candidate masks from the multi-view images and extracting open vocabulary semantic features corresponding to the two-dimensional candidate masks; establishing association weights between the two-dimensional candidate masks and the three-dimensional scene units based on camera projection relationships and three-dimensional visibility relationships, and aggregating the open vocabulary semantic features of the two-dimensional candidate masks into the three-dimensional scene units according to the association weights; constructing multiple fine-grained three-dimensional candidates based on the geometric position, semantic features, and co-occurrence relationships of the two-dimensional masks, and establishing a candidate relationship graph using the fine-grained three-dimensional candidates as nodes. Based on the association scores between candidates in the candidate relationship graph, multi-granularity hierarchical aggregation is performed on the fine-grained 3D candidates to form a multi-level candidate set containing low-level, mid-level, and high-level candidates, and the parent-child relationship between candidates is recorded; an open-vocabulary text query is received, and the text query is parsed into target semantic items, parent subject semantic items, attribute semantic items, and spatial relationship semantic items; a subject candidate is determined in the multi-level candidate set according to the parent subject semantic item, and the target candidate is scored within the constraints of the subject candidate to obtain an initial target candidate; when the confidence of the initial target candidate does not meet the preset conditions, cross-level candidate re-retrieval and re-sorting are performed in the multi-level candidate set to obtain a corrected target candidate; based on the corrected target candidate or the initial target candidate, the 2D segmentation mask and 3D target region corresponding to the target are output.

[0007] Preferably, the three-dimensional scene is expressed as a three-dimensional Gaussian field, and the three-dimensional scene unit is a three-dimensional Gaussian unit.

[0008] As a further preferred embodiment, each three-dimensional Gaussian element includes: a center position Scale vector Rotation parameters The covariance matrix determined by the scale vector and rotation parameters Opacity Color characteristics and semantic features .

[0009] Preferably, the two-dimensional candidate mask is generated on images from various viewpoints using a general candidate segmentation model or an instance segmentation model, and the open-vocabulary semantic features are extracted using a visual language model.

[0010] Preferably, the association weight From the 3D scene unit in the first Visibility contribution from each perspective , No. Quality score of a two-dimensional candidate mask and the depth consistency score between 2D candidates and 3D scene units. Jointly determined: ， Preferably, the aggregation of open-vocabulary semantic features of two-dimensional candidate masks into three-dimensional scene units includes: ， in, For the first From the perspective of the first Semantic features of a two-dimensional candidate mask To prevent small positive numbers with a denominator of zero, This is a normalization operation.

[0011] Preferably, the candidate relationship graph is represented as follows: ,in, It is a fine-grained set of 3D candidate nodes. The set of edges between candidate nodes; edge association score Based on the semantic similarity between candidates 3D geometric adjacency Multi-view projection consistency Co-occurrence with 2D masks Weighted average: ， in, , , and These are non-negative weighting coefficients.

[0012] Preferably, the multi-granularity hierarchical aggregation includes: taking the fine-grained 3D candidate as the initial node, merging the two candidates into a candidate of the next higher level when the association score between the two candidates meets the aggregation condition and satisfies the spatial connectivity constraint and the candidate area growth constraint; updating the geometric range, semantic features, area or volume, graph edge weights with other candidates and parent-child affiliation relationship of the merged candidate after each merger; stopping the aggregation when the association score is lower than a preset association threshold, or the number of candidate layers reaches a preset number, or there are no candidate pairs that meet the conditions.

[0013] Preferably, the parent-child affiliation record records the hierarchical affiliation chain from low-level candidates to direct parent candidates and then to high-level candidates, including: component candidates → direct parent candidates → equipment body candidates.

[0014] Preferably, the text query is parsed into target semantic items. Parent-level main semantic items Attribute semantic items and spatial relation semantic items In this context, non-existent semantic terms are set to empty; the parsing method includes at least one of the following: parsing based on the parent-component vocabulary of the power distribution equipment, rule matching, syntactic analysis, or text encoder.

[0015] Preferably, the scoring of target candidates within the subject candidate constraint range includes: for each candidate, calculating a comprehensive score that includes semantic similarity score, parent affiliation consistency score, spatial inclusion score, scale ratio consistency score, cross-view consistency score, and cross-device conflict score.

[0016] Preferably, the confidence level not meeting the preset conditions includes at least one of the following: (a) there is no valid attribution relationship between the target candidate and the parent subject candidate; (b) the area of ​​the target candidate. With parent candidate area The ratio exceeds a reasonable range, where only the lower limit is specified and no upper limit is set, and the lower limit is 0.3; (c) the initial target candidate is empty.

[0017] Preferably, the cross-level candidate re-retrieval and re-ranking includes: when the confidence of the initial target candidate does not meet the conditions, re-retrieval is performed in the adjacent level candidates of the multi-level candidate set, and the candidates are re-scored and ranked in combination with semantic similarity, parent affiliation consistency, spatial inclusion degree, area ratio and cross-perspective consistency, and the best candidate after re-ranking is used to replace the initial target candidate.

[0018] This invention also provides an open vocabulary segmentation system for power distribution network scenarios that integrates multiple fine-grained semantics, comprising: an image acquisition module for acquiring multi-view image sequences and camera parameters of a power distribution network scenario; a 3D scene construction module for constructing a 3D scene representation including multiple 3D scene units based on the multi-view images and camera parameters; a 2D candidate generation module for generating 2D candidate masks for each view image and extracting open vocabulary semantic features; a semantic aggregation module for aggregating 2D candidate semantic features into 3D scene units based on visibility relationships; a multi-granularity candidate construction module for constructing fine-grained 3D candidates and candidate relationship graphs, and performing multi-granularity hierarchical aggregation to form a multi-level candidate set containing parent-child attribution relationships; a query parsing module for parsing open vocabulary queries into target semantic items, parent subject semantic items, attribute semantic items, and spatial relationship semantic items; a candidate selection module for scoring and selecting target candidates within the constraints of parent subject candidates; a confidence correction module for performing cross-level candidate re-retrieval and re-sorting when the initial target candidate confidence is insufficient; and an output module for outputting the 2D segmentation mask and 3D target region corresponding to the target.

[0019] The present invention also provides an electronic device, including a processor and a memory, the memory storing a computer program, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0021] With the above structure, the present invention has the following beneficial effects: Compared with the existing three-dimensional open vocabulary segmentation technology, the beneficial effects of the present invention are mainly reflected in the following aspects.

[0022] First, this invention improves the granularity adaptability of open vocabulary segmentation in power distribution network scenarios. Existing methods typically match text semantics directly with 3D region features, and candidate regions tend to be biased towards complete devices or local fragments. This invention constructs a candidate hierarchy composed of local components, constituent regions, and complete devices through multi-granularity hierarchical aggregation, enabling the simultaneous querying and segmentation of targets with different granularities such as "switch cabinet," "operation panel," "switch cabinet handle," "button," and "indicator light" within the same scenario.

[0023] Secondly, this invention improves the spatial attribution accuracy of local component segmentation. For queries of local components such as "switch cabinet handle," "button," "dashboard," and "insulator skirt," this invention first determines the main anchoring region, then retrieves component candidates within or near the main region, and reorders the candidates based on main content, area ratio, coverage ratio, and cross-layer consistency. This avoids the expansion of local component query results into complete main regions and reduces segmentation results that are semantically related but spatially incorrect.

[0024] Furthermore, this invention balances the accuracy of 2D mask boundaries with the consistency of 3D scenes. Fine-grained 2D candidate masks provide clearer local boundaries, and multi-view projection and association with 3D scene units can form stable 3D target regions. By aggregating 2D candidate semantics to 3D scene units, this invention preserves local component boundaries while ensuring consistent 3D representation of the same target across different viewpoints.

[0025] Query quadruple middle, For the target semantic item, For parent-level main semantic items, For attribute semantic items, This represents the semantic terms of spatial relations; non-existent semantic terms are set to empty.

[0026] The comprehensive score includes six evaluation dimensions: target semantic similarity, parent affiliation consistency, spatial inclusion degree, scale ratio consistency, cross-view consistency, and cross-device conflict degree.

[0027] The corresponding system includes: an image acquisition module to acquire multi-view images; a 3D scene construction module to generate a 3D scene representation; a 2D candidate generation module to extract candidate masks and semantic features; a semantic aggregation module to realize 2D to 3D semantic mapping; a multi-level aggregation module to construct candidate levels and relationship graphs; a query parsing module to parse natural language queries; a component perception scoring module to score under subject constraints; a low-confidence re-retrieval module to correct candidates across levels; and a result output module to generate 2D masks and 3D target regions.

[0028] Furthermore, this invention improves segmentation stability under complex query conditions. This invention performs semantic granularity parsing on natural language queries, dividing the query into target semantic items, parent subject semantic items, attribute semantic items, and spatial relationship semantic items, and sets different candidate selection processes for different query types. For cases with low confidence, empty masks, excessively large masks, or unstable candidate scores, this invention further performs low-confidence cross-level candidate re-retrieval and re-ranking, performing secondary ranking on multi-level candidates, thereby improving the stability of results under difficult queries.

[0029] Experimental results show that in the open-vocabulary 3D segmentation validation, after introducing low-confidence cross-level candidate re-retrieval and re-ranking, the overall mIoU improved from 0.5462 to 0.6219, indicating that the present invention can improve the overall open-vocabulary segmentation accuracy. In this validation example, the consistency between local component query results and the real region was improved. For cases where components are submerged by complete subjects, after adopting component-aware semantic anchoring, the local component IoU improved from 0.0642 to 0.9840, demonstrating that the present invention can significantly improve the localization accuracy and masking accuracy of fine-grained component queries.

[0030] Therefore, compared with existing technologies, this invention can achieve more stable and refined open vocabulary segmentation in power distribution network scenarios, and is particularly suitable for overall equipment identification, local component location, attribute target query, and 3D interactive inspection tasks in the inspection of switchgear in power distribution substations or distribution rooms. This method helps reduce the workload of manual labeling and component searching, improves the inspection system's ability to identify unknown targets and fine-grained components, and provides more reliable technical support for intelligent power distribution network inspection, equipment status awareness, and defect location. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the open vocabulary segmentation method for power distribution network scenarios that integrates multiple fine-grained semantics in this invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings.

[0033] Combined with appendix Figure 1 This invention provides an open vocabulary segmentation method for power distribution network scenarios that integrates multiple fine-grained semantics. The method is designed for multi-view images and three-dimensional scene representations of power distribution networks. Based on the natural language query input by the user, it outputs a two-dimensional segmentation mask or a three-dimensional target region corresponding to the query semantics.

[0034] In a preferred embodiment, the power distribution network scenario is a switchgear inspection scenario in a power distribution substation or distribution room. The objects to be segmented include complete equipment, constituent areas, and partial components such as switchgear, cabinet doors, operation panels, buttons, handles, knobs, indicator lights, instrument panels, and nameplates. In other embodiments, the method is also applicable to other power distribution network equipment scenarios such as poles, conductors, insulators, hardware, and clamps.

[0035] The technical solution adopted by this invention to solve its technical problem is: An open vocabulary segmentation method for power distribution network scenarios that integrates multi-fine-grained semantics, the method comprising the following steps: S100: Acquire multi-view image sequences and camera parameters for power distribution network scenarios.

[0036] S200: Construct a 3D scene representation including multiple 3D scene units based on the multi-view image sequence and camera parameters.

[0037] S300: Generate two-dimensional candidate masks in images from various viewpoints and extract open-vocabulary semantic features corresponding to the two-dimensional candidate masks.

[0038] S400: Based on camera projection relationship and 3D visibility relationship, establish the association weight between 2D candidate mask and 3D scene unit, and aggregate 2D semantic features into 3D scene unit.

[0039] S500: Based on the geometric position, semantic features, and co-occurrence relationship of 3D scene units, fine-grained 3D candidates are constructed, and a candidate relationship graph is established.

[0040] S600: Perform multi-granularity hierarchical aggregation based on the candidate aggregation association score in the candidate relationship graph to form a multi-level candidate set and record the parent-child relationship between candidates.

[0041] S700: Parse the text query into target semantic items, parent subject semantic items, attribute semantic items and spatial relationship semantic items, calculate the candidate comprehensive score under the parent subject candidate constraint, and obtain the initial target candidate.

[0042] S800: When the confidence level of the initial target candidate does not meet the preset conditions, perform low-confidence cross-level candidate re-retrieval and re-ranking in the multi-level candidate set to obtain the corrected target candidate.

[0043] S900: Generates and outputs the two-dimensional segmentation mask and three-dimensional target region corresponding to the target based on the initial target candidate or the modified target candidate.

[0044] Specifically, this invention achieves unified open vocabulary segmentation of complete equipment, constituent regions, and local components in power distribution network scenarios through the process of "multi-view image acquisition—3D scene representation construction—2D candidate and semantic feature generation—2D and 3D association and semantic aggregation—fine-grained 3D candidate and relationship graph construction—multi-granularity hierarchical aggregation—query quadruple parsing and parent subject constraint scoring—low confidence cross-level candidate re-retrieval and re-ranking—2D and 3D result output".

[0045] Furthermore, in S100, the input is a multi-view image sequence of the power distribution network scenario. ,in This represents the v-th view image, where V represents the number of views; each view image corresponds to camera parameters. A 3D scene representation is constructed based on the multi-view images and camera parameters. ,in This represents the k-th 3D scene unit, where K represents the number of 3D scene units.

[0046] In a preferred embodiment, the three-dimensional scene is represented as a three-dimensional Gaussian field, and each three-dimensional scene unit... Including the space center Scale vector and covariance matrix Opacity Color characteristics and semantic features Through multi-view reconstruction or optimization, the 3D Gaussian field can render a scene appearance consistent with the input image from different viewpoints. The semantic features... The initialization phase is empty or randomly initialized, and the results are obtained by aggregating the semantic features of the multi-view two-dimensional candidate mask in subsequent steps.

[0047] In other embodiments, the three-dimensional scene is represented as a point cloud, voxel, mesh, fine-grained three-dimensional candidate, or other three-dimensional representation, and the three-dimensional scene unit corresponds to a three-dimensional point, voxel unit, mesh patch, or fine-grained three-dimensional candidate unit.

[0048] Furthermore, in S200, for each viewpoint image Generate a fine-grained set of two-dimensional candidate masks ,in This represents the nth two-dimensional candidate mask from the vth viewpoint. This indicates the number of candidate masks from this viewpoint.

[0049] For each two-dimensional candidate mask Extracting visual semantic features The visual semantic features include region appearance features, open-vocabulary semantic features, attribute features, and mask quality features. Region appearance features are used to describe the color, texture, shape, and boundary of candidate regions; open-vocabulary semantic features are used for similarity matching with natural language queries; attribute features are used to describe color, state, and material; and mask quality features are used to describe mask area, boundary integrity, and region compactness.

[0050] For 3D scene units Project it onto the v-th viewpoint to obtain the projection position. and visibility contribution .like Located in a two-dimensional candidate mask Internal and visibility contribution Not lower than the preset threshold Then, a three-dimensional scene unit is established. With two-dimensional candidate masks The relationships between them.

[0051] Association weight The visibility contribution of 3D scene units, the quality score of 2D candidate masks, and the consistency of projection depth are jointly determined, and are specifically expressed as follows: , in, Represents a two-dimensional candidate mask The mask quality score, Representing a 3D scene unit With two-dimensional candidate masks The depth consistency score.

[0052] Based on the multi-view correlation, the visual semantic features of the two-dimensional candidate mask are aggregated onto the three-dimensional scene unit to obtain the open-vocabulary semantic features of the three-dimensional scene unit: , Here, Normalize represents the feature normalization operation. Through this step, each 3D scene unit obtains open-vocabulary semantic features from the multi-view 2D candidate mask.

[0053] Furthermore, in S300, fine-grained 3D candidate units are generated based on the spatial and semantic relationships between 3D scene units. Specifically, this includes: S510: Construct spatial adjacency relationships based on the spatial distance between 3D scene units. If the distance between two 3D scene units is less than a preset spatial threshold, they are considered to have a spatial proximity relationship. S520: Construct mask co-occurrence relationship based on multi-view projection results. If two 3D scene units fall into the same 2D candidate mask or adjacent 2D candidate masks in multiple views, they are considered to have a multi-view co-occurrence relationship. S530: Calculate semantic similarity based on the semantic features of three-dimensional scene units. If the semantic feature similarity of two three-dimensional scene units is higher than the preset semantic threshold, they are considered to have a semantically consistent relationship. S540: Based on spatial adjacency, multi-view co-occurrence, and semantic consistency, perform region growing or graph clustering on 3D scene units to obtain the lowest-level fine-grained 3D candidate set. ; S550: Using each fine-grained 3D candidate unit as a graph node, a candidate region relationship graph is established based on the spatial proximity, projective adjacency, semantic similarity, and mask co-occurrence among candidates. .

[0054] In the candidate region relation graph, the edge weight W_{ij} represents the candidate region. With candidates The probability of being aggregated into the same upper-level candidate is as follows: , in, Indicates the semantic similarity of candidates. Represents three-dimensional geometric proximity. Indicates the consistency of multi-view projection. This indicates the co-occurrence relationship of two-dimensional masks. , , , These are adjustable weighting coefficients.

[0055] Through S300, the method obtains fine-grained 3D candidate bases that are spatially continuous, semantically consistent, and able to preserve local component boundaries, providing input for subsequent multi-granularity hierarchical aggregation.

[0056] The feasibility of this invention has been verified through prototype system and open vocabulary 3D segmentation experiments. The verification results show that the multi-granularity hierarchical aggregation, component-aware semantic anchoring, and low-confidence cross-level candidate re-retrieval and re-ranking process proposed in this invention can effectively improve the granularity adaptability and local component segmentation accuracy of open vocabulary segmentation. A specific embodiment is given below.

[0057] It should be noted that the low-level, mid-level, and high-level candidates are algorithmic scale levels formed by candidate aggregation and do not mechanically correspond to the device semantic level. By incorporating knowledge from the power distribution equipment domain, semantic correspondences between the equipment layer, sub-component layer, and component layer can be further established; for example, switchgear belongs to the equipment layer, cabinet doors and operating panels belong to the sub-component layer, and handles, buttons, indicator lights, knobs, insulator skirts, and hardware belong to the component layer.

[0058] Furthermore, in S600, the input for multi-granularity hierarchical aggregation is a fine-grained three-dimensional candidate set and a candidate relationship graph R=(VC,EC). In the candidate relationship graph, each node corresponds to a fine-grained three-dimensional candidate, and the candidate aggregation association score... Based on the semantic similarity between candidates 3D geometric adjacency Multi-view projection consistency Co-occurrence with 2D masks Weighted average.

[0059] The aggregation process employs a bottom-up hierarchical merging strategy: when the association scores between two candidates satisfy the aggregation conditions, and also meet the spatial connectivity constraints and candidate area growth constraints, the two candidates are merged into the next higher-level candidate. After each merge, the 3D geometric extent, semantic features, area or volume, graph edge weights with other candidates, and parent-child relationships of the merged candidate are updated.

[0060] The aggregation stopping condition includes at least one of the following: (a) the association score between all candidate pairs is lower than the preset association threshold τagg, where τagg = 0.5; (b) the number of candidate layers has reached the preset maximum number of layers Lmax, where Lmax = 3; (c) there are no candidate pairs in the current layer that meet the aggregation condition. After aggregation, a multi-level candidate set is formed, including low-level candidate Cl, mid-level candidate Cm, and high-level candidate Ch, and the parent-child affiliation chain such as handle → cabinet door → switch cabinet, button → operation panel → switch cabinet, umbrella skirt → insulator → insulator string is recorded.

[0061] Furthermore, in S700, open-lexical text queries are parsed into query quadruples. ,in This represents the target semantic item to be segmented. This indicates the semantic item of the parent body to which the target belongs. This represents semantic items such as color, material, and state. This indicates semantic terms representing spatial relationships such as left, top, inside, and adjacent; non-existent semantic terms are set to null values.

[0062] The actual parsing methods used include at least one of the following: parsing based on the parent-component knowledge relationship of power distribution equipment, rule matching, syntactic analysis, or text encoder. For example, for the query "switch cabinet handle", the parsing result is... =“handle” =“Switch cabinet” =empty, =empty; For the query "red button on switch cabinet", the parsed result is: =“button” =“Switch cabinet” =“red” =“Located on…”.

[0063] After the parent candidate is determined, the target candidate is scored within the three-dimensional spatial region of the candidate or its subtree candidate set. For each candidate i, its comprehensive score Si can be determined by the target semantic similarity, parent affiliation consistency, spatial inclusion degree, scale ratio consistency, cross-view consistency, and cross-device conflict degree; where each scoring item corresponds to Sisem, Siparent, Sicontain, Siscale, Sicross, and Siconflict, respectively, and β1 to β6 are the corresponding weights.

[0064] Furthermore, in S800, the calculable determination that the initial target candidate confidence does not meet the conditions includes at least one of the following: (a) there is no valid attribution relationship between the target candidate and the parent subject candidate, that is, the target candidate is not within the three-dimensional spatial influence range of the subject candidate or is not in the sub-candidate tree of the subject candidate; (b) the ratio of the target candidate area Atarget to the parent subject candidate area Aparent is less than a preset ratio threshold, the preset ratio threshold being 0.3; (c) the initial target candidate is empty, that is, there are no candidates that meet the basic threshold.

[0065] When any of the above conditions are met, low-confidence cross-level candidate re-retrieval and re-ranking are triggered. Specifically, this includes: re-retrieval among adjacent level candidates in the multi-level candidate set, re-scoring and ranking the retrieved candidates based on semantic similarity, parent affiliation consistency, spatial inclusion degree, area ratio, and cross-perspective consistency, and selecting the candidate with the highest score as the corrected target candidate; if the corrected candidate score still does not meet the conditions, the retrieval scope can be expanded to higher levels until the conditions are met or the highest level is reached.

[0066] Specific Implementation Example 1: An Implementation Example of Open Vocabulary Segmentation for Switchgear Scenarios in Substations.

[0067] In this embodiment, the scenario to be processed is a switchgear inspection scenario in a power distribution substation or distribution room. The data acquisition equipment can be an RGB camera, depth camera, or multi-view image acquisition device mounted on an inspection robot, or a handheld camera can be used to take multi-view pictures around the switchgear. The objects to be acquired include the switchgear as a whole, cabinet doors, operation panels, handles, buttons, knobs, indicator lights, instrument panels, and nameplates.

[0068] The required input data includes: multi-view image sequences, camera intrinsic and extrinsic parameters for each viewpoint, a 3D scene representation reconstructed from the multi-view images, and an open-ended vocabulary query input by the user. The 3D scene representation is preferably a 3D Gaussian field, but can also be replaced with point clouds, voxels, meshes, or fine-grained 3D candidate structures.

[0069] The specific implementation process is as follows.

[0070] S100: Acquire multi-view image sequences and camera parameters for power distribution network scenarios.

[0071] S200: Construct a 3D scene representation including multiple 3D scene units based on the multi-view image sequence and camera parameters.

[0072] S300: Generate two-dimensional candidate masks in images from various viewpoints and extract open-vocabulary semantic features corresponding to the two-dimensional candidate masks.

[0073] S400: Based on camera projection relationship and 3D visibility relationship, establish the association weight between 2D candidate mask and 3D scene unit, and aggregate 2D semantic features into 3D scene unit.

[0074] S500: Based on the geometric position, semantic features, and co-occurrence relationship of 3D scene units, fine-grained 3D candidates are constructed, and a candidate relationship graph is established.

[0075] S600: Perform multi-granularity hierarchical aggregation based on the candidate aggregation association score in the candidate relationship graph to form a multi-level candidate set and record the parent-child relationship between candidates.

[0076] S700: Parse the text query into target semantic items, parent subject semantic items, attribute semantic items and spatial relationship semantic items, calculate the candidate comprehensive score under the parent subject candidate constraint, and obtain the initial target candidate.

[0077] S800: When the confidence level of the initial target candidate does not meet the preset conditions, perform low-confidence cross-level candidate re-retrieval and re-ranking in the multi-level candidate set to obtain the corrected target candidate.

[0078] S900: Generates and outputs the two-dimensional segmentation mask and three-dimensional target region corresponding to the target based on the initial target candidate or the modified target candidate.

[0079] Example 2: Prototype Verification of 3D Segmentation for Open Vocabulary To verify the effectiveness of the multi-granularity hierarchical aggregation, component-aware semantic anchoring, and low-confidence cross-level candidate re-retrieval and re-ranking mechanisms in this invention, an open-vocabulary 3D segmentation prototype system was constructed for experimental verification. The experimental system uses a 3D Gaussian field as the basis for scene representation, generates 3D candidates by utilizing the correlation between multi-view 2D candidate masks and 3D scene units, and performs open-vocabulary queries on multi-level candidate sets.

[0080] The experimental environment includes: a computing server equipped with an NVIDIA RTX 4090 GPU, a Python deep learning environment, a 3D Gaussian field rendering module, a 2D candidate mask generation module, a visual language feature extraction module, a multi-granularity candidate aggregation module, a query semantic parsing module, a component-aware semantic anchoring module, and a low-confidence cross-level candidate re-retrieval and re-ranking module.

[0081] In the experiment, a baseline open-vocabulary 3D segmentation result was first constructed. Then, component-aware semantic anchoring and a low-confidence cross-level candidate re-retrieval and re-ranking mechanism were added for comparison. Evaluation metrics included Intersection over Union (IoU), mean IoU (mIoU), mean accuracy (mAcc), mean precision (mP), mean recall (mR), and F1 score.

[0082] Experimental results show that in the open vocabulary 3D segmentation validation, the overall mIoU of the baseline method is 0.5462. After introducing low-confidence cross-level candidate re-retrieval and re-ranking, the overall mIoU is improved to 0.6219, an improvement of 0.0757. The upper limit of the oracle of the candidate set reaches 0.7362, indicating that the constructed multi-granularity candidate space can cover more potential correct targets.

[0083] In local component query validation, the baseline method tends to expand local component queries into complete subject regions. For example, in a component query example, the IoU of the baseline result is 0.0642, and the predicted mask area is 154099. After adopting component-aware semantic anchoring, the method first determines the subject anchoring region, and then selects fine-grained candidates within the subject that meet the component semantics and area constraints. Finally, the IoU is improved to 0.9840, and the predicted mask area is 9761, which is close to the actual labeled area of ​​9906. This result proves that the present invention can effectively suppress the problem of local component queries being submerged by complete subject regions.

[0084] Meanwhile, experiments also show that for extremely small, low-saliency, and unstable component targets, if there are no stable and correct fine-grained candidates in the multi-granularity candidate set, component-aware semantic anchoring will still be affected by candidate quality. Therefore, in practical applications, it is necessary to ensure that the initial two-dimensional candidate mask has sufficient fine-grainedness, and to improve candidate coverage through multi-view association and cross-layer consistency.

[0085] In summary, the feasibility of this invention has been verified through prototype system and open-vocabulary 3D segmentation experiments. Experimental results demonstrate that this invention can construct an effective multi-granularity candidate space and improve the accuracy and stability of open-vocabulary segmentation results through query semantic parsing, component-aware semantic anchoring, and low-confidence cross-level candidate re-retrieval and re-ranking. This method can be deployed in the above embodiments in the scenario of switchgear inspection in power distribution rooms or distribution substations for open-vocabulary segmentation of complete equipment, constituent areas, local components, and attribute targets.

[0086] In summary, the key improvement of this invention lies in proposing a complete open-vocabulary segmentation process for power distribution network scenarios. This process combines multi-view images, 3D scene representation, fine-grained candidate masks, hierarchical candidate aggregation, and natural language query parsing to achieve unified segmentation of complete equipment, constituent regions, and local components. Unlike methods that directly select target regions based solely on text similarity, this invention first constructs a multi-granularity candidate space, and then performs parent subject constraints, attribute constraints, spatial relationship constraints, and target candidate scoring and selection based on the non-empty semantic items in the query quadruple, thereby improving the granularity adaptability and spatial attribution accuracy of the open-vocabulary segmentation results.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any equivalent substitutions, simple modifications, or improvements made within the spirit and principles of the present invention should fall within the protection scope of the present invention.

Claims

1. An open vocabulary segmentation method for power distribution network scenarios that integrates multi-fine-grained semantics, characterized in that, include: A multi-view image sequence and camera parameters of a power distribution network scenario are acquired, and a three-dimensional scene representation is constructed based on the multi-view images and camera parameters. The three-dimensional scene representation includes multiple three-dimensional scene units. A two-dimensional candidate mask is generated for the multi-view image, and the open vocabulary semantic features corresponding to the two-dimensional candidate mask are extracted; Based on the camera projection relationship and the 3D visibility relationship, the association weight between the 2D candidate mask and the 3D scene unit is established, and the open vocabulary semantic features of the 2D candidate mask are aggregated into the 3D scene unit according to the association weight; Based on the geometric position, semantic features, and co-occurrence relationship of the 3D scene unit, multiple fine-grained 3D candidates are constructed, and a candidate relationship graph is established with the fine-grained 3D candidates as nodes. According to the association score between candidates in the candidate relationship graph, multi-granularity hierarchical aggregation is performed on the fine-grained 3D candidates to form a multi-level candidate set containing low-level candidates, mid-level candidates, and high-level candidates, and the parent-child relationship between candidates is recorded. The system receives open-vocabulary text queries, parses the text queries into target semantic items, parent subject semantic items, attribute semantic items, and spatial relationship semantic items; determines subject candidates in the multi-level candidate set based on the parent subject semantic items, and scores the target candidates within the constraints of the subject candidates to obtain initial target candidates. When the confidence level of the initial target candidate does not meet the preset condition, cross-level candidate re-retrieval and re-sorting are performed in the multi-level candidate set to obtain the corrected target candidate; Based on the corrected target candidate or the initial target candidate, output the two-dimensional segmentation mask and the three-dimensional target region corresponding to the target.

2. The method according to claim 1, characterized in that, The three-dimensional scene is represented as a three-dimensional Gaussian field, and the three-dimensional scene unit is a three-dimensional Gaussian unit.

3. The method according to claim 2, characterized in that, Each 3D Gaussian element includes: center position Scale vector Rotation parameters The covariance matrix determined by the scale vector and rotation parameters Opacity Color characteristics and semantic features .

4. The method according to claim 1, characterized in that, The two-dimensional candidate mask is generated on images from various viewpoints using a general candidate segmentation model or an instance segmentation model, and the open vocabulary semantic features are extracted using a visual language model.

5. The method according to claim 1, characterized in that, The correlation weight From the 3D scene unit in the first Visibility contribution from each perspective , No. Quality score of a two-dimensional candidate mask and the depth consistency score between 2D candidates and 3D scene units. Jointly determined: 。 6. The method according to claim 1, characterized in that, The aggregation of open-vocabulary semantic features of two-dimensional candidate masks into three-dimensional scene units includes: ， in, For the first From the perspective of the first Semantic features of a two-dimensional candidate mask To prevent small positive numbers with a denominator of zero, This is a normalization operation.

7. The method according to claim 1, characterized in that, The candidate relationship graph is represented as follows: ,in, It is a fine-grained set of 3D candidate nodes. The set of edges between candidate nodes; edge association score Based on the semantic similarity between candidates 3D geometric adjacency Multi-view projection consistency Co-occurrence with 2D masks Weighted average: ， in, , , and These are non-negative weighting coefficients.

8. The method according to claim 1, characterized in that, The multi-granularity hierarchical aggregation includes: taking the fine-grained 3D candidate as the initial node, merging the two candidates into the next layer candidate when the association score between the two candidates meets the aggregation condition and satisfies the spatial connectivity constraint and the candidate area growth constraint; updating the geometric range, semantic features, area or volume, graph edge weights with other candidates and parent-child affiliation relationship of the merged candidate after each merger; stopping the aggregation when the association score is lower than the preset association threshold, or the number of candidate layers reaches the preset number, or there are no candidate pairs that meet the conditions.

9. The method according to claim 1, characterized in that, The parent-child affiliation record is a hierarchical affiliation chain from low-level candidates to direct parent candidates and then to high-level candidates, including: component candidates → direct parent candidates → equipment body candidates.

10. The method according to claim 1, characterized in that, The process of parsing text queries into target semantic items Parent-level main semantic items Attribute semantic items and spatial relation semantic items In this context, non-existent semantic terms are set to empty; the parsing method includes at least one of the following: parsing based on the parent-component vocabulary of the power distribution equipment, rule matching, syntactic analysis, or text encoder.

11. The method according to claim 1, characterized in that, The scoring of target candidates within the constraints of the main candidate includes: for each candidate, calculating a comprehensive score that includes semantic similarity score, parent affiliation consistency score, spatial inclusion score, scale ratio consistency score, cross-view consistency score, and cross-device conflict score.

12. The method according to claim 1, characterized in that, The confidence level not meeting the preset conditions includes at least one of the following: a) there is no valid attribution relationship between the target candidate and the parent subject candidate; b) the area of ​​the target candidate. With parent candidate area The ratio exceeds a reasonable range, where only the lower limit is specified and no upper limit is set, with the lower limit being 0.3; c. The initial target candidates are empty.

13. The method according to claim 1, characterized in that, The cross-level candidate re-retrieval and re-ranking includes: when the confidence of the initial target candidate does not meet the conditions, re-retrieval is performed in the adjacent level candidates of the multi-level candidate set, and the candidates are re-scored and ranked in combination with semantic similarity, parent affiliation consistency, spatial inclusion degree, area ratio and cross-perspective consistency, and the best candidate after re-ranking is used to replace the initial target candidate.

14. An open vocabulary segmentation system for power distribution network scenarios that integrates multi-fine-grained semantics using the method described in any one of claims 1 to 13, characterized in that, include: The image acquisition module is used to acquire multi-view image sequences and camera parameters of power distribution network scenarios; A 3D scene construction module is used to construct a 3D scene representation including multiple 3D scene units based on the multi-view images and camera parameters; a 2D candidate generation module is used to generate 2D candidate masks for each view image and extract open vocabulary semantic features; The semantic aggregation module is used to aggregate two-dimensional candidate semantic features into three-dimensional scene units based on visibility relationships. The multi-granularity candidate construction module is used to construct fine-grained three-dimensional candidates and candidate relationship graphs, and perform multi-granularity hierarchical aggregation to form a multi-level candidate set containing parent-child relationship; The query parsing module is used to parse open vocabulary queries into target semantic items, parent subject semantic items, attribute semantic items, and spatial relationship semantic items; The candidate selection module is used to score and select target candidates within the constraints of the parent subject's candidate selection. The confidence correction module is used to perform cross-level candidate re-retrieval and re-ranking when the initial target candidate confidence is insufficient. The output module is used to output the two-dimensional segmentation mask and the three-dimensional target region corresponding to the target.

15. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images