3DGS equipment monomer method based on dynamic segmentation of SAM2 large model

Abstract

The invention relates to the technical field of computer vision and three-dimensional modeling, and discloses a 3DGS (three-dimensional ground structure) equipment monomer method based on dynamic segmentation of an SAM2 large model, which comprises the following steps of: calling a visual large model to segment a multi-view two-dimensional image to obtain a component contour mask bearing semantic information; the method comprises the following steps of: establishing a mask of a three-dimensional point cloud, establishing space mapping of the mask and the three-dimensional point cloud, executing structure segmentation on the point cloud under double dimensions of fusing geometry and semantics, and further generating a three-dimensional monomer model for each separated equipment component. The segmentation processing of the three-dimensional point cloud is no longer limited to fuzzy geometric inference, but is carried out under clear semantic affiliation, so that the accuracy and robustness of monomer separation in the component dense adjacent region are improved.

Description

Technical Field

[0001] This invention relates to a method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model, belonging to the field of computer vision and 3D modeling technology. Background Technology

[0002] Currently, technicians typically employ either deep learning-based segmentation of multi-view 2D images to reconstruct 3D models or direct analysis and clustering of 3D point cloud data based on pure geometric features. However, when these technologies are applied to real-world, highly complex industrial equipment scenarios, their inherent limitations stemming from the singularity of information modalities become clearly apparent. The core contradiction lies in the fact that 2D image analysis, while possessing high-level semantic understanding capabilities, inherently lacks accurate 3D spatial information, while point cloud data, possessing real 3D spatial information, lacks the ability to semantically recognize the functional components of the equipment. This separation between semantics and geometry often leaves existing technologies in a dilemma when faced with complex situations where different functional components are physically closely fitted or even seamlessly connected.

[0003] Specifically, existing technologies suffer from the following shortcomings: 1. There is a lack of effective linkage mechanism between the semantic understanding of 2D images and the geometric structure of 3D point clouds, making it difficult to balance semantic accuracy and spatial realism in segmentation results; 2. When processing regions where different functional components are physically closely connected, relying solely on information from any one modality easily leads to boundary blurring and segmentation errors, making true individual component separation impossible; 3. Existing segmentation models mostly adopt fixed structures or rules, lacking dynamic adaptability and generalization performance when facing industrial equipment with diverse shapes or complex layouts. Therefore, how to establish a mechanism that can deeply integrate the deep semantic understanding capability of 2D images with the precise spatial representation capability of 3D point clouds, and avoid the inherent limitations of pure geometric segmentation through semantic guidance, thereby achieving automated individual component modeling of equipment in complex industrial scenarios, becomes the technical problem to be solved by this invention. Summary of the Invention

[0004] This invention provides a method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model. Its main purpose is to solve the problem in the prior art that it is difficult to accurately separate complex device components that are physically closely connected due to the separation of two-dimensional semantics and three-dimensional geometric information.

[0005] To achieve the above objectives, this invention provides a method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model. This method includes the following steps: Step a: Obtain two-dimensional images of the industrial equipment to be processed, including multiple viewpoints, as well as the three-dimensional point cloud data of the industrial equipment; call the SAM2 vision large model to segment the two-dimensional images of multiple viewpoints, and generate a two-dimensional segmentation mask containing the contour information of each component inside the industrial equipment. Step b: Establish the spatial projection mapping relationship between the two-dimensional segmentation mask and the three-dimensional point cloud data, and associate the semantic information carried by the two-dimensional segmentation mask with the corresponding three-dimensional point cloud to form a semantic point cloud; Step c: In the semantic point cloud, geometric features and semantic information are fused to perform structural segmentation processing, extract and separate point cloud subsets corresponding to each device component, and generate a three-dimensional single-unit model for each separated point cloud subset.

[0006] Preferably, the step of acquiring two-dimensional images of the industrial equipment to be processed, including multiple viewpoints, specifically includes: in the constructed three-dimensional scene, automatically capturing two-dimensional images from at least the front and back sides of the industrial equipment, with the front and back two-dimensional images together ensuring the complete acquisition of the geometric contour information of the equipment components.

[0007] Preferably, the step of performing structural segmentation further includes: calculating the comprehensive distance between any two points in the semantic point cloud based on a joint semantic and geometric distance metric function, wherein the comprehensive distance is determined by the following formula: in, For point With point The combined distance between them; and Points With point The geometric feature vector contains the normal vector, curvature, and local point density information of the point. and Points With point semantic category; It is a semantic penalty function, which has a value of one when the two points have different semantic categories and a value of zero when they have the same semantic category; and The first and second weighting coefficients are used; and the semantic point cloud is clustered based on the comprehensive distance.

[0008] Preferably, the step of performing structural segmentation further includes determining adaptive clustering parameters before clustering; the step of determining adaptive clustering parameters includes: for each two-dimensional segmentation mask, calculating its pixel area and boundary average gradient density to obtain a quantized structural complexity index; and setting a target number of clusters based on the quantized structural complexity index, so that the point cloud region corresponding to the two-dimensional segmentation mask with a high structural complexity index is assigned a higher target number of clusters.

[0009] Preferably, after calling the SAM2 visual large model, the method further includes: inputting the two-dimensional segmentation mask generated by the SAM2 visual large model into an instance segmentation network for processing, identifying and assigning a semantic category label for each two-dimensional segmentation mask, the semantic category label for the device component includes sleeve, radiator and oil conservator; when establishing the spatial projection mapping relationship, the semantic category label is used as part of the semantic information and associated with the corresponding three-dimensional point cloud.

[0010] Preferably, the step of establishing a spatial projection mapping relationship specifically includes: obtaining the camera's intrinsic parameter matrix and the rigid transformation matrix between their coordinate systems through joint calibration of the camera and the 3D point cloud acquisition device; calculating the projected pixel coordinates of each point in the 3D point cloud on the 2D image using the intrinsic parameter matrix and the rigid transformation matrix; and querying the semantic information of the 2D segmentation mask where the projected pixel coordinates are located based on the projected pixel coordinates, and inversely associating the semantic information with the corresponding 3D point cloud point.

[0011] Preferably, after acquiring the two-dimensional image and the three-dimensional point cloud data, the method further includes a data preprocessing step; the data preprocessing step includes: using a bilateral filtering algorithm to denoise the two-dimensional image; using a cloth simulation filtering algorithm to filter out ground points in the three-dimensional point cloud data; and using a statistical outlier filtering algorithm to remove isolated noise points.

[0012] Preferably, the step of generating a 3D single-unit model for each separated point cloud subset further includes: exporting each generated 3D single-unit model as one of IFC, OBJ or FBX formats; and automatically connecting the exported model to a 3DGS management system to complete the model import and visualization data integration.

[0013] Preferably, after performing structural segmentation processing, the method further includes: constructing each separated point cloud subset as a graph node, and constructing graph edges based on the spatial adjacency relationship and semantic relevance of each point cloud subset to form a structural graph representing the overall assembly logic of the industrial equipment; and using a graph neural network to process the structural graph, inferring and outputting the structural connection relationship and hierarchical relationship between each equipment component to form a structure tree model.

[0014] Preferably, the step of generating a 3D single-unit model for each separated point cloud subset specifically includes: adopting a differentiated modeling strategy based on the semantic category of the equipment component associated with the point cloud subset; when the semantic category of the equipment component is a sleeve, generating a 3D single-unit model by fitting a columnar geometric model to the point cloud; when the semantic category of the equipment component is a radiator, generating a 3D single-unit model by voxelizing the point cloud and reconstructing a closed mesh; and when the semantic category of the equipment component is an oil conservator, generating a 3D single-unit model by interpolation based on a parametric template.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention first generates segmentation masks of component outlines by calling a large visual model on multi-view two-dimensional images, and then projects these two-dimensional masks carrying category information onto three-dimensional point cloud data. This gives reliable semantic assignment to discrete points that originally only had spatial coordinates. This process is not a simple data superposition, but establishes a guiding mapping relationship from macroscopic image semantics to microscopic geometric entities. Based on this, the subsequent structural segmentation processing is no longer a fuzzy inference based solely on the geometric proximity or surface changes of the point cloud, but is transformed into a more explicit and stable target recognition and separation within a semantically pre-divided space. This effectively avoids component adhesion or incorrect segmentation problems caused by a lack of prior knowledge when dealing with complex boundaries such as the connection between the sleeve and the oil tank, where the geometry is closely fitted but the functional attributes are completely different. This helps to improve the structural integrity and boundary accuracy of the final device unitization result.

[0016] 2. In performing structural segmentation processing, this invention introduces a dynamic segmentation method that integrates geometric features and semantic information. It not only considers the spatial distance and morphological similarity between points, but also uses the semantic category of points as a key constraint, guiding the clustering process to mainly occur within semantically consistent regions. At the same time, by dynamically adjusting the fineness of the segmentation in advance based on the contour complexity of the two-dimensional segmentation mask, the method can adapt to the structural differences of different equipment components. This mechanism enables the system to automatically adopt a well-balanced segmentation strategy when facing radiator groups with varying shapes or oil tanks with simple structures. This avoids the waste of computational resources and model redundancy caused by over-segmentation of simple structures, and also realizes the detailed decomposition of complex structures. Thus, while ensuring model accuracy, it improves the universality and processing efficiency of the technical solution for equipment with different shapes in industrial sites.

[0017] 3. After accurately separating the point cloud subsets of each component of the equipment, this invention does not stop at generating a set of discrete single-unit models. Instead, it further constructs each separated point cloud subset as a node of a graph and builds the edges of the graph based on their spatial adjacency and semantic association. Finally, it uses a graph neural network to process the resulting structural graph. This process allows the technical solution to go beyond simple geometric shape restoration and enter into a deep understanding and reconstruction of the internal assembly logic of the equipment. Its output is no longer a collection of scattered components, but a structure tree model that contains the physical connection and hierarchical relationship between the components. This structured data containing assembly knowledge provides a data foundation for more complex applications such as virtual disassembly and assembly, maintenance path planning, or deep integration with the operation and maintenance system. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the system operation of the present invention. Figure 2 This is a comparison chart of the performance indicators of the method of the present invention and the control method in terms of the accuracy of single-unit reconstruction; Figure 3 This is a schematic diagram of the state evolution logic of the data model of the present invention; Figure 4 This is a schematic diagram of the core technology workflow of the present invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] A method for individualizing 3DGS equipment based on dynamic segmentation of a large SAM2 model is proposed. The system operation begins with the acquisition and preprocessing of multimodal data. Then, a large visual model is invoked to perform semantic segmentation in the 2D image domain, establishing a spatial projection mapping between 2D semantic information and 3D point cloud data to form semantic point clouds with semantic attributes. Based on this, the system integrates geometric features and semantic constraints to perform dynamic structural segmentation processing, separating point cloud subsets corresponding to each independent equipment component. Finally, a standardized 3D individual model is generated for each subset, completing the model import and data integration for the 3DGS system. In the construction of 3D geometric systems for industrial equipment, there is a common technical challenge in accurately decomposing the overall equipment model into independent functional components such as sleeves and heat sinks, especially when different functional parts... When components are physically tightly fitted, relying solely on the geometric structure of 3D point clouds for segmentation often results in segmentation errors at component boundaries due to a lack of semantic understanding. To address this challenge, the initial step of this technical solution is configured to acquire 2D images of the industrial equipment to be processed, including multiple viewpoints, as well as the 3D point cloud data of the equipment. Specifically, in the constructed 3D scene, 2D images are automatically captured from at least two perspectives—the front and back—around the target industrial equipment. This multi-view acquisition method aims to ensure increased coverage of the geometric contour information of the equipment components, thereby reducing the impact of occlusion problems from single-view acquisition. After acquiring the raw data, to improve the accuracy of subsequent processing, the system also performs a data preprocessing step. For the 2D images, a bilateral filtering algorithm is used for noise reduction, the process of which is based on the formula... The process involves smoothing noise while preserving critical device edge details; correspondingly, for 3D point cloud data, a cloth simulation filtering algorithm is used to filter out ground points, and a statistical outlier filtering algorithm is used based on judgment conditions. Isolated noise points are removed; thus, the collection and purification of the original multimodal data provides an input source for subsequent semantic-guided segmentation.

[0022] Furthermore, the lack of an effective linkage mechanism between the semantic understanding of 2D images and the geometric structure of 3D point clouds is a core obstacle hindering the accurate segmentation of physically connected regions. Therefore, the system employs a guided mapping procedure from 2D semantics to 3D geometry. First, the SAM2 vision model is used to segment 2D images from multiple viewpoints, generating 2D segmentation masks containing the contour information of equipment components. To ensure these masks possess clear engineering semantics, the system also inputs the generated 2D segmentation masks into an instance segmentation network for processing, identifying and assigning each 2D segmentation mask a semantic category label for an equipment component, such as a sleeve, radiator, or oil conservator. Subsequently, the system performs a deterministic spatial projection mapping establishment step, which obtains the camera's intrinsic parameter matrix through joint calibration of the camera and the 3D point cloud acquisition device. and the rigid transformation matrix between their coordinate systems. And using projection relations The projected pixel coordinates of each coordinate point in the 3D point cloud onto the 2D image are calculated. Then, the semantic information of the 2D segmentation mask where the point is located is queried based on the projected coordinates. This semantic information is then back-linked to the corresponding 3D point cloud point, thus forming a semantic point cloud containing both spatial location and semantic labels. By establishing this guiding mapping relationship from macroscopic image semantics to microscopic geometric entities, discrete points that originally only had spatial coordinates are given semantic attributes. This makes the subsequent structural segmentation process no longer a fuzzy geometric inference, but a stable separation performed under clear semantic pre-division.

[0023] However, when faced with industrial equipment of varying shapes and structural complexities, using fixed segmentation rules or clustering parameters can easily lead to insufficient segmentation of complex structures or over-segmentation of simple structures. To achieve dynamic adaptability, the system performs a structural segmentation process in the semantic point cloud that integrates geometric features and semantic information and possesses adaptive capabilities. This process first calculates the comprehensive distance between any two points in the semantic point cloud using a joint semantic and geometric distance metric function. This comprehensive distance is given by the formula... It is confirmed that, among them, For point With point The combined distance between them and These are the geometric eigenvectors of two points. and The semantic categories of the two points, This is a semantic penalty function; its value is one when two points have different semantic categories and zero when they have the same category. This term ensures that the clustering process mainly takes place within semantically consistent regions. and These are the first and second weighting coefficients. Before clustering, to ensure that the segmentation granularity matches the actual structural complexity of the components, the system also performs a step to determine adaptive clustering parameters. This step, for each two-dimensional segmentation mask, calculates its pixel area. With boundary average gradient density And based on the relation To set the target number of clusters This allows for the setting of a higher target cluster number for structurally complex regions to achieve fine-grained segmentation, while a lower cluster number is set for structurally simple regions to achieve overall extraction. Based on the comprehensive distance and the adaptively determined number of clusters, the system uses a clustering algorithm to process the semantic point cloud, thereby extracting and separating the point cloud subsets corresponding to each device component.

[0024] After accurately separating the point cloud subsets of each component of the equipment, in order to reconstruct the internal assembly logic of the equipment, the system further constructs each separated point cloud subset as a graph node, and constructs the edges of the graph based on the spatial adjacency and semantic relevance of each point cloud subset, forming a structural graph representing the overall assembly logic of the industrial equipment; subsequently, the system uses a graph neural network to process this structural graph, through methods such as... The system uses an information propagation mechanism to infer and output the structural connection and hierarchical relationships between various equipment components, forming a structure tree model. Finally, to ensure that the segmentation results can be directly used by engineering application software, the system generates a 3D single-unit model for each separated point cloud subset. Based on the semantic category of the equipment component associated with the point cloud subset, a differentiated modeling strategy is adopted: when the semantic category is a sleeve, a point cloud fitting columnar geometric model is used; when the semantic category is a radiator, point cloud voxelization and closed mesh reconstruction are used; when the semantic category is an oil conservator, interpolation based on parametric templates is used. The generated 3D single-unit models are finally exported in one of the following formats: IFC, OBJ, or FBX, and automatically connected to a 3DGS management system to complete the model entry and visualization data integration. The output is no longer a collection of scattered components, but a structured data containing the physical connection and hierarchical relationships between the components.

[0025] Example 1: In a single-unit modeling application of a 500kV substation main transformer, the main transformer system has a complex structure, numerous components with varying shapes, and particularly densely arranged radiators that are visually highly coupled with the background environment. Meanwhile, the high- and low-voltage bushings are physically seamlessly connected to the main tank. This presents an objective challenge to traditional modeling methods that rely on a single information modality in accurately identifying component boundaries and effectively deconstructing internal structures. The specific operation of the technical solution of this invention in this scenario begins with acquiring multi-view high-resolution RGB images and full-domain 3D point cloud data covering the main transformer system. The solution does not perform geometric analysis on the 3D point cloud in the initial stage; instead, it first calls the SAM2 visual large model to segment the multi-view images, generating 2D segmentation masks for the outlines of components such as bushings, radiators, oil conservators, and the main tank. This step... The output is a two-dimensional mask with clear component boundaries, providing a fundamental semantic constraint for subsequent structural segmentation processing. The subsequent dynamic segmentation strategy utilizes this semantic constraint to fully realize its effectiveness, forming a synergistic effect between technical features. Furthermore, the system establishes a spatial projection mapping relationship between the two-dimensional segmentation mask and the three-dimensional point cloud data through joint calibration of the camera and LiDAR. The semantic category labels, such as sleeves or heat sinks, assigned by the instance segmentation network are accurately assigned to each corresponding three-dimensional point cloud point, forming a semantic point cloud. This process, by completing the deep fusion of image semantics and point cloud geometry before performing geometric segmentation, solves the problem of semantic and geometric information separation in existing technologies. This allows the system to access the functional attributes and spatial coordinates of any point simultaneously within a single data structure, providing a complete information foundation for subsequent processing.

[0026] Thus, the industry challenge of inferring fuzzy boundaries from massive, undifferentiated point clouds using pure geometric features has been redefined as optimizing and extracting accurate models of each component from a semantic point cloud pre-divided by functional parts; when the system performs structural segmentation processing, the joint semantic and geometric distance metric function it relies on... It utilizes the semantic categories assigned by the preceding steps. and As a strong constraint, at the physical connection between the casing root and the tank surface, even if their geometric features are highly continuous, two points belonging to different categories will still be penalized by the semantic penalty function. The system achieves a significantly increased comprehensive distance through this mechanism, allowing for automatic separation by clustering algorithms. This makes the previously most challenging boundary identification problem of heterogeneous component adhesion regions straightforward and stable under the new processing framework. Simultaneously, for regions with repetitive and complex internal structures, such as heat sink assemblies, the system calculates a high structural complexity index based on the contour features of their two-dimensional segmentation mask, and then uses this index to perform relational calculations. A high number of target clusters is automatically set. This approach enables precise disassembly of each individual heat sink, avoiding under-segmentation caused by using fixed segmentation parameters. Ultimately, the system generates independent, engineering-standard 3D unit models for each separated point cloud subset, including high and low voltage bushings, oil conservators, and each individual heat sink. Using graph neural network processing to generate structural diagrams, the assembly logic relationships between these unit models are reconstructed, realizing digital unit modeling of this complex main transformer system. The overall architecture of this solution is based on first establishing a macroscopic semantic understanding of the scene through a visual model, and then using this understanding to guide and constrain subsequent microscopic geometric processing.

[0027] Example 2: To objectively verify the effectiveness of the present invention's technical solution in the individualization separation of physically tightly connected equipment components, this comparative experiment was established. The purpose of the experiment is to quantitatively evaluate the performance of the present invention's solution in terms of segmentation accuracy and semantic consistency after introducing the semantic information provided by the SAM2 large model as a guiding constraint for segmentation processing, compared with traditional segmentation methods that rely solely on three-dimensional geometric features. The experimental platform was built based on an NVIDIA A100 GPU, and the algorithm was implemented using PyTorch and Open3D libraries. The experimental data came from multi-view images and three-dimensional point cloud acquisitions of a total of ten main transformer devices in three different substations. All point cloud data had been pre-segmented manually using component-level ground truth fine segmentation as a benchmark for evaluating subsequent automated segmentation results.

[0028] A key configuration in the experiment was determining the joint semantic and geometric distance metric function. First weighting coefficient With the second weighting coefficient The core technical trade-off in this setting lies in striking a balance between the confidence level of geometric features and the confidence level of semantic labels. When the value is too high, the segmentation result will rely excessively on geometric continuity, which may lead to the erroneous merging of semantically different but morphologically similar parts. Conversely, when the value is too low, the segmentation result will rely too heavily on geometric continuity. When the value is too high, unexpected breaks may occur within the same geometrically continuous component due to accidental mislabeling by the semantic segmentation network. Therefore, the parameter setting rule is configured to correlate with the average semantic confidence score of the corresponding point cloud region; that is, when the average semantic confidence score of the region is high, the value should be increased. The weights are set to enhance the effectiveness of semantic constraints. On the dataset used in this experiment, due to the good image acquisition quality, the average confidence of the semantic labels output by the instance segmentation network is above 0.9. Therefore, a set of unrestricted configurations is adopted, setting... It is 0.3. It is 0.7.

[0029] The experiment included a control group and an experimental group. The control group used a region growth segmentation algorithm based solely on point cloud geometric features, with growth criteria considering only the angle between normal vectors and curvature differences between adjacent points. The experimental group used the complete technical solution of this invention. Both experiments were run on the same preprocessed dataset, and the segmented point cloud subsets were compared with the ground truth values ​​to calculate the average intersection-union ratio, single-unit reconstruction accuracy, and semantic matching accuracy. During the experiment, a phenomenon was observed: when the control group processed the connection area between the casing and the main tank, the segmentation boundary frequently leaked, resulting in the root of the generated casing single-unit model including part of the tank surface. In contrast, the segmentation boundary of the experimental group in this area was clear and complete. Table 1 shows the key performance index data recorded in a representative experiment for three typical components.

[0030] Table 1: Performance Indicator Comparison Table.

[0031]

[0032] The data in Table 1 shows that the quantitative results of the method of the present invention are higher than those of the control method in all assessment components and evaluation indicators. Among them, the improvement in the accuracy of single-unit reconstruction can be attributed to the distance metric function used in the method of the present invention. Includes semantic penalty terms This is precisely what the pure geometric algorithm in the control group lacks. When dealing with semantically different but physically adjacent point pairs such as casing and oil conservator, this penalty term maximizes their combined distance, thereby constructing an enhanced segmentation boundary at the algorithm level. This mechanism directly suppresses the boundary leakage phenomenon commonly seen in the control group. The experimental data objectively confirms that by establishing a guided mapping from image semantics to spatial geometry, the accuracy and robustness of individualization separation in densely adjacent parts regions can be improved.

[0033] Example 3: This example combines Figures 1 to 3 This section explains the method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model, such as... Figure 1As shown, the process begins with two parallel initial data inputs: multi-view 2D images containing multiple angles such as the front and back, and 3D point cloud data representing the original geometric information of the device. These two inputs enter the data processing stage together, aiming to improve data quality through image denoising and point cloud filtering. After preprocessing, the 2D images are fed into the step of calling the SAM2 visual large model to generate a 2D mask containing the outline of the device parts. This mask is then input into the instance segmentation network processing stage to assign semantic category labels to the parts. At the same time, the system performs a semantic point cloud generation operation, establishing a spatial projection mapping between 2D semantics and 3D point clouds to associate the generated semantic information with the 3D point cloud. Based on the generated semantic point cloud, the system starts two key modules in parallel. The first key module determines adaptive clustering parameters. The module consists of two parts. First, it dynamically sets the target clustering number based on the structural complexity index of a two-dimensional mask to achieve refined segmentation. Second, it integrates geometric and semantic structural segmentation modules, which perform clustering based on a comprehensive distance metric function to separate point cloud subsets of each component. These separated point cloud subsets then enter the 3D unit model generation stage, generating models based on differentiated modeling strategies such as fitting, voxelization, and parametric templates. Subsequently, through model format export and import steps, the models are connected to the 3DGS management system in standard formats such as IFC, OBJ, and FBX, completing model import and visualization data integration. Third, the separated point cloud subsets can selectively enter the overall assembly logic reconstruction stage of key module equipment, using graph neural networks to process the structural diagram, inferring the connections and hierarchical relationships between components, and forming a structure tree model.

[0034] like Figure 2 As shown in the figure, the horizontal axis represents various industrial equipment components, including high-voltage bushings, radiator assemblies, oil conservators, through-wall bushings, switch mechanisms, main oil tanks, terminal blocks, and oil conservators. The vertical axis represents the IoU value, comparing the individual reconstruction accuracy measured by the traditional geometric segmentation method (represented by dashed lines) and the method of this invention (represented by solid lines). The figure clearly shows that for all the tested equipment component types, the IoU value corresponding to the solid lines representing the method of this invention is significantly higher than that of the dashed lines representing the traditional geometric segmentation method. This objectively proves the performance advantage of this invention in the individual segmentation task.

[0035] like Figure 3As shown, the initial state is State 1, raw data, at which point the data only contains discrete geometry and pixel information, with no semantic association. After executing the event of performing 2D semantic extraction and 3D spatial mapping, the system enters State 2, semantic point cloud. In this state, the 3D point cloud has been assigned the semantic affiliation of the 2D image associated through mapping. Based on this, by executing the dynamic structural segmentation event of fusing geometry and semantics, the system evolves to State 3, a set of segmented individual units, characterized by the model being separated into point cloud subsets corresponding to each independent device component. Subsequently, if the event of reconstructing the overall assembly logic of the device is executed, the system finally reaches State 4: structured model. At this point, the model not only contains individual units but also implies the assembly logic and hierarchical relationships between components. Finally, the entire processing flow is completed through the model import and data integration events.

[0036] like Figure 4 As shown, the process begins with multi-view acquisition and point cloud acquisition. After image and point cloud preprocessing, it enters the core steps of SAM2 image segmentation and semantic assignment. Then, semantic point clouds are generated through mask projection. On this basis, dynamic clustering, structural segmentation and object measurement are performed. Furthermore, through BIM modeling, attribute extraction and model export, the technical loop of 3DGS system data entry is finally realized.

[0037] Example 4: After separating the point cloud subsets of the arc-extinguishing chamber, insulation support, and operating mechanism box of a substation circuit breaker, the system faces a new challenge: these discrete component model sets do not contain the physical assembly logic and spatial topological relationships between them. This makes subsequent applications such as virtual maintenance or fault propagation analysis impossible due to the lack of structured data support. To address this challenge, the system executes a deterministic structure diagram construction and reasoning procedure to transform the discrete component point cloud subsets into a structure tree model that carries the equipment assembly logic. This procedure begins by separating each point cloud subset, i.e., the point cloud subset representing the arc-extinguishing chamber. Construct as a graph node All nodes together form the vertex set of the graph. For each node The node attributes are deterministically computed as a set of vectors containing the three-dimensional coordinates of the geometric center of this subset of points, the principal orientation vector determined by the principal axes of its covariance matrix, and the one-hot encoding of its assigned semantic category label.

[0038] Furthermore, the system establishes connections between all nodes to form the edge set of the graph. Thus, an attribute graph is formed. , where any two nodes and one edge between A point cloud can be established if and only if two conditions are met simultaneously: first, the point cloud subsets corresponding to the two nodes are... and The minimum Euclidean distance between them is less than a preset adjacency distance threshold. Secondly, the semantic categories of two nodes are marked as connectable in the system's predefined component adjacency relation matrix; adjacency distance threshold. The settings are based on the general manufacturing tolerances and minimum component gaps of the equipment, which are set to 5 cm in this embodiment; the component adjacency matrix encodes the equipment assembly knowledge, marking the connection between the arc-extinguishing chamber and the insulating support as connectable, while marking the connection between two independent arc-extinguishing chambers as not directly connectable; for each established edge Its edge attributes are calculated as vectors containing the distance between the center points of two nodes and the cosine of the angle between the principal orientation vectors of the two nodes; through the above procedure, the system deterministically transforms a set of discrete point cloud subsets into an attribute graph carrying the topology and spatial relationships of the equipment assembly. This attribute graph is then used as input to a graph neural network, and processed as follows: The information transmission and processing ultimately outputs a structure tree model that clearly defines the connection and hierarchical relationships between the components. This structured model, which contains assembly logic, provides a data foundation with inherent logic for subsequent advanced applications, compared to a set of discrete single-unit models.

[0039] Example 5: Before applying the technical solution of the present invention to a new type of circuit breaker equipment, a component adjacency matrix needs to be constructed first. This procedure loads the 3D BIM model data provided by the manufacturer of the new type of equipment and parses the structure tree of the model in a programming manner to systematically identify all independent components and their semantic categories. Subsequently, the system traverses all component pairs and determines whether there is a physical adjacency relationship by calculating the minimum Euclidean distance between the 3D models of each pair of components. When the minimum distance between two components is less than the preset manufacturing tolerance threshold, the entries corresponding to the semantic categories of the two components are set to a connectable state in the component adjacency matrix. This generates a component adjacency matrix that can objectively reflect the inherent assembly logic of the new type of equipment and can be called in the subsequent structural drawing construction stage.

[0040] Before performing data acquisition on specific industrial equipment on-site, the system first executes a joint calibration procedure for the camera and the 3D point cloud acquisition device. This procedure involves placing a checkerboard calibration board of known size near the device under test, and simultaneously acquiring 2D images and 3D point clouds of the calibration board from the camera and the 3D point cloud acquisition device. The system automatically extracts the pixel coordinates of the checkerboard corner points from the 2D image and the 3D spatial coordinates of the calibration board plane from the 3D point cloud. Finally, an optimization algorithm is used to solve for the camera intrinsic parameter matrix and rigid transformation matrix that minimize the reprojection error of the 3D points on the 2D image. The output of this procedure is a set of camera intrinsic parameter matrices and rigid transformation matrices that reflect the relative relationship between the coordinate systems of the two sensors at the current deployment location, providing a data foundation for subsequently establishing the spatial projection mapping relationship between the 2D segmentation mask and the 3D point cloud data.

[0041] Example 6: In the initial stage of deploying the technical solution of this invention in a specific substation scenario, to ensure the accuracy and robustness of model generation, the system executes a standardized pre-parameter calibration and model verification procedure. This procedure first calibrates the geometric processing parameters of the point cloud data, and analyzes the voxel size of the voxel downsampling by analyzing the sample point cloud collected on site. The value of is set based on the minimum structural feature size that needs to be retained in the target scene, and its value does not exceed half of that minimum feature size. At the same time, the number of neighborhood points required by the outlier filtering algorithm is counted. With distance threshold coefficient The noise reduction parameters are determined by scanning a standard planar target and using the measured average density of the point cloud and the standard deviation of the noise distribution, thus matching the actual performance of the field acquisition equipment.

[0042] This procedure then verifies and strengthens the adaptability of the semantic recognition model. Operators collect a small number of 2D images containing typical on-site equipment under different lighting and angles, and input them into a pre-trained instance segmentation network. The initial semantic labels and boundary masks output by the network are manually reviewed and corrected by the operators. This corrected data with high-confidence annotations is then used for targeted incremental training or fine-tuning of the network to improve its ability to recognize the unique equipment morphology and environmental features of this site. In the subsequent automated processing, the system also has a built-in fault-tolerant processing logic, that is, when the semantic category confidence score output by the instance segmentation network for a certain image region is lower than a preset confidence threshold. When this happens, the system will abandon applying semantic constraints to the point cloud corresponding to that region and instead use a segmentation algorithm based on pure geometric features for processing, and mark the region for subsequent manual review.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model, characterized in that, The method includes the following steps: Step a: Obtain two-dimensional images of the industrial equipment to be processed, including multiple viewpoints, as well as the three-dimensional point cloud data of the industrial equipment; call the SAM2 vision large model to segment the two-dimensional images of multiple viewpoints, and generate a two-dimensional segmentation mask containing the contour information of each component inside the industrial equipment. Step b: Establish the spatial projection mapping relationship between the two-dimensional segmentation mask and the three-dimensional point cloud data, and associate the semantic information carried by the two-dimensional segmentation mask with the corresponding three-dimensional point cloud to form a semantic point cloud; Step c: In the semantic point cloud, geometric features and semantic information are fused to perform structural segmentation processing, extract and separate point cloud subsets corresponding to each device component, and generate a three-dimensional single-unit model for each separated point cloud subset.

2. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, The steps for acquiring two-dimensional images of the industrial equipment to be processed, including multiple perspective views, specifically include: in the constructed three-dimensional scene, automatically capturing two-dimensional images from at least the front and back sides of the industrial equipment, with the front and back two-dimensional images together ensuring the complete acquisition of geometric contour information of the equipment components.

3. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, The step of performing structural segmentation further includes: for any two points in the semantic point cloud, calculating their combined distance based on a joint semantic and geometric distance metric function, wherein the combined distance is determined by the following formula: in, For point With point The combined distance between them; and Points With point The geometric feature vector contains the normal vector, curvature, and local point density information of the point. and Points With point semantic category; It is a semantic penalty function, which has a value of one when the two points have different semantic categories and a value of zero when they have the same semantic category; and The first and second weighting coefficients are used; and the semantic point cloud is clustered based on the comprehensive distance.

4. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 3, characterized in that, The steps of performing structural segmentation processing also include determining adaptive clustering parameters before clustering; the steps of determining adaptive clustering parameters include: for each two-dimensional segmentation mask, calculating its pixel area and boundary average gradient density to obtain a quantized structural complexity index; and setting the target number of clusters based on the quantized structural complexity index, so that the point cloud region corresponding to the two-dimensional segmentation mask with a high structural complexity index is assigned a higher target number of clusters.

5. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, After calling the SAM2 visual large model, the method also includes: inputting the two-dimensional segmentation mask generated by the SAM2 visual large model into an instance segmentation network for processing, identifying and assigning a semantic category label for each two-dimensional segmentation mask, including sleeves, radiators and oil conservators; when establishing spatial projection mapping relationship, the semantic category label is used as part of the semantic information and associated with the corresponding three-dimensional point cloud.

6. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, The steps for establishing a spatial projection mapping relationship specifically include: obtaining the camera's intrinsic parameter matrix and the rigid transformation matrix between their coordinate systems through joint calibration of the camera and the 3D point cloud acquisition device; calculating the projected pixel coordinates of each point in the 3D point cloud on the 2D image using the intrinsic parameter matrix and the rigid transformation matrix; and querying the semantic information of the 2D segmentation mask where the projected pixel coordinates are located based on the projected pixel coordinates, and then inversely associating the semantic information with the corresponding 3D point cloud point.

7. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, After acquiring two-dimensional images and three-dimensional point cloud data, the method also includes a data preprocessing step; The data preprocessing steps include: using a bilateral filtering algorithm to denoise the two-dimensional image; Furthermore, a cloth-simulation filtering algorithm is used to filter out ground points in the 3D point cloud data, and a statistical outlier filtering algorithm is used to remove isolated noise points.

8. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, The step of generating a 3D unit model for each separated point cloud subset further includes: exporting each generated 3D unit model as one of IFC, OBJ or FBX formats; and automatically connecting the exported model to a 3DGS management system to complete the model import and visualization data integration.

9. The method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, After performing structural segmentation, the method further includes: constructing each separated point cloud subset as a graph node, and constructing graph edges based on the spatial adjacency and semantic relevance of each point cloud subset to form a structural graph representing the overall assembly logic of industrial equipment; and using a graph neural network to process the structural graph, inferring and outputting the structural connection relationships and hierarchical relationships between each equipment component to form a structure tree model.

10. A method for individualizing 3DGS devices based on dynamic segmentation of the SAM2 large model according to claim 1, characterized in that, The steps for generating a 3D single-unit model for each separated point cloud subset specifically include: adopting a differentiated modeling strategy based on the semantic category of the equipment component associated with the point cloud subset; when the semantic category of the equipment component is a sleeve, generating a 3D single-unit model by fitting a columnar geometric model to the point cloud; when the semantic category of the equipment component is a radiator, generating a 3D single-unit model by voxelizing the point cloud and reconstructing a closed mesh; and when the semantic category of the equipment component is an oil conservator, generating a 3D single-unit model by interpolation based on a parametric template.

Images