Image point cloud segmentation method and device for substation scene, equipment and storage medium

Abstract

The invention discloses an image point cloud segmentation method, device and equipment for a transformer substation scene and a storage medium, and the method comprises the steps: obtaining a real-time two-dimensional image and a real-time three-dimensional point cloud of a transformer substation, inputting the real-time two-dimensional image and the real-time three-dimensional point cloud into a transformer substation image point cloud segmentation model, and outputting a transformer substation scene segmentation result, the transformer substation image point cloud segmentation model comprises a pseudo label prediction flow module and an original flow module, and the two modules both comprise bimodal fusion modules. Therefore, the transformer substation image point cloud segmentation model can fuse the information under the two modes through the bimodal fusion module, and can adaptively learn the complementary information between the image and the point cloud in the training process, so that the model can realize advantage complementation and deep fusion of the features of the two modes; therefore, the segmentation efficiency of the substation scene is improved.

Description

Technical Field

[0001] This application relates to the field of power equipment images, and more specifically, to a method, apparatus, device, and storage medium for image point cloud segmentation of a substation scene. Background Technology

[0002] In the field of intelligent substation inspection, accurate segmentation of key targets such as equipment and fittings is the core foundation for condition monitoring and digital twin construction. In traditional single-modal supervised learning methods, although visible light images contain rich texture information, they lack robustness in complex lighting and occlusion scenarios and lack three-dimensional geometric structure; LiDAR point clouds can provide accurate distance and geometry, but lack high-frequency texture and color features, making it difficult to distinguish between similar-looking devices.

[0003] Existing image segmentation methods can only provide two-dimensional pixel-level labels and cannot obtain the three-dimensional structure of the device; point cloud segmentation methods require complex preprocessing, are inefficient and cannot meet real-time requirements.

[0004] Therefore, how to combine the advantages of images and point clouds to segment substation scenes and improve the segmentation efficiency of substation scenes is an issue that needs attention. Summary of the Invention

[0005] In view of the above problems, this application provides an image point cloud segmentation method, apparatus, device and storage medium for substation scenes to improve the segmentation efficiency of substation scenes.

[0006] To achieve the above objectives, the following specific solutions are proposed:

[0007] A method for segmenting point clouds in an image of a substation scene includes:

[0008] Acquire real-time 2D images and real-time 3D point clouds of the substation;

[0009] The real-time 2D image and the real-time 3D point cloud are input into a pre-trained substation image point cloud segmentation model, and the substation scene segmentation result is output.

[0010] The training process of the substation image point cloud segmentation model includes:

[0011] Acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0012] Establish a pairing association between the pixels of the two-dimensional image and the points of the three-dimensional point cloud;

[0013] The substation equipment point clusters and their paired associated two-dimensional images are input into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and the substation structure point clusters and their paired associated two-dimensional images are input into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0014] After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained.

[0015] Optionally, the pseudo-label prediction stream module has the same architecture as the original stream module, both including a 3D encoder, a 3D decoder, a 2D encoder, a 2D decoder, and several dual-modal fusion sub-modules.

[0016] The 3D encoder includes a 3×3 core 3D convolutional layer, a batch normalization layer, an activation layer, and a 2×2×2 core max pooling layer;

[0017] The 3D decoder includes a 2×2×2 kernel convolutional layer, a first feature splicing submodule, a 3D convolutional layer, a batch normalization layer, and an activation layer.

[0018] The 2D encoder includes a deep convolutional network, a dilated convolutional layer, a dilated spatial pyramid pooling layer, and a 1×1 kernel 2D convolutional layer;

[0019] The 2D decoder includes a first upsampling submodule, a 1×1 kernel 2D convolutional layer, a second feature splicing submodule, a 3×3 kernel 2D convolutional layer, and a second upsampling submodule.

[0020] The dual-modal fusion submodule is used to fuse the latent features of point clouds and images at each level of the 3D decoder and the 2D decoder to obtain three-dimensional fused features.

[0021] Optionally, the dual-modal fusion submodule is used for:

[0022] The 3D features output by the 3D decoder are subjected to Minkowski convolution to obtain 3D key features and 3D value features, and the 2D features output by the 2D decoder are subjected to standard convolution to obtain 2D query features.

[0023] The three-dimensional key features and the two-dimensional query features are sequentially subjected to dot product, summation and first-level softmax operations to generate a three-dimensional attention map;

[0024] The multi-head attention feature is obtained by weighted summation of the three-dimensional attention map and the three-dimensional value feature;

[0025] The multi-head attention feature is extended to the same dimension as the three-dimensional feature and then concatenated with the three-dimensional feature to obtain a three-dimensional concatenated feature.

[0026] The three-dimensional stitched features are subjected to Minkowski convolution to obtain the three-dimensional fused features.

[0027] Optionally, the loss function for training the substation image point cloud segmentation model is:

[0028]

[0029] in, To train the loss term, This refers to the i-th point-level semantic feature output by the substation image point cloud segmentation model during training. For pixel-level semantic features that are paired and associated with the i-th point-level semantic feature, N is the total number of pairings between the pixels of the two-dimensional image and the points of the three-dimensional point cloud.

[0030] Optionally, the pseudo-label prediction stream module is used for:

[0031] During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the substation structural point clusters to obtain coarse-grained three-dimensional pseudo-labels.

[0032] During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the two-dimensional images that are paired and associated with the substation structural point clusters, resulting in coarse-grained two-dimensional pseudo-labels.

[0033] The substation image point cloud segmentation model also includes a pseudo-label optimization module, which is used for:

[0034] Obtain the substation structural point cluster with coarse-grained three-dimensional pseudo-labels, and the two-dimensional image with coarse-grained two-dimensional pseudo-labels paired and associated with the substation structural point cluster.

[0035] The coarse-grained two-dimensional pseudo-label is projected onto the target matching point to obtain the projected three-dimensional pseudo-label.

[0036] The coarse-grained 3D pseudo-labels are optimized using the first label optimization formula, which is:

[0037] ;

[0038] in, The substation structural point cluster, To optimize 3D pseudo-labels, The coarse-grained three-dimensional pseudo-label, The projected 3D pseudo-label, For confidence function, The confidence threshold;

[0039] The coarse-grained 3D pseudo-labels are projected onto the target image plane to extract the target region composed of pixels not projected onto the target image plane. The target region is then averaged and pooled to output the dense 2D pseudo-labels with the maximum dimension.

[0040] The coarse-grained two-dimensional pseudo-labels are optimized using the second label optimization formula, which is as follows:

[0041]

[0042] in, A two-dimensional image associated with the structural point cluster of the substation. To optimize 2D pseudo-labels, The coarse-grained two-dimensional pseudo-label, These are dense two-dimensional pseudo-labels.

[0043] Optionally, the parameter update formula for the pseudo-label prediction stream module is:

[0044]

[0045] in, These are the module parameters for the pseudo-label prediction stream module, and the initial values ​​of these module parameters are determined based on the original stream module. Update the weight coefficients for the parameters. Here, s represents the updated value of the module parameters, and s represents the training step number of the original stream module. The parameters of the original stream module after being updated in the s-th training step. This is a preset threshold.

[0046] Optionally, after acquiring the two-dimensional image and three-dimensional point cloud of the substation scene, the method further includes:

[0047] The two-dimensional image and the three-dimensional point cloud are optimized to obtain the optimized two-dimensional image and the three-dimensional point cloud.

[0048] An image point cloud segmentation device for a substation scene includes:

[0049] The real-time image point cloud information acquisition unit is used to acquire real-time two-dimensional images and real-time three-dimensional point clouds of the substation.

[0050] The model segmentation unit is used to input the real-time two-dimensional image and the real-time three-dimensional point cloud into a pre-trained substation image point cloud segmentation model and output the substation scene segmentation result.

[0051] The training data acquisition unit is used to acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0052] A pairing association establishment unit is used to establish pairing associations between pixels of the two-dimensional image and points of the three-dimensional point cloud;

[0053] The dual-stream module training unit is used to input the substation equipment point clusters and their paired associated two-dimensional images into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and to input the substation structure point clusters and their paired associated two-dimensional images into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0054] The training unit is used to obtain the trained point cloud segmentation model of the substation image after both the pseudo-label prediction stream module and the original stream module have been trained.

[0055] An image point cloud segmentation device for a substation scene includes a memory and a processor;

[0056] The memory is used to store programs;

[0057] The processor is used to execute the program to implement the various steps of the image point cloud segmentation method for substation scenes as described above.

[0058] A storage medium storing a computer program, which, when executed by a processor, implements the various steps of the image point cloud segmentation method for a substation scene as described above.

[0059] Using the above technical solution, this application acquires real-time two-dimensional images and real-time three-dimensional point clouds of a substation, inputs the real-time two-dimensional images and real-time three-dimensional point clouds into a pre-trained substation image point cloud segmentation model, and outputs substation scene segmentation results. The training process of the substation image point cloud segmentation model is as follows: acquire two-dimensional images and three-dimensional point clouds of the substation scene, establish pairing associations between pixels in the two-dimensional images and points in the three-dimensional point clouds, input substation equipment point clusters and their paired associated two-dimensional images into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and input substation structure point clusters and their paired associated two-dimensional images into the original stream module of the substation image point cloud segmentation model to train the original stream module. After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained. As can be seen, the substation image point cloud segmentation model has a pseudo-label prediction flow module and a raw flow module. Both modules contain a dual-modal fusion module, which can fuse information from the two modalities. During training, it can adaptively learn the complementary information between the image and the point cloud, enabling the model to achieve complementary advantages and deep fusion of the two modal features, thereby improving the segmentation efficiency of the substation scene. Attached Figure Description

[0060] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0061] Figure 1 A schematic diagram illustrating a process for implementing image point cloud segmentation in a substation scene, provided in an embodiment of this application;

[0062] Figure 2(a) is a schematic diagram of a processing flow of the original stream module provided in an embodiment of this application;

[0063] Figure 2(b) is a schematic diagram of a processing flow of the pseudo-label prediction stream module provided in an embodiment of this application;

[0064] Figure 3 An architecture diagram of a pseudo-label prediction stream module or a raw stream module provided in an embodiment of this application;

[0065] Figure 4 A flowchart illustrating the calculation of a three-dimensional fusion feature of a dual-modal fusion module provided in this application embodiment;

[0066] Figure 5 A schematic diagram of a processing flow for joint processing of a 3D encoder and a 3D decoder provided in an embodiment of this application;

[0067] Figure 6 A schematic diagram of a processing flow for joint processing of a 2D encoder and a 2D decoder provided in an embodiment of this application;

[0068] Figure 7 A schematic diagram of a device structure for implementing image point cloud segmentation in a substation scene, provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of an image point cloud segmentation device for a substation scene provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a computer, cloud, or server.

[0072] Next, combined Figure 1 The image point cloud segmentation method for substation scenes described in this application may include the following steps:

[0073] Step S110: Obtain real-time two-dimensional images and real-time three-dimensional point clouds of the substation.

[0074] Specifically, ground-based fixed cameras and mobile robots equipped with cameras and lidar can be deployed within the substation. Combined with cameras and lidar carried by drones, high-resolution real-time two-dimensional images and real-time three-dimensional point clouds covering the main areas and key equipment of the substation can be systematically collected.

[0075] Step S120: Input the real-time 2D image and real-time 3D point cloud into the pre-trained substation image point cloud segmentation model, and output the substation scene segmentation result.

[0076] Specifically, before inputting real-time 2D images and real-time 3D point clouds into a pre-trained substation image point cloud segmentation model, the real-time 2D images and real-time 3D point clouds can be optimized, such as removing out-of-focus, severely motion-blurred, or poorly lit invalid images from the real-time 2D images; and performing spatial denoising and filtering on the real-time 3D point clouds.

[0077] The training process for the substation image point cloud segmentation model may include:

[0078] S121. Obtain two-dimensional images and three-dimensional point clouds of the substation scene.

[0079] The 3D point cloud can include labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0080] Specifically, ground-based fixed cameras and mobile robots equipped with cameras and LiDAR are deployed within the substation. Combined with cameras and LiDAR carried by drones, this systematically collects high-resolution two-dimensional images and three-dimensional point cloud spatial information covering the main areas and key equipment of the substation. The ground-based perspective focuses on equipment details and near-ground structures, while the drone perspective emphasizes acquiring images and point clouds of hard-to-access areas such as roof structures and overhead busbars, forming a comprehensive multi-scale, multi-angle coverage of the complex substation environment.

[0081] Furthermore, considering that the original image data of the substation may be out of focus, blurred due to severe motion, or have poor lighting conditions, it is necessary to optimize the two-dimensional image and the three-dimensional point cloud to obtain optimized two-dimensional image and three-dimensional point cloud.

[0082] Specifically, for 2D image optimization, sharpness assessment and filtering can be performed to remove invalid images that are out of focus, severely motion-blurred, or have extremely poor lighting conditions, retaining valid image datasets that clearly show the details of equipment and structural boundaries. For 3D point cloud optimization, aerial triangulation can be performed on UAV oblique images. By constructing a geometric constraint network between images, the camera orientation parameters and the 3D coordinates of the target area are simultaneously optimized to generate high-precision digital surface models and digital elevation models. Based on multi-view image feature matching, dense matching technology is used to obtain corresponding points, thereby producing a high-density color 3D point cloud. Subsequently, spatial denoising is performed on the generated initial point cloud data: after loading the point cloud and completing the pre-cleaning of invalid points, a coordinate axis (such as the Z-axis) is selected according to the actual scene requirements, and a reasonable threshold range is set to define the retention area; a pass-through filtering algorithm is called, input parameters are used to perform filtering, and discrete noise points outside the set range are removed; if stricter spatial constraints are required, this threshold filtering process can be repeated sequentially on other coordinate axes (such as the X and Y axes), and finally, the optimized point cloud data after spatial range filtering is output.

[0083] Understandably, to support the subsequent training of the substation image point cloud segmentation model, it is necessary to perform necessary fine-grained annotation on a small number of samples in the optimized substation point cloud dataset. This mainly involves dividing the equipment point cloud into equipment point clusters, i.e., assigning predefined category labels to the points (including equipment categories such as transformers, circuit breakers, insulators, and busbars, as well as background categories). Substation structures do not require manual annotation; they will be automatically generated in the image labels through point cloud label projection and feature alignment loss, significantly reducing the annotation cost required for substation scene segmentation tasks.

[0084] S122. Establish the pairing relationship between the pixels of the two-dimensional image and the points of the three-dimensional point cloud.

[0085] Specifically, since 3D point clouds contain spatial coordinate information and 2D images are in RGB format, the two can establish a precise point-pixel pairing relationship (i.e., each 3D point p) through pre-calibrated sensor intrinsic and extrinsic parameters. i With the corresponding image pixel x i (Related).

[0086] S123. Input the two-dimensional images of substation equipment point clusters and their paired associations into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and input the two-dimensional images of substation structure point clusters and their paired associations into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0087] The original stream module is used to simultaneously segment the point cloud and the image, while the pseudo-label prediction stream module is used to predict pseudo-labels for the unlabeled point cloud and its corresponding image, so as to perform self-training in the original stream. The specific training process of the original stream module is shown in Figure 2(a), and the specific training process of the pseudo-label prediction stream module is shown in Figure 2(b).

[0088] Specifically, the training data input to the substation image point cloud segmentation model can be divided into two categories: the first category can be labeled point clouds and their paired images, and the second category can be unlabeled point clouds and their paired images. The substation image point cloud segmentation model can include a raw stream module and a pseudo-label prediction stream module. Unlabeled point clouds and their paired images are processed by the raw stream module, while labeled point clouds and their paired images are processed by the pseudo-label prediction stream module. Within each module, the point cloud input 3D encoder-decoder branch can extract geometric features, and the image input 2D encoder-decoder branch can extract texture features.

[0089] S124. After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained.

[0090] Specifically, after completing the training of the substation image point cloud segmentation model, the real-time two-dimensional image and real-time three-dimensional point cloud of the substation can be input into the pre-trained substation image point cloud segmentation model to output the substation scene segmentation result.

[0091] More specifically, the substation image point cloud segmentation model can input the substation scene point cloud data to be segmented and its strictly registered corresponding image data into the original stream model that has been trained and converged. The model processes the point cloud and image through its integrated 3D and 2D branches respectively, and uses a dual-modal fusion module to dynamically aggregate the geometric structural features and texture semantic information of both. Finally, the model outputs 3D point cloud segmentation results carrying accurate semantic labels (including equipment categories such as transformers and insulators), clearly identifying the 3D spatial location, morphological structure, and semantic category of each key device in the substation scene, providing directly usable structured semantic information for subsequent equipment status analysis, 3D digital twin model construction, and intelligent inspection.

[0092] The substation scene image point cloud segmentation method provided in this embodiment acquires real-time two-dimensional images and real-time three-dimensional point clouds of the substation, inputs the real-time two-dimensional images and real-time three-dimensional point clouds into a pre-trained substation image point cloud segmentation model, and outputs the substation scene segmentation result. The training process of the substation image point cloud segmentation model is as follows: acquire two-dimensional images and three-dimensional point clouds of the substation scene, establish pairing associations between pixels in the two-dimensional images and points in the three-dimensional point clouds, input substation equipment point clusters and their paired associated two-dimensional images into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and input substation structure point clusters and their paired associated two-dimensional images into the original stream module of the substation image point cloud segmentation model to train the original stream module. After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained. As can be seen, the substation image point cloud segmentation model has a pseudo-label prediction flow module and a raw flow module. Both modules contain a dual-modal fusion module, which can fuse information from the two modalities. During training, it can adaptively learn the complementary information between the image and the point cloud, enabling the model to achieve complementary advantages and deep fusion of the two modal features, thereby improving the segmentation efficiency of the substation scene.

[0093] In some embodiments of this application, the pseudo-label prediction stream module and the original stream module mentioned in the above embodiments are further described. Specifically, the pseudo-label prediction stream module and the original stream module have the same architecture, as shown in the reference... Figure 3 This architecture can include a 3D encoder, a 3D decoder, a 2D encoder, a 2D decoder, and several bimodal fusion submodules. Multiple bimodal fusion submodules can be connected between the 2D decoder and the 3D decoder.

[0094] The 3D encoder can include a 3D convolutional layer with 3×3 kernels, a batch normalization layer, an activation layer, and a max pooling layer with 2×2×2 kernels. The 3D decoder can include a convolutional layer with 2×2×2 kernels, a first feature stitching submodule, a 3D convolutional layer, a batch normalization layer, and an activation layer.

[0095] For specific details on the joint processing flow of the 3D encoder and 3D decoder, please refer to [link / reference]. Figure 4 The 3D encoder performs repeated 3D convolutions using 3×3 kernel 3D convolutional layers, extracts features through batch normalization layers and ReLU activation layers, and gradually reduces the spatial size using 2×2×2 kernel max pooling layers while doubling the number of channels to form high-dimensional abstract features. The 3D decoder gradually restores the original size through 2×2×2 kernel convolutional layers, and after upsampling at each level, it concatenates the features of the encoder at the same level through the first feature concatenation submodule to fuse detailed information. Then, it refines the features and reduces the number of channels through 3D convolutional layers + batch normalization layers + ReLU activation layers, finally reconstructing a high-resolution feature map.

[0096] A 2D encoder may include a deep convolutional network, dilated convolutional layers, dilated spatial pyramid pooling layers, and 1×1 kernel 2D convolutional layers. A 2D decoder may include a first upsampling submodule, a 1×1 kernel 2D convolutional layer, a second feature concatenation submodule, a 3×3 kernel 2D convolutional layer, and a second upsampling submodule.

[0097] The workflow for joint processing of 2D encoders and 2D decoders can be found in [reference]. Figure 5 The 2D encoder progressively extracts multi-scale features from the input image. First, it extracts basic features using a deep convolutional network (DCNN), then expands the receptive field through atrous convolutions, followed by spatial pyramid pooling (ASPP) layers to fuse multi-scale contextual information. Finally, a 1×1 kernel 2D convolutional layer compresses the feature channels. The 2D decoder (lower branch) performs feature restoration and refinement. First, the 2D encoder output is upsampled by 4x using a first upsampling submodule. After processing to the aligned channels using a 1×1 kernel 2D convolutional layer, it is concatenated with intermediate features from the 2D encoder using a two-feature concatenation submodule to fuse high- and low-level semantics. Then, a 3×3 kernel 2D convolutional layer optimizes the feature representation. Finally, a second upsampling submodule performs a second 4x upsampling to restore the original image resolution for output prediction. The entire process achieves a complete segmentation workflow from image feature extraction and multi-scale information fusion to spatial resolution restoration, taking into account both global context awareness and local detail reconstruction.

[0098] The dual-modal fusion submodule can be used at each level of the 3D decoder and 2D decoder to fuse the latent features of the point cloud with the image to obtain 3D fused features.

[0099] For details, please refer to Figure 6 The dual-modal fusion submodule can be used for:

[0100] S1. Perform Minkowski convolution on the 3D features output by the 3D decoder to obtain the 3D key features. and three-dimensional value features The two-dimensional features output by the 2D decoder are then subjected to standard convolution to obtain the two-dimensional query features. .

[0101] S2, Key 3D Features and two-dimensional query features The three-dimensional attention map is generated by sequentially performing dot product, summation, and first-level softmax operations. .

[0102] S3, 3D Attention Map and three-dimensional value features We perform a weighted summation to obtain the multi-head attention features.

[0103] S4. Extend multi-head attention features to 3D features. Same dimension, and with three-dimensional features By splicing, three-dimensional splicing features are obtained.

[0104] S5. Perform Minkowski convolution on the 3D stitched features to obtain the 3D fused features. .

[0105] Specifically, 3D fusion features The calculation process can be summarized as follows:

[0106]

[0107] in, Minkowski convolution is represented. This represents the dot product operation. This indicates a feature splicing operation.

[0108] In addition, two-dimensional fusion features The calculation process can be summarized as follows:

[0109]

[0110] in, Represents the convolution function. , Two-dimensional key features, This is a three-dimensional query feature.

[0111] Understandably, compared to direct feature concatenation, the multi-head attention-based fusion mechanism can extract more discriminative and information-dense features from bimodal data.

[0112] Considering the consistency between the bimodal features learned in the constrained original stream output feature space, some embodiments of this application introduce the loss function during training of the substation image point cloud segmentation model mentioned in the above embodiments. This loss function can be:

[0113]

[0114] in, To train the loss term, This represents the i-th point-level semantic feature output by the substation image point cloud segmentation model during training. For each pixel-level semantic feature paired with the i-th point-level semantic feature, N represents the total number of pairings between pixels in the 2D image and points in the 3D point cloud. This represents the L2 norm.

[0115] Considering the inherent limitations of the two modalities (i.e., the lack of texture information in the point cloud and the lack of depth information in the image), the 3D and 2D encoder-decoder branches predict the corresponding pseudo-labels based on the geometric structure and texture features of the target. Based on this, some embodiments of this application introduce the label optimization method of the above-mentioned substation image point cloud segmentation model during training.

[0116] During the training process of the pseudo-label prediction stream module, it can predict pseudo-labels for substation structural point clusters, obtaining coarse-grained 3D pseudo-labels. During the training process of the pseudo-label prediction stream module, it can also predict pseudo-labels for 2D images paired and associated with substation structural point clusters, obtaining coarse-grained 2D pseudo-labels.

[0117] Understandably, the 3D encoder-decoder branch and the 2D encoder-decoder branch of the pseudo-label prediction stream module will predict the corresponding pseudo-labels based on the target's geometric structure and texture features.

[0118] Building upon this, the substation image point cloud segmentation model can also include a pseudo-label optimization module. To ensure the effectiveness of self-training of unlabeled data in the original stream, a voting-based pseudo-label optimization module is adopted to improve the reliability of pseudo-labels. The pseudo-label optimization module only processes paired unlabeled points and pixels, and its coordinate set can be represented as... , This represents the number of unlabeled matching pairs. The pseudo-label optimization module takes the coarse pseudo-labels of unlabeled points and pixels as input and outputs their corresponding optimized pseudo-labels.

[0119] Specifically, the pseudo-tag optimization module can be used for:

[0120] S1. Obtain the substation structural point cluster with coarse-grained three-dimensional pseudo-labels, and the two-dimensional image with coarse-grained two-dimensional pseudo-labels paired and associated with the substation structural point cluster.

[0121] S2. Project the coarse-grained two-dimensional pseudo-labels onto the target matching points to obtain projected three-dimensional pseudo-labels.

[0122] S3. Optimize coarse-grained 3D pseudo-labels using the first label optimization formula.

[0123] The optimization formula for the first label is:

[0124] ;

[0125] in, For the substation structural point cluster, To optimize 3D pseudo-labels, This is a coarse-grained 3D pseudo-label. For projecting 3D pseudo-labels, For confidence function, This is the confidence threshold.

[0126] S4. Project the coarse-grained 3D pseudo-labels onto the target image plane to extract the target region composed of pixels not projected onto the target image plane. Perform average pooling on the target region to output the dense 2D pseudo-labels with the maximum dimension.

[0127] S5. Optimize coarse-grained two-dimensional pseudo-labels using the second label optimization formula.

[0128] The optimization formula for the second label is as follows:

[0129]

[0130] in, Two-dimensional images that are paired and associated with the structural point clusters of the substation. To optimize 2D pseudo-labels, This is a coarse-grained two-dimensional pseudo-label. These are dense two-dimensional pseudo-labels.

[0131] Furthermore, the optimized pseudo-labels output by the pseudo-label optimization module can be used for supervised model training. Specifically, a teacher-student model mechanism can be adopted, where the optimized pseudo-labels supervise the training of the student model, the student parameters are updated by calculating the unsupervised loss, and the teacher model is improved simultaneously using an exponential moving average mechanism.

[0132] Understandably, the exponential moving average mechanism can retain historical information through a gradual update strategy, thereby mitigating the negative impact of erroneous false labels.

[0133] Based on this, the parameter update formula for the pseudo-label prediction stream module can be:

[0134]

[0135] in, The module parameters for the pseudo-label prediction stream module can be initially determined based on the original stream module. Update the weight coefficients for the parameters. Here, represents the updated values ​​for the module parameters, and s represents the training step number of the original stream module. These are the parameters of the original streaming module after the s-th training step. This is a preset threshold.

[0136] The following describes the apparatus for segmenting image point clouds of a substation scene provided in the embodiments of this application. The apparatus for segmenting image point clouds of a substation scene described below can be referred to in correspondence with the method for segmenting image point clouds of a substation scene described above.

[0137] See Figure 7 , Figure 7 This is a schematic diagram of a device structure for realizing image point cloud segmentation in a substation scene, as disclosed in an embodiment of this application.

[0138] like Figure 7 As shown, the device may include:

[0139] The real-time image point cloud information acquisition unit 11 is used to acquire real-time two-dimensional images and real-time three-dimensional point clouds of the substation.

[0140] Model segmentation unit 12 is used to input the real-time two-dimensional image and the real-time three-dimensional point cloud into a pre-trained substation image point cloud segmentation model and output the substation scene segmentation result.

[0141] The training data acquisition unit 13 is used to acquire two-dimensional images and three-dimensional point clouds of the substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0142] The pairing association establishment unit 14 is used to establish the pairing association between the pixels of the two-dimensional image and the points of the three-dimensional point cloud;

[0143] The dual-stream module training unit 15 is used to input the substation equipment point clusters and their paired associated two-dimensional images into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and to input the substation structure point clusters and their paired associated two-dimensional images into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0144] Training unit 16 is used to obtain the trained point cloud segmentation model of the substation image after both the pseudo-label prediction stream module and the original stream module have been trained.

[0145] Optionally, the pseudo-label prediction stream module has the same architecture as the original stream module, both including a 3D encoder, a 3D decoder, a 2D encoder, a 2D decoder, and several dual-modal fusion sub-modules.

[0146] The 3D encoder includes a 3×3 core 3D convolutional layer, a batch normalization layer, an activation layer, and a 2×2×2 core max pooling layer;

[0147] The 3D decoder includes a 2×2×2 kernel convolutional layer, a first feature splicing submodule, a 3D convolutional layer, a batch normalization layer, and an activation layer.

[0148] The 2D encoder includes a deep convolutional network, a dilated convolutional layer, a dilated spatial pyramid pooling layer, and a 1×1 kernel 2D convolutional layer;

[0149] The 2D decoder includes a first upsampling submodule, a 1×1 kernel 2D convolutional layer, a second feature splicing submodule, a 3×3 kernel 2D convolutional layer, and a second upsampling submodule.

[0150] The dual-modal fusion submodule is used to fuse the latent features of point clouds and images at each level of the 3D decoder and the 2D decoder to obtain three-dimensional fused features.

[0151] Optionally, the dual-modal fusion submodule is used for:

[0152] The 3D features output by the 3D decoder are subjected to Minkowski convolution to obtain 3D key features and 3D value features, and the 2D features output by the 2D decoder are subjected to standard convolution to obtain 2D query features.

[0153] The three-dimensional key features and the two-dimensional query features are sequentially subjected to dot product, summation and first-level softmax operations to generate a three-dimensional attention map;

[0154] The multi-head attention feature is obtained by weighted summation of the three-dimensional attention map and the three-dimensional value feature;

[0155] The multi-head attention feature is extended to the same dimension as the three-dimensional feature and then concatenated with the three-dimensional feature to obtain a three-dimensional concatenated feature.

[0156] The three-dimensional stitched features are subjected to Minkowski convolution to obtain the three-dimensional fused features.

[0157] Optionally, the loss function for training the substation image point cloud segmentation model is:

[0158]

[0159] in, To train the loss term, This refers to the i-th point-level semantic feature output by the substation image point cloud segmentation model during training. For pixel-level semantic features that are paired and associated with the i-th point-level semantic feature, N is the total number of pairings between the pixels of the two-dimensional image and the points of the three-dimensional point cloud.

[0160] Optionally, the pseudo-label prediction stream module is used for:

[0161] During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the substation structural point clusters to obtain coarse-grained three-dimensional pseudo-labels.

[0162] During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the two-dimensional images that are paired and associated with the substation structural point clusters, resulting in coarse-grained two-dimensional pseudo-labels.

[0163] The substation image point cloud segmentation model also includes a pseudo-label optimization module, which is used for:

[0164] Obtain the substation structural point cluster with coarse-grained three-dimensional pseudo-labels, and the two-dimensional image with coarse-grained two-dimensional pseudo-labels paired and associated with the substation structural point cluster.

[0165] The coarse-grained two-dimensional pseudo-label is projected onto the target matching point to obtain the projected three-dimensional pseudo-label.

[0166] The coarse-grained 3D pseudo-labels are optimized using the first label optimization formula, which is:

[0167] ;

[0168] in, The substation structural point cluster, To optimize 3D pseudo-labels, The coarse-grained three-dimensional pseudo-label, The projected 3D pseudo-label, For confidence function, The confidence threshold;

[0169] The coarse-grained 3D pseudo-labels are projected onto the target image plane to extract the target region composed of pixels not projected onto the target image plane. The target region is then averaged and pooled to output the dense 2D pseudo-labels with the maximum dimension.

[0170] The coarse-grained two-dimensional pseudo-labels are optimized using the second label optimization formula, which is as follows:

[0171]

[0172] in, A two-dimensional image associated with the structural point cluster of the substation. To optimize 2D pseudo-labels, The coarse-grained two-dimensional pseudo-label, These are dense two-dimensional pseudo-labels.

[0173] Optionally, the parameter update formula for the pseudo-label prediction stream module is:

[0174]

[0175] in, These are the module parameters for the pseudo-label prediction stream module, and the initial values ​​of these module parameters are determined based on the original stream module. Update the weight coefficients for the parameters. Here, s represents the updated value of the module parameters, and s represents the training step number of the original stream module. The parameters of the original stream module after being updated in the s-th training step. This is a preset threshold.

[0176] Optionally, the device may also include:

[0177] The image point cloud optimization unit is used to optimize the two-dimensional image and the three-dimensional point cloud after acquiring the two-dimensional image and the three-dimensional point cloud of the substation scene, so as to obtain the optimized two-dimensional image and the three-dimensional point cloud.

[0178] The image point cloud segmentation device for substation scenes provided in this application embodiment can be applied to devices for image point cloud segmentation in substation scenes, such as terminals: mobile phones, computers, etc. Optionally, Figure 8 The hardware structure block diagram of the device for image point cloud segmentation in a substation scene is shown. (Refer to...) Figure 8 The hardware structure of the device for image point cloud segmentation in a substation scene may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.

[0179] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0180] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0181] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0182] The memory stores a program, which the processor can call. The program is used for:

[0183] Acquire real-time 2D images and real-time 3D point clouds of the substation;

[0184] The real-time 2D image and the real-time 3D point cloud are input into a pre-trained substation image point cloud segmentation model, and the substation scene segmentation result is output.

[0185] The training process of the substation image point cloud segmentation model includes:

[0186] Acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0187] Establish a pairing association between the pixels of the two-dimensional image and the points of the three-dimensional point cloud;

[0188] The substation equipment point clusters and their paired associated two-dimensional images are input into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and the substation structure point clusters and their paired associated two-dimensional images are input into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0189] After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained.

[0190] Optionally, the refined and extended functions of the program can be found in the description above.

[0191] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0192] Acquire real-time 2D images and real-time 3D point clouds of the substation;

[0193] The real-time 2D image and the real-time 3D point cloud are input into a pre-trained substation image point cloud segmentation model, and the substation scene segmentation result is output.

[0194] The training process of the substation image point cloud segmentation model includes:

[0195] Acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters.

[0196] Establish a pairing association between the pixels of the two-dimensional image and the points of the three-dimensional point cloud;

[0197] The substation equipment point clusters and their paired associated two-dimensional images are input into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and the substation structure point clusters and their paired associated two-dimensional images are input into the original stream module of the substation image point cloud segmentation model to train the original stream module.

[0198] After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained.

[0199] Optionally, the refined and extended functions of the program can be found in the description above.

[0200] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0202] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for segmenting point clouds in an image of a substation scene, characterized in that, include: Acquire real-time 2D images and real-time 3D point clouds of the substation; The real-time 2D image and the real-time 3D point cloud are input into a pre-trained substation image point cloud segmentation model, and the substation scene segmentation result is output. The training process of the substation image point cloud segmentation model includes: Acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters. Establish a pairing association between the pixels of the two-dimensional image and the points of the three-dimensional point cloud; The substation equipment point clusters and their paired associated two-dimensional images are input into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and the substation structure point clusters and their paired associated two-dimensional images are input into the original stream module of the substation image point cloud segmentation model to train the original stream module. After both the pseudo-label prediction stream module and the original stream module have completed training, the trained substation image point cloud segmentation model is obtained.

2. The method according to claim 1, characterized in that, The pseudo-label prediction stream module has the same architecture as the original stream module, both including a 3D encoder, a 3D decoder, a 2D encoder, a 2D decoder, and several dual-modal fusion sub-modules; The 3D encoder includes a 3×3 core 3D convolutional layer, a batch normalization layer, an activation layer, and a 2×2×2 core max pooling layer; The 3D decoder includes a 2×2×2 kernel convolutional layer, a first feature splicing submodule, a 3D convolutional layer, a batch normalization layer, and an activation layer. The 2D encoder includes a deep convolutional network, a dilated convolutional layer, a dilated spatial pyramid pooling layer, and a 1×1 kernel 2D convolutional layer; The 2D decoder includes a first upsampling submodule, a 1×1 kernel 2D convolutional layer, a second feature splicing submodule, a 3×3 kernel 2D convolutional layer, and a second upsampling submodule. The dual-modal fusion submodule is used to fuse the latent features of point clouds and images at each level of the 3D decoder and the 2D decoder to obtain three-dimensional fused features.

3. The method according to claim 2, characterized in that, The dual-modal fusion submodule is used for: The 3D features output by the 3D decoder are subjected to Minkowski convolution to obtain 3D key features and 3D value features, and the 2D features output by the 2D decoder are subjected to standard convolution to obtain 2D query features. The three-dimensional key features and the two-dimensional query features are sequentially subjected to dot product, summation and first-level softmax operations to generate a three-dimensional attention map; The multi-head attention feature is obtained by weighted summation of the three-dimensional attention map and the three-dimensional value feature; The multi-head attention feature is extended to the same dimension as the three-dimensional feature and then concatenated with the three-dimensional feature to obtain a three-dimensional concatenated feature. The three-dimensional stitched features are subjected to Minkowski convolution to obtain the three-dimensional fused features.

4. The method according to claim 1, characterized in that, The loss function for training the substation image point cloud segmentation model is: in, To train the loss term, This refers to the i-th point-level semantic feature output by the substation image point cloud segmentation model during training. For pixel-level semantic features that are paired and associated with the i-th point-level semantic feature, N is the total number of pairings between the pixels of the two-dimensional image and the points of the three-dimensional point cloud.

5. The method according to claim 1, characterized in that, The pseudo-label prediction stream module is used for: During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the substation structural point clusters to obtain coarse-grained three-dimensional pseudo-labels. During the training process of the pseudo-label prediction stream module, pseudo-labels are predicted for the two-dimensional images that are paired and associated with the substation structural point clusters, resulting in coarse-grained two-dimensional pseudo-labels. The substation image point cloud segmentation model also includes a pseudo-label optimization module, which is used for: Obtain the substation structural point cluster with coarse-grained three-dimensional pseudo-labels, and the two-dimensional image with coarse-grained two-dimensional pseudo-labels paired and associated with the substation structural point cluster. The coarse-grained two-dimensional pseudo-label is projected onto the target matching point to obtain the projected three-dimensional pseudo-label. The coarse-grained 3D pseudo-labels are optimized using the first label optimization formula, which is: ； in, The substation structural point cluster, To optimize 3D pseudo-labels, The coarse-grained three-dimensional pseudo-label, The projected 3D pseudo-label, For confidence function, The confidence threshold; The coarse-grained 3D pseudo-labels are projected onto the target image plane to extract the target region composed of pixels not projected onto the target image plane. The target region is then averaged and pooled to output the dense 2D pseudo-labels with the maximum dimension. The coarse-grained two-dimensional pseudo-labels are optimized using the second label optimization formula, which is as follows: in, A two-dimensional image associated with the structural point cluster of the substation. To optimize 2D pseudo-labels, The coarse-grained two-dimensional pseudo-label, These are dense two-dimensional pseudo-labels.

6. The method according to claim 5, characterized in that, The parameter update formula for the pseudo-label prediction stream module is as follows: in, These are the module parameters for the pseudo-label prediction stream module, and the initial values ​​of these module parameters are determined based on the original stream module. Update the weight coefficients for the parameters. Here, s represents the updated value of the module parameters, and s represents the training step number of the original stream module. The parameters of the original stream module after being updated in the s-th training step. This is a preset threshold.

7. The method according to any one of claims 1-6, characterized in that, After acquiring the two-dimensional image and three-dimensional point cloud of the substation scene, the method further includes: The two-dimensional image and the three-dimensional point cloud are optimized to obtain the optimized two-dimensional image and the three-dimensional point cloud.

8. An image point cloud segmentation device for a substation scene, characterized in that, include: The real-time image point cloud information acquisition unit is used to acquire real-time two-dimensional images and real-time three-dimensional point clouds of the substation. The model segmentation unit is used to input the real-time two-dimensional image and the real-time three-dimensional point cloud into a pre-trained substation image point cloud segmentation model and output the substation scene segmentation result. The training data acquisition unit is used to acquire two-dimensional images and three-dimensional point clouds of a substation scene. The three-dimensional point cloud includes labeled substation equipment point clusters and unlabeled substation structure point clusters. A pairing association establishment unit is used to establish pairing associations between pixels of the two-dimensional image and points of the three-dimensional point cloud; The dual-stream module training unit is used to input the substation equipment point clusters and their paired associated two-dimensional images into the pseudo-label prediction stream module of the substation image point cloud segmentation model to train the pseudo-label prediction stream module, and to input the substation structure point clusters and their paired associated two-dimensional images into the original stream module of the substation image point cloud segmentation model to train the original stream module. The training unit is used to obtain the trained point cloud segmentation model of the substation image after both the pseudo-label prediction stream module and the original stream module have been trained.

9. An image point cloud segmentation device for a substation scene, characterized in that, Including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the image point cloud segmentation method for substation scenes as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the image point cloud segmentation method for substation scenes as described in any one of claims 1-7.

Images