Tower collection method and system based on spatial relationship

Abstract

The invention discloses a tower collection method and system based on a spatial relationship. The method comprises the following steps: generating a Planck coordinate code, and fusing the Planck coordinate code with an original image to form a multi-channel image; performing feature extraction on the multi-channel image, adding the Patch code of the original image and the Planck coordinate code after linear mapping, and performing pre-training to extract high-order semantic features; extracting target area features from the feature map and fusing position information; modeling target area features by alternately executing global attention and single-map attention to learn a relationship between targets and generate advanced features; and calculating the similarity of the target region based on the advanced features, and judging the attribution of the target region through the similarity to realize tower matching and de-weighting. According to the method, the relation between the targets in the multi-view image is effectively modeled, the target matching accuracy in a complex scene is improved, the model can deduce the potential position of the shielded target, a single-view blind area is made up, and the information integrity of a single image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a tower aggregation method in the field of power systems, in particular to a tower aggregation method based on spatial relationship, and also relates to a tower aggregation system based on spatial relationship. BACKGROUND

[0002] In power inspection, when a device such as a UAV takes pictures of a tower from multiple angles, due to the complexity of the target structure and the occlusion effect, a single image cannot completely capture the three-dimensional information of the tower. For example, the equipment on the back side of the tower or small elements (such as insulator strings) are not visible at a certain angle of view, resulting in information blind spots in single-image perception. In order to construct a complete three-dimensional model of the tower, multiple-view images must be fused to complement the spatial information, but traditional two-dimensional images lack effective cross-view correlation mechanisms.

[0003] The same target (such as equipment or defects) appears repeatedly in different angles, causing data redundancy and matching ambiguity. If multiple-view image information is simply superimposed, errors will accumulate due to target position offset and scale changes. For example, the lightning arrester at the top of the tower appears in different forms in the overhead and side-view images, and simple addition will destroy data consistency. Therefore, the existing tower aggregation method has the problems of incomplete information and inaccurate matching. SUMMARY

[0004] To solve the technical problems of incomplete information and inaccurate matching in the existing tower aggregation method, the present application provides a tower aggregation method based on spatial relationship and a system thereof.

[0005] The present application adopts the following technical solutions: a tower aggregation method based on spatial relationship, comprising the following steps: S1: generating a Planck coordinate code based on the positioning coordinates of the camera and the tilt angle of the gimbal, fusing the Planck coordinate code with the original image through channel splicing to form a multi-channel image containing color and three-dimensional spatial information; S2: extracting features from the multi-channel image, adding the Patch code of the original image to the linearly mapped Planck coordinate code, and pre-training through a DINOv2 model to extract high-order semantic features; S3: extracting target region features from the feature map extracted from the DINOv2 model and fusing position information; S4: modeling multi-view relationships of the target region features by alternately performing global attention and single-image attention to learn the relationships between targets and generate high-level features; S5: calculating the similarity of the target region based on the high-level features, and determining the ownership of the target region through the similarity to realize tower matching and deduplication.

[0006] The application converts the camera coordinates and the gimbal angle into three-dimensional space coding by Plank coordinate coding fusion and DINOv2 space feature adaptation, so that the two-dimensional image carries the spatial position information of the tower, and the DINOv2 model simultaneously analyzes the RGB texture and the three-dimensional structure through the coding addition mechanism, and can perceive the spatial relationship of the target in the occlusion scene (such as the equipment on the back side of the tower), so that the model can infer the potential position of the occluded target, compensate for the single-view blind area, and improve the integrity of the single-image information. The application calculates the cross-image target similarity through the alternating attention mechanism and feature similarity matching to establish the spatial correlation, and at the same time, the local features of the targets in the same image are strengthened, the same target is determined through the similarity, and the repeated counting error is eliminated, so that the collection accuracy is obviously improved, thereby solving the technical problems of incomplete information and inaccurate matching existing in the existing tower collection method.

[0007] As a further improvement of the above scheme, in step S1, the internal and external parameters of the camera are calculated based on the real-time dynamic differential positioning coordinates recorded when the camera shoots the tower and the elevation angle, and then the internal and external parameters are converted into the Plank coordinate coding.

[0008] As a further improvement of the above scheme, in step S2, the high-order semantic features of the image are extracted by inputting the multi-channel image into a feature extraction network, and the semantic information of the multi-channel image is extracted by the DINOv2 model to obtain visual features.

[0009] As a further improvement of the above scheme, in step S3, according to the boundary box information obtained by target detection based on a single image, the features of each target region are extracted from the feature map using ROI-Align, and the position information features in the target region features are strengthened based on the position coding of the boundary box information.

[0010] As a further improvement of the above scheme, in step S4, the target region features are input into a Transformer network for processing, and a mask design graph neural network is designed by controlling the mask.

[0011] Further, the modeling method comprises the following steps: in the global attention stage, the self-attention mechanism of the Transformer network is configured to allow the features of all target regions of all images to interact with each other; in the single-image attention stage, the self-attention mechanism of the Transformer network is configured to allow only the target region features within the same image to interact with each other.

[0012] Further, in step S5, first, the feature similarity between different target regions is calculated according to the features output by the Transformer network, and then it is judged whether the feature similarity reaches a preset similarity, yes or no, that is, whether different target regions belong to the same target or not.

[0013] As a further improvement of the above scheme, the alternately performing in step S4 includes the following steps: (1) First, global attention is performed: the feature similarity of the target regions in all images is calculated; (2) Then, single-image attention is performed: the feature similarity of the target regions in the same image is calculated; (3) Steps (1) and (2) are alternately performed until the number of alternation rounds reaches a preset number.

[0014] As a further improvement of the above scheme, in step S1, the three channels of the original image and the six channels of the Planck coordinate encoding are spliced to generate a nine-channel image.

[0015] The application also provides a tower aggregation system based on spatial relationship, which applies any of the above-mentioned tower aggregation methods based on spatial relationship; the system comprises: a three-dimensional space information fusion module for generating Planck coordinate encoding based on the positioning coordinates of the camera and the tilt angle of the gimbal, fusing the Planck coordinate encoding with the original image through channel splicing, and forming a multi-channel image containing color and three-dimensional space information; a high-order semantic feature extraction module for extracting features from the multi-channel image, adding the Patch encoding of the original image to the linearly mapped Planck coordinate encoding, and pre-training through a DINOv2 model to extract high-order semantic features; a target region feature extraction module for extracting target region features and fusing position information from the feature map extracted by the DINOv2 model; a multi-view relationship modeling module for modeling the multi-view relationship of the target region features by alternately performing global attention and single-image attention, learning the relationship between targets, and generating high-level features; a target matching and aggregation module for calculating the similarity of target regions based on high-level features, and judging the ownership of target regions through the similarity, realizing tower matching and deduplication.

[0016] Compared with existing tower aggregation methods, the tower aggregation method and system based on spatial relationship of the application have the following beneficial effects: 1. The tower grouping method based on spatial relationship, which converts camera coordinates and gimbal angles into three-dimensional spatial encoding through Planck coordinate encoding fusion and DINOv2 spatial feature adaptation, enables two-dimensional images to carry spatial position information of towers, and enables the DINOv2 model to simultaneously analyze RGB texture and three-dimensional structure through an encoding addition mechanism, perceive the spatial relationship of the target in a blocked scene (such as the equipment on the back side of the tower), enable the model to infer the potential position of the blocked target, compensate for the single-view blind area, improve the integrity of single-image information, and thus solve the technical problem of incomplete information existing in the existing tower grouping method.

[0017] 2. The tower grouping method based on spatial relationship, which calculates the similarity of targets across images to establish spatial correlation through an alternating attention mechanism and feature similarity matching, strengthens the local features of targets within the same image, determines the same target through similarity, eliminates repeated counting errors, and thus significantly improves the grouping accuracy, thereby solving the technical problem of inaccurate matching existing in the existing tower grouping method.

[0018] 3. The tower grouping method based on spatial relationship, which effectively integrates three-dimensional spatial information (camera internal and external parameters) into two-dimensional images through Planck coordinate encoding, provides rich three-dimensional spatial information for subsequent feature extraction and target matching, compensates for the limitations of traditional two-dimensional images, enhances the model's perception of the three-dimensional structure of the target, and improves the accuracy of target matching in complex scenes. Moreover, in order to adapt DINOv2 to input images containing Planck coordinate encoding, the structure and pre-training weights of DINOv2 are modified, and the patch encoding of the original RGB image is added to the linearly mapped Planck coordinate encoding. This modification not only retains the general feature extraction capability of the pre-training weights of DINOv2, but also enables the model to utilize three-dimensional spatial information from both RGB images and Planck coordinate encoding.

[0019] 4. The tower grouping method based on spatial relationship, which realizes the design of a graph neural network (GNN) by controlling the mask of the Transformer network, effectively models the relationship between targets in multi-view images. By alternately performing "global attention" and "single-image attention" stages, the model can simultaneously capture global and local relationships, thereby better understanding the target relationships in multi-view images and improving the accuracy of target matching.

[0020] 5. The tower grouping method based on spatial relationship, which automatically completes multi-view image alignment, target matching, and information integration through a complete closed loop of fusion, feature extraction, relationship modeling, and matching, significantly improves processing efficiency compared to traditional manual verification, and saves labor costs. Moreover, the algorithm can handle hundreds of input images in parallel, meeting the needs of power corridor line inspection.

[0021] The tower grouping system based on spatial relationship has the same beneficial effects as the tower grouping based on spatial relationship, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of the tower grouping method based on spatial relationship of the embodiment 1 of the present application.

[0023] Figure 2 A grouping framework of the tower grouping method based on spatial relationship of the embodiment 1 of the present application. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0025] Embodiment 1 Please refer to Figure 1 The embodiment provides a tower grouping method based on spatial relationship, which can be designed as an algorithm to realize the automation of tower matching and deduplication. The grouping method of the embodiment is a robust multi-view matching algorithm, which can accurately identify and remove duplicate targets in power inspection images, integrate multi-view information, and perfect tower information. The overall framework is as follows Figure 2 The figure shows an algorithm for grouping according to two pictures, which can be easily extended to the scene of multiple picture inputs. The grouping method includes the following steps (S1-S5) S1: Based on the positioning coordinates of the camera and the tilt angle of the gimbal, Plücker coordinate encoding is generated. The Plücker coordinate encoding is fused with the original image through channel splicing to form a multi-channel image containing color and three-dimensional spatial information. The multi-channel image contains RGB color information and three-dimensional position information. This step provides accurate three-dimensional spatial information for subsequent feature extraction and target matching, making up for the lack of spatial information in two-dimensional images and enhancing the overall framework's ability to perceive the three-dimensional structure of the target.

[0026] In the embodiment, the real-time dynamic differential positioning coordinates (i.e. RTK (Real-time kinematic) coordinate information) and the tilt angle recorded when the camera shoots the tower are first used to calculate the internal and external parameters of the camera, and then the internal and external parameters are converted into Plücker coordinate encoding. The encoding is an encoding method that can effectively represent three-dimensional spatial information. It can map three-dimensional coordinates to high-dimensional space, so that the network can better learn and utilize these information, thus effectively representing the three-dimensional spatial position information of the pixels in the image.

[0027] In some embodiments, the three channels of the original image can be spliced with the six channels of the Planck coordinate coding to generate a nine-channel image. Of course, other numbers of channels of the fused image can also be generated.

[0028] S2: Feature extraction is performed on the multi-channel image, the Patch encoding of the original image is added to the linearly mapped Planck coordinate coding, and a DINOv2 model is pre-trained to extract high-order semantic features. In this embodiment, the high-order semantic features of the image are extracted by inputting the multi-channel image into a feature extraction network, and the semantic information of the multi-channel image is extracted by the DINOv2 model to obtain visual features. DINOv2 realizes the visual feature generalization ability close to human level through automatic data construction, multi-task loss fusion and efficient distributed training.

[0029] After research, the embodiment selects DINOv2 as the pre-training network, because DINOv2 uses a self-supervised learning method to pre-train on a large-scale dataset for a long time. It can extract rich semantic information in the image and generate highly robust and general visual features, which perform well in various downstream tasks such as image classification, object detection, and semantic segmentation. Since the input image contains Planck coordinate coding, the model parameters of DINOv2 cannot be directly used. In the specific design, the Patch encoding part is modified, the Patch encoding of the original RGB image is added to the Planck coordinate coding after linear mapping, and then input into the subsequent DINOv2 network operation, which makes the model maintain the general feature extraction ability of the pre-training weight, and can also integrate the three-dimensional spatial information of the RGB image and the Planck coordinate coding, providing reliable feature representation for subsequent target matching.

[0030] S3: Extracting target region features and fusing position information from the feature map extracted by the DINOv2 model. In this embodiment, according to the bounding box information obtained based on single image target detection, the ROI-Align (Region of Interest Align) is used to extract the feature of each target region from the feature map, and the position coding based on the bounding box information is used to strengthen the position information feature in the target region feature. ROI-Align eliminates quantization errors through "floating point coordinates + bilinear interpolation", which is a key technology to improve the accuracy of target detection / segmentation. In the power tower clustering, it is combined with position information enhancement and multi-view attention modeling to solve the problem of small target matching in complex scenes and provide high-precision feature basis for three-dimensional reconstruction of towers.

[0031] S4: Multi-view relationship modeling is performed on the target region features by alternately performing global attention and single-image attention to learn the inter-target relationship and generate high-level features. In this embodiment, the target region features are input into a Transformer network for processing, and a graph neural network (GNN) is designed by controlling the mask. The modeling method includes the following steps: in the global attention stage, the self-attention mechanism of the Transformer network is configured to allow the features of all target regions of all images to interact with each other; in the single-image attention stage, the self-attention mechanism of the Transformer network is configured to allow the features of the target regions within the same image to interact with each other.

[0032] In this embodiment, the Transformer network learns the relationship between target regions by using the self-attention mechanism and extracts higher-level feature representations. Moreover, the alternating "global attention-single image attention" form is performed in the Transformer. The specific way of the modeling method of this embodiment is that the alternating "global attention-single image attention" form is performed in the Transformer, first in the global attention stage, the self-attention mechanism of the Transformer is configured to allow the features of all target regions of all images to interact with each other, which helps to capture long-distance dependency relationships between different target regions, so that the model can understand the relationship between different targets in different images, for example, whether the targets in two images are the same target; in the single-image attention stage, the self-attention mechanism of the Transformer is configured to allow the features of the target regions within the same image to interact with each other, which helps to capture the local relationship between the target regions within the same image and extract more fine-grained feature representations. By alternately performing the global attention and single-image attention stages, the model can capture both global and local relationships, so that the model can better understand the target relationship in multi-view images and improve the accuracy of target matching.

[0033] In some other embodiments, the alternately performing in step S4 can include the following steps: (1) first performing global attention: calculating the feature similarity of the target regions in all images; (2) then performing single-image attention: only calculating the feature similarity of the target regions within the same image; (3) alternately performing steps (1) and (2) until the alternating rounds reach a preset number of times.

[0034] S5: Calculate the target region similarity based on the high-level features, and determine the target region attribution through the similarity to realize the matching and deduplication of the towers. Among them, the feature similarity between different target regions is calculated by using the features output by the Transformer network, and whether different target regions belong to the same target is determined according to the feature similarity, and finally the matching result is output. According to the matching result, the automatic matching and deduplication of the targets in the multi-view image are realized, and finally the collection at the tower level is completed.

[0035] In the embodiment, the specific judgment method is to first calculate the feature similarity between different target regions according to the features output by the Transformer network, and then determine whether the different target regions belong to the same target by judging whether the feature similarity reaches a preset similarity, yes or no.

[0036] In summary, compared with the existing tower collection method, the tower collection method based on spatial relationship of the embodiment has the following beneficial effects: 1. The tower collection method based on spatial relationship, by Planck coordinate coding fusion and DINOv2 spatial feature adaptation, converts the camera coordinates and gimbal angles into three-dimensional space coding, so that the two-dimensional image carries the spatial position information of the tower, and the DINOv2 model analyzes the RGB texture and three-dimensional structure through the coding addition mechanism, and can still perceive the spatial relationship of the target in the occlusion scene (such as the equipment on the back side of the tower), so that the model can infer the potential position of the occluded target, make up for the single-view blind area, and improve the integrity of the single-image information, thereby solving the technical problem of incomplete information existing in the existing tower collection method.

[0037] 2. The tower collection method based on spatial relationship, by alternating attention mechanism and feature similarity matching, calculates the cross-image target similarity to establish spatial correlation, while strengthening the local features of the targets within the same image, determines the same target through similarity, eliminates the repeated counting error, and significantly improves the collection accuracy, thereby solving the technical problem of inaccurate matching existing in the existing tower collection method.

[0038] 3、The tower aggregation method based on spatial relationship effectively integrates three-dimensional spatial information (camera internal and external parameters) into two-dimensional images through Plenoptic coordinate encoding, providing rich three-dimensional spatial information for subsequent feature extraction and target matching, making up for the limitations of traditional two-dimensional images, enhancing the model's perception of the target's three-dimensional structure, and improving the accuracy of target matching in complex scenes. Moreover, to adapt DINOv2 to input images containing Plenoptic coordinate encoding, the structure and pre-training weights of DINOv2 are modified, and the Patch encoding of the original RGB image is added to the linearly mapped Plenoptic coordinate encoding. This modification not only retains the general feature extraction capability of DINOv2 pre-training weights, but also enables the model to utilize three-dimensional spatial information in RGB images and Plenoptic coordinate encoding.

[0039] 4、The tower aggregation method based on spatial relationship realizes the design of Graph Neural Network (GNN) by controlling the mask of the Transformer network, effectively modeling the relationships between targets in multi-view images. By alternating "global attention" and "single-image attention" stages, the model can capture both global and local relationships, better understanding the target relationships in multi-view images and improving the accuracy of target matching.

[0040] 5、The tower aggregation method based on spatial relationship automatically completes multi-view image alignment, target matching, and information integration through a complete closed loop of fusion, feature extraction, relationship modeling, and matching, significantly improving processing efficiency compared to traditional manual verification and saving labor costs. Moreover, the algorithm can handle hundreds of input images in parallel, meeting the needs of power corridor line inspection.

[0041] Embodiment 2 This embodiment provides a tower aggregation system based on spatial relationship, which applies the tower aggregation method based on spatial relationship in Embodiment 1 and specifically includes a three-dimensional spatial information fusion module, a high-order semantic feature extraction module, a target region feature extraction module, a multi-view relationship modeling module, and a target matching and aggregation module.

[0042] The three-dimensional space information fusion module is configured to generate a Planck coordinate code based on the camera-based positioning coordinates and the gimbal tilt angle, fuse the Planck coordinate code with the original image through channel splicing, and form a multi-channel image containing color and three-dimensional space information. The high-order semantic feature extraction module is configured to extract features from the multi-channel image, add the patch code of the original image to the linearly mapped Planck coordinate code, and pre-train through a DINOv2 model to extract high-order semantic features. The target region feature extraction module is configured to extract target region features from the feature map extracted by the DINOv2 model and fuse the position information. The multi-view relationship modeling module is configured to model the multi-view relationship of the target region features by alternately performing global attention and single-image attention, learn the relationship between targets, and generate high-level features. The target matching and collection module is configured to calculate the similarity of target regions based on the high-level features, determine the ownership of the target regions through the similarity, and realize the matching and deduplication of towers.

[0043] Embodiment 3 The embodiment provides a computer terminal including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor implements the steps of the tower collection method based on spatial relationship of embodiment 1 when executing the program.

[0044] The method of embodiment 1 can be applied in the form of software, such as a program designed to run independently, installed on a computer terminal, which can be a computer, a smart phone, a control system, and other Internet of Things devices, etc. The method of embodiment 1 can also be designed as an embedded program running on a computer terminal, such as a single-chip microcomputer.

[0045] Embodiment 4 The embodiment provides a computer-readable storage medium having a computer program stored thereon. The program is executed by a processor to implement the steps of the tower collection method based on spatial relationship of embodiment 1.

[0046] The method of embodiment 1 can be applied in the form of software, such as a program designed to run independently on a computer-readable storage medium, which can be a U disk, designed as a U disk, and designed as a program that starts the entire method through external triggering through a U disk.

[0047] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A space relation based tower clustering method, characterized in that, It comprises the following steps: S1: based on the camera positioning coordinates and the gimbal tilt angle to generate the Planck coordinate code, through channel splicing to fuse the Planck coordinate code with the original image, forming a multi-channel image containing color and three-dimensional space information; S2: feature extraction is performed on the multi-channel image, the patch code of the original image is added to the linearly mapped Planck coordinate code, and a DINOv2 model is pre-trained to extract high-order semantic features; S3: target region features are extracted from the feature map extracted from the DINOv2 model and fused with position information; S4: the target region features are modeled for multi-view relationship by alternately performing global attention and single-image attention to learn the relationship between targets and generate high-level features; S5: based on the high-level features, the similarity of the target region is calculated, and the target region ownership is judged through the similarity to realize the tower matching and deduplication.

2. The space relation based tower aggregation method as claimed in claim 1, wherein, In step S1, the internal and external parameters of the camera are calculated based on the real-time dynamic differential positioning coordinates recorded when the camera shoots the tower and the tilt angle, and then the internal and external parameters are converted into the Planck coordinate code.

3. The space relation based tower aggregation method as claimed in claim 1, wherein, In step S2, the high-order semantic features of the image are extracted by inputting the multi-channel image into a feature extraction network, and the semantic information of the multi-channel image is extracted by the DINOv2 model to obtain visual features.

4. The space relation based tower aggregation method as claimed in claim 1, wherein, In step S3, according to the bounding box information obtained based on single-image target detection, the features of each target region are extracted from the feature map using ROI-Align, and the position information features in the target region features are enhanced based on the position coding of the bounding box information.

5. The space relation based tower aggregation method as claimed in claim 1, wherein, In step S4, the target region features are input into a Transformer network for processing, and a mask design neural network is controlled.

6. The space relation based tower aggregation method as claimed in claim 5, wherein, The modeling method comprises the following steps: in the global attention stage, the self-attention mechanism of the Transformer network is configured to allow the features of all target regions of all images to interact with each other; in the single-image attention stage, the self-attention mechanism of the Transformer network is configured to allow only the target region features within the same image to interact with each other.

7. The space relation based tower aggregation method as claimed in claim 6, wherein, In step S5, first, the feature similarity between different target regions is calculated according to the features output by the Transformer network, and then it is judged whether the feature similarity reaches a preset similarity, yes, it is judged that different target regions belong to the same target, otherwise, different target regions do not belong to the same target.

8. The space relation based tower aggregation method as claimed in claim 1, wherein, The alternately performing in step S4 comprises the following steps: (1) first global attention: calculate the feature similarity of the target regions in all images; (2) then single-image attention: only calculate the feature similarity of the target regions within the same image; (3) alternately perform steps (1) and (2) until the alternation reaches a preset number of times.

9. The space relation based tower aggregation method as claimed in claim 1, wherein, In step S1, the three channels of the original image are spliced with the six channels of the Planck coordinate code to generate a nine-channel image.

10. A space relation based tower aggregation system, characterized in that, The application of the tower clustering method based on spatial relationship as claimed in any one of claims 1-9; the system comprises: a three-dimensional space information fusion module for generating Planck coordinate coding based on the positioning coordinates of the camera and the tilt angle of the holder, fusing the Planck coordinate coding with the original image through channel splicing to form a multi-channel image containing color and three-dimensional space information; a high-order semantic feature extraction module for extracting features from the multi-channel image, adding the Patch coding of the original image to the linearly mapped Planck coordinate coding, and pre-training through a DINOv2 model to extract high-order semantic features; a target area feature extraction module for extracting target area features from the feature map extracted from the DINOv2 model and fusing the position information; a multi-view relationship modeling module for modeling the multi-view relationship of the target area features by alternately performing global attention and single-image attention to learn the relationship between targets and generate high-level features; a target matching and clustering module for calculating the similarity of target areas based on high-level features, judging the ownership of target areas through the similarity, and realizing tower matching and deduplication.

Images