A power transmission line segmentation method based on binocular images

Abstract

The application discloses a kind of based on binocular image transmission line segmentation method: step 1: obtaining binocular transmission line image segmentation dataset, corrects picture in data set;Step 2: construct binocular image-based transmission line segmentation network, including two main networks, feature pyramid network, binocular disparity depth estimation subnetwork, multi-scale disparity color fusion module and spatial information enhanced segmentation head;Step 3: the unsupervised pre-training of disparity estimation task is carried out to binocular disparity depth estimation subnetwork;Step 4: the segmentation network is carried out on the updated dataset disparity estimation and transmission line segmentation double-task joint training;Step 5: the binocular transmission line image to be identified is input into the trained segmentation network and identified, and the disparity map and transmission line identification segmentation result are obtained.The application improves the accuracy of segmentation result, effectively reduces the discontinuity of line in segmentation result.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a transmission line segmentation method based on binocular images. Background Art

[0002] Drones are widely used in power transmission line inspections. Inspectors operate drones to capture images of transmission lines, which are then analyzed and identified. Transmission line location and segmentation are critical tasks during drone inspections. These tasks are used for path planning and safe navigation of transmission line inspection drones, preventing them from deviating from their intended flight path during inspections and preventing collisions between them and transmission lines. Because drone inspection images often contain complex backgrounds, transmission line segmentation presents a challenge. Transmission line identification and segmentation methods based on traditional digital image technology have significant limitations and low accuracy. Some researchers have applied deep learning-based image segmentation methods, such as U-net and the DeepLab series, to transmission line segmentation, achieving better results than traditional digital image processing methods. However, these methods still have some drawbacks: they are easily affected by objects in the image, such as branches, leaves, and buildings, resulting in a high level of noise in the segmented lines. Summary of the Invention

[0003] The purpose of the present invention is to provide a power transmission line segmentation method based on binocular images to solve the problems of excessive noise and line discontinuity in the line segmentation results of existing power transmission line segmentation methods.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A method for segmenting a power transmission line based on binocular images comprises the following steps:

[0006] Step 1: Obtain a binocular power transmission line image segmentation dataset, perform distortion correction and epipolar correction on the images in the dataset, and obtain the corrected left and right images, that is, the updated dataset;

[0007] Step 2: Construct a binocular image-based power transmission line segmentation network, which includes two backbone networks, a feature pyramid network, a binocular disparity depth estimation subnetwork, a multi-scale disparity color fusion module, and a spatial information enhanced segmentation head; wherein:

[0008] The two backbone networks are used to extract features from the corrected left and right images respectively. Specifically, the left and right images are input into two parallel backbone networks respectively to obtain four left feature maps of different scales. and right feature map Their sizes are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original image;

[0009] The binocular disparity depth estimation sub-network is used to calculate the disparity map corresponding to the left feature map and the right feature map extracted by the two backbone networks using the Coarse-to-Fine strategy;

[0010] The feature pyramid network is used to extract the four left feature maps of different scales from the backbone network. Perform top-down information transfer to obtain the feature pyramid corresponding to the left image

[0011] The multi-scale color disparity fusion module is used to perform multi-scale fusion of the feature pyramid and the left disparity map to obtain an RGBD feature pyramid {N2, N3, N4, N5} embedded with depth information;

[0012] The segmentation head with spatial information enhancement is used to calculate the final transmission line segmentation result based on the RGBD feature pyramid, and contains a spatial information transfer module, which is used to transfer the structural information of the target object in the feature map along the width and height directions;

[0013] Step 3: Perform unsupervised pre-training of the binocular disparity depth estimation subnetwork on the FlyingThings3D dataset for the disparity estimation task;

[0014] Step 4: The binocular image-based transmission line segmentation network constructed in step 2 is jointly trained on the disparity estimation and transmission line segmentation tasks on the updated dataset obtained in step 1 to obtain a trained binocular image-based transmission line segmentation network.

[0015] Step 5: Input the binocular transmission line image to be identified into the trained binocular image-based transmission line segmentation network for identification, and obtain the disparity map and transmission line identification and segmentation results.

[0016] Furthermore, in step 2, the binocular disparity depth estimation sub-network uses the Coarse-to-Fine strategy to calculate the disparity map of the left feature map and the right feature map extracted by the two backbone networks. The specific process is: the four pairs of left feature maps of different scales extracted by the two backbone networks are combined into a single matrix. and right feature map The number of channels is compressed by the BasicBlock residual unit; then the four pairs of left and right feature maps after channel compression are input into the matching cost volume calculation module of the corresponding scale to obtain two sets of matching cost volumes, namely the left matching cost volume and right matching cost Then, these two sets of matching cost volumes are upsampled and accumulated in the order of scale from small to large to obtain the new left matching cost volume. and right matching cost Finally, the new matching cost volume is subjected to disparity regression calculation to obtain the left disparity map corresponding to the left and right images in the data set obtained in step 1. and right disparity map The sizes of these two sets of disparity maps are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original images.

[0017] Furthermore, in step 2, the multi-scale color disparity fusion module implements the process of: inputting the feature map of the left disparity map output by the binocular disparity depth estimation subnetwork after three downsampling operations into a convolution layer, a batch normalization layer, and an activation layer to obtain a disparity map pyramid, and fusing the feature pyramid with the disparity map pyramid at the corresponding scale to obtain an RGBD feature pyramid embedded with depth information.

[0018] Furthermore, in step 2, the spatial information enhanced segmentation head increases the size of the RGBD feature pyramid embedded with depth information to 1 / 4 of the size of the original image through several upsampling layers, and then adds and fuses the four upsampled feature maps to obtain a fused feature map; then, the structural information of the target object in the fused feature map is sequentially transmitted along the width and height directions in the spatial dimensions through the spatial information transmission module to obtain a feature enhanced feature map; a 1×1 convolution layer and a four-fold bilinear upsampling operation are used on the feature enhanced feature map to output a segmentation probability map with the same length and width as the original input image.

[0019] Furthermore, the matching cost volume calculation module has two sets of inputs and two sets of outputs, the inputs are the left and right feature maps, and the outputs are the left and right matching cost volumes. The calculation process of the left matching cost volume is as follows: first, the right feature map F with a size of h×w×c is right Perform dimension conversion to obtain F with size h×c×w r ' igt , then F r ' igt Input a 1×1 convolution layer to obtain a feature map K of size h×c×w right , and at the same time the left feature map F left Another 1×1 convolutional layer is also input to obtain a feature map Q of scale h×w×c left , and finally Q left and K right Perform batch matrix multiplication to calculate the left matching cost volume V of size h×w×w left .

[0020] Furthermore, the disparity regression calculation refers to using a soft argmax operation to calculate the disparity map corresponding to the binocular image according to the matching cost volume.

[0021] Furthermore, the loss function used in the network training in step 4 is composed of a disparity estimation loss function and an image segmentation loss function, wherein the disparity estimation loss function is composed of a photometric loss function, a disparity smoothing loss function, a disparity consistency loss function and an auxiliary disparity loss function, and the image segmentation loss function is composed of a Dice loss function and a binary cross entropy loss function.

[0022] The beneficial effects of the present invention are as follows:

[0023] (1) The present invention designs a multi-scale color-disparity fusion module, which embeds the depth information in the disparity map into the color feature map, enhances the anti-interference ability of the network model to complex backgrounds, and improves the accuracy of the segmentation results.

[0024] (2) The present invention adds a spatial information transmission module to the network, which sequentially transmits the structural information of the transmission line in each slice, making the final line segmentation result more refined and smooth, and effectively reducing the occurrence of line discontinuity in the segmentation result.

[0025] (3) The end-to-end transmission line segmentation model proposed in the present invention can not only segment the transmission lines in the UAV inspection images, but also output the corresponding disparity map / depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a diagram showing the overall architecture of the binocular image-based power transmission line segmentation network in the present invention;

[0027] Figure 2 are the left and right images after epipolar correction;

[0028] Figure 3 This is the structural diagram of the binocular disparity depth estimation subnetwork;

[0029] Figure 4 This is the structural diagram of the matching cost volume calculation module;

[0030] Figure 5 This is the structural diagram of the multi-scale color disparity fusion module;

[0031] Figure 6 This is the structure diagram of the color parallax fusion submodule;

[0032] Figure 7 Segmentation head structure diagram for spatial information enhancement;

[0033] Figure 8 Schematic diagram of feature map slicing;

[0034] Figure 9 Schematic diagram of how spatial information is transferred between slices;

[0035] Figure 10 The following are the test results of the transmission line segmentation model based on binocular images on some pictures. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1

[0038] The power transmission line segmentation method based on binocular images provided in this embodiment includes the following steps:

[0039] Step 1: Obtain a binocular power transmission line image dataset, perform distortion correction and epipolar correction on the images in the dataset, and obtain the corrected left and right images.

[0040] In this example, the binocular power transmission line segmentation dataset contains 948 pairs of binocular images of power transmission lines, captured by a drone equipped with a ZED binocular camera. Most of the images in the dataset contain multiple power transmission lines, while a few contain only backgrounds. All images in the dataset have been binocularly distorted and epipolar corrected using the OpenCV remap function. After correction, the corresponding pixels on the left and right images are located on the same horizontal epipolar line, such as Figure 2 shown.

[0041] Step 2: Construct a binocular image-based power line segmentation network, which includes a backbone network, a feature pyramid network (FPN), a binocular disparity depth estimation subnetwork, a multi-scale disparity color fusion module, and a segmentation head with spatial information enhancement.

[0042] The two backbone networks are used to extract features from the corrected left and right images respectively, and obtain four left feature maps of different scales. and right feature map Their sizes are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original image;

[0043] The binocular disparity depth estimation sub-network is used to calculate the disparity map corresponding to the left feature map and the right feature map extracted by the two backbone networks using the Coarse-to-Fine strategy;

[0044] Feature Pyramid Network (FPN) is used to extract the four left feature maps of different scales from the backbone network. Perform top-down information transfer to obtain the feature pyramid corresponding to the left image

[0045] The multi-scale color disparity fusion module is used to fully multi-scale fuse the feature pyramid with the left disparity map to obtain the RGBD feature pyramid {N2, N3, N4, N5} embedded with depth information;

[0046] The segmentation head with spatial information enhancement is used to calculate the transmission line segmentation probability map based on the RGBD feature pyramid.

[0047] The detailed steps of calculating the disparity map by the binocular disparity depth estimation subnetwork in step 2 are as follows:

[0048] like Figure 3 As shown, first, the four pairs of left feature maps of different scales extracted by the two backbone networks are and right feature map Input the BasicBlock residual unit to compress the number of channels to reduce the amount of computation required for subsequent matching cost volume calculations. Then, the four pairs of left and right feature maps with compressed channel numbers are input into the matching cost volume calculation module of the corresponding scale to obtain two sets of matching cost volumes, namely, the left matching cost volume and the right matching cost volume. and right matching cost Then, these two sets of matching cost volumes are upsampled and accumulated in the order of scale from small to large to obtain the new left matching cost volume. and right matching cost exist Figure 3 In order to keep it simple, only the left matching cost volume is shown. The upsampling and accumulation process of the left matching cost volume is as follows:

[0049]

[0050] The symbol ↑ represents a two-fold trilinear interpolation upsampling operation. The upsampling and accumulation process for the right matching cost volume is similar to the above process. The accumulation of multiple matching cost volumes of different scales from small to large corresponds to the cost aggregation step in traditional stereo matching algorithms.

[0051] Finally, disparity regression is performed on these two sets of new matching cost volumes to obtain the left disparity map corresponding to the left and right feature maps respectively. and right disparity map The sizes of these two sets of disparity maps are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original images.

[0052] Disparity regression is used to calculate the disparity map corresponding to the binocular image based on the new matching cost volume. Traditional stereo matching methods usually use the WTA strategy when calculating disparity values, that is, using the argmax function to find the point with the highest similarity (lowest matching cost) of the pixel in the left image on the same epipolar line in the right image. However, the result obtained by the argmax function is discrete, and it is impossible to obtain a more accurate sub-pixel disparity value. Moreover, since the argmax function is not differentiable, argmax cannot be simply used to obtain the best matching point in the neural network. The disparity regression module uses the soft argmax function instead of the argmax function for approximate calculations. Soft argmax can be regarded as a softened and differentiable version of the argmax function. The soft argmax function first performs a softmax operation, exponentially normalizes the vector, and then performs a weighted sum on the subscript of each element in the vector (the weight is the normalized element value). The result is an approximate value of the argmax function, as shown in the following formula:

[0053]

[0054] After performing a soft argmax operation on each vector in the third dimension of the matching cost volume, the disparity map is obtained, as shown in the following formula:

[0055] D i,j =softargmax(V i,j, ·)

[0056] Where D represents the disparity map, V represents the matching cost volume, i represents the row coordinate, j represents the column coordinate, and the symbol · represents any value on the corresponding dimension of the tensor.

[0057] The implementation of the multi-scale color disparity fusion module in step 2 is as follows:

[0058] The structure of the multi-scale color disparity fusion module is as follows: Figure 5 As shown, it includes four color parallax fusion submodules (such as Figure 6 As shown in Figure 2, each color disparity fusion submodule first downsamples the left disparity map output by the binocular disparity depth estimation subnetwork three times, and then inputs the left disparity map into a convolution layer, a batch normalization layer, and an activation layer for feature extraction and noise removal to obtain a disparity map pyramid. It is then concatenated with the feature map in the feature pyramid in the channel direction, and then fused with features through another convolution layer to obtain an RGBD feature map of the corresponding scale. The multi-scale color disparity fusion module first downsamples the left disparity map output by the binocular disparity depth estimation subnetwork into the left disparity map. Perform three downsamplings and reduce the disparity value in the left disparity map by a corresponding multiple to obtain the disparity map pyramid. Then, the corresponding levels (i.e., corresponding scales) of the disparity map pyramid and the feature pyramid are fused through four color disparity fusion sub-modules to obtain the RGBD feature pyramid {N2, N3, N4, N5}.

[0059] The process of the segmentation head for spatial information enhancement in step 2 is as follows:

[0060] like Figure 7 As shown, we begin with the smallest feature map at the top of the RGBD feature pyramid (1 / 32 of the original image size), which embeds depth information. This feature map is then passed through three upsample stages to obtain a feature map that is 1 / 4 the size of the original image. The upsample stage consists of a 3×3 convolutional layer, a group normalization layer, a ReLU activation layer, and a parameter-free bilinear upsampling layer. Similarly, similar operations are performed on the RGBD feature maps at the two middle scales in the RGBD feature pyramid. These maps are each raised to 1 / 4 the size of the original image through several upsample stages. The feature map at the bottom of the RGBD feature pyramid is already 1 / 4 the size of the original image and is directly processed by a single convolutional layer (conv). The four upsampled feature maps are summed and fused. A spatial information transfer module is used to sequentially transfer the structural information of the target object in the fused feature map along the width and height dimensions, thereby enhancing the transmission line features in the feature map. Finally, the enhanced feature map is passed through a 1×1 convolutional layer, a fourfold bilinear upsampling operation, and a sigmoid activation function to output a transmission line segmentation probability map (logit) with the same length and width as the original image. The values ​​in the segmentation probability map represent the probability that the pixel at the corresponding position in the image belongs to the transmission line.

[0061] The spatial information transmission module is used to sequentially transmit the structural information of the target object in the feature map along the width and height directions in the spatial dimension (specifically, it is divided into four small modules connected in series, and the four small modules are respectively in the positive and negative directions of the width and height directions, that is, from top to bottom, from bottom to top, from left to right and from right to left to transmit information). It can effectively reduce the situation where the transmission line is interrupted or discontinuous in the segmentation result. This is because the transmission line occupies fewer pixels and the background is complex. It is easy to cause the segmentation result to have a transmission line interruption or discontinuity. Adding a spatial information transmission module can effectively reduce this situation. Taking the spatial information transmission module as an example of information transmission in the positive direction of the width direction of the feature map (i.e., from left to right), the steps are as follows:

[0062] (1) For a feature map of size c×h×w, first slice the feature map along the width direction to generate w two-dimensional feature map slices of size c×h, as shown in Figure 8 shown.

[0063] (2) The first slice is input into a convolution layer with a convolution kernel size of 5×1, and the output of the convolution layer is added to the next slice to generate a new slice; then the new slice is used as the input of the convolution layer with a convolution kernel size of 5×1, and the above operation is repeated to continuously pass information to subsequent slices, such as Figure 9 As shown in the figure, the update is completed until the last slice is updated; finally, the slices are spliced ​​in the original order to obtain the feature map after spatial information enhancement.

[0064] The operations along the reverse direction of the width direction, the positive direction of the height direction and the reverse direction of the height direction (ie from right to left, from top to bottom, from bottom to top) are the same as the above steps, differing only in the direction.

[0065] Step 3: Pre-train the binocular disparity depth estimation subnetwork for disparity estimation on the Flyingthings3D dataset.

[0066] Loss function L for disparity estimation training disp It consists of photometric loss, parallax smoothness loss, parallax consistency loss and parallax auxiliary loss, as shown in the following formula:

[0067] L disp =L photo +0.1L s +0.1L c +L int

[0068] Among them L photo Indicates the luminosity loss, L s represents the parallax smoothing loss, L c represents the parallax consistency loss, L int represents the parallax assistance loss.

[0069] Since the binocular disparity depth estimation subnetwork is used to calculate the left and right disparity maps respectively, these four losses are composed of two mirror (symmetrical) losses corresponding to the left and right disparity maps. (That is, the loss function calculation method for the left and right disparity maps is the same)

[0070] Photometric loss function L photo The image reconstruction (Warping) method is used to construct the image. The image reconstruction process is as follows: Let the left image in the binocular image be I l , the right picture is I r , the left disparity map predicted by the binocular disparity sub-network is D l, for each pixel in the left image (i is the row number, j is the column number), according to its corresponding disparity value In the right picture I r Find the corresponding sub-pixel point on the corresponding epipolar line After the position is determined, bilinear sampling is performed and all points in the left image are sampled to obtain the reconstructed left image. The reconstructed left image Compared with the original left image I l Compare the errors and calculate the reconstructed photometric loss of the left disparity image. The reconstructed photometric loss of the left disparity image is shown in the following formula:

[0071]

[0072] Where N is the number of valid pixels in the image, α is the proportional coefficient of the two losses, which can be taken as 0.85, and SSIM is the image structure similarity index function.

[0073] In order to keep the disparity map smooth locally, the disparity smoothing loss function is used to suppress the gradient of the disparity map to prevent a large number of gradient mutations and discontinuous disparity changes in the disparity map. The definition of the disparity smoothing loss function for the left disparity map is as follows:

[0074]

[0075] in is the gradient on the x-axis, is the gradient on the y-axis, and D is the disparity map.

[0076] In order to establish a connection between the left and right disparity maps calculated by the binocular disparity sub-network, the disparity consistency loss is introduced to constrain the consistency of the left and right disparity maps. The calculation process of the disparity consistency loss of the left disparity map is as follows: First, the left image I l According to the right disparity map, Warping operation is performed to map it to the right image perspective, and we get Then, according to the left disparity map Perform the Warping operation again and map it back to the left view to obtain The disparity consistency loss function of the left disparity map is defined as follows:

[0077]

[0078] The binocular disparity depth estimation sub-network outputs a total of four scales of left disparity maps In addition to the largest disparity map In addition to supervision, auxiliary supervision is also performed on the remaining three smaller-scale disparity maps to calculate the auxiliary disparity loss L intThe calculation process of the left parallax auxiliary loss is as follows: the left and right images of the binocular image are bilinearly downsampled three times to obtain the original size. Figure 1 Three pairs of left and right images of / 8, 1 / 16 and 1 / 32 and Then use the three disparity maps of corresponding scales right Reconstruct and get And calculate the L1 loss of these images after reconstruction. The definition of the disparity auxiliary loss of the left disparity map is as follows:

[0079]

[0080] The batch size during training was set to 5, and the learning rate adjustment strategy used a simple step strategy. The learning rate was set to 0.0005 for the first five epochs of training and then reduced to one-tenth of the original rate for the next five epochs. Data augmentation was performed during training using random cropping, with a 50% probability of upside-down flipping.

[0081] Step 4: The binocular image-based transmission line segmentation network constructed in step 2 is trained on the dataset obtained in step 1 for the dual tasks of disparity estimation and transmission line segmentation.

[0082] The loss function during training consists of image segmentation loss and disparity estimation loss. The overall loss function is defined as follows:

[0083] L total =L seg +0.1L disp

[0084] Among them L seg represents the image segmentation loss function, L disp is the disparity estimation loss function described in step 2. Image segmentation loss function L seg The definition of is as follows:

[0085] L seg =0.5L bce +0.5L dice

[0086] Where L dice Denotes the Diess loss function, L bce The binary cross entropy loss function supervises the task of power line image segmentation. bce The definition of is as follows:

[0087] L bce =∑ i,j w pos ·yi ·logx i +(1-y i )·log(1-x i )

[0088] where w pos Represents the positive and negative sample loss weight ratio coefficient, x i Indicates the predicted value of the network model for a single pixel sample, ranging from 0 to 1, y i Indicates the true category label of the pixel, the positive category is 1 and the negative category is 0. Since the proportion of transmission line pixels in the image is too small, the positive and negative sample weight ratio coefficient w in the binary cross entropy loss function is set during training. pos Set to 10.

[0089] The definition of the Dice loss function is as follows:

[0090]

[0091] where x i is the model’s predicted value for a single pixel sample, y i is the true category label of the pixel point, and smooth is the smoothing coefficient, which is used to maintain the numerical stability of the calculation result.

[0092] The adaptive Adam optimizer was selected for training, with a base learning rate of 0.0004, a number of epochs of 50, and an L2 regularization coefficient of 0.0005 to appropriately attenuate the network weights and prevent overfitting. During training, data augmentation was performed on the power line image training set, and the left and right images were randomly cropped simultaneously.

[0093] Step 5: Input the binocular transmission line image to be identified into the trained binocular image-based transmission line segmentation network model for identification and segmentation.

[0094] After the binocular transmission line image is input into the trained binocular image-based transmission line segmentation network model, the left disparity map output by the network model is converted to A fourfold bilinear upsampling is performed, and the disparity value of each point is also quadrupled accordingly, resulting in a left disparity map with the same resolution as the original image. The segmentation probability map output by the network model is also upsampled fourfold, and points with a probability greater than 0.5 are considered to belong to the power transmission line category.

[0095] The recognition results of the network model for some pictures are as follows Figure 10As shown in the figure, the first image on the left of each group of images is the left image of the binocular image, the middle image is the predicted disparity map corresponding to the left image, and the rightmost image is the transmission line segmentation result. The transmission line segmentation results show that even in the presence of relatively complex backgrounds, this method still correctly classifies the majority of pixels. The disparity map also well reflects the depth of each imaging point in the image. The depth of the transmission line in the image is significantly different from the background, and the outline of the transmission line can be clearly distinguished from the disparity map. Therefore, the network-predicted disparity map can provide guidance for the transmission line segmentation task.

[0096] This paper conducts comparative experiments on some other commonly used image segmentation methods based on deep learning networks. The compared methods include (1) FCN-8s: This method upsamples the results of different pooling layers and then combines these results to optimize the output, that is, improving the resolution of the segmentation results through deconvolution. (2) DeepLabV3+: This method uses a heavyweight network structure, uses a large number of dilated convolutions in the backbone network to improve the network's receptive field, and uses the dilated spatial pyramid pooling module to integrate feature maps of multiple different receptive fields. (3) Semantic FPN: This method uses FPN to extract multi-scale features. It is a lightweight asymmetric encoder-decoder structure that focuses on the detailed features of the image. (4) ParallelNet: A semantic segmentation method based on binocular images. This method mines the implicit depth information in binocular images and combines it with RGB features. (5) The method proposed by Zhang Di et al.: Based on ParallelNet, a cross-level feature guidance module and an attention module are added. The experimental results are shown in Table 1. From the experimental results, it can be seen that the method proposed in this paper has made great improvements in the transmission line segmentation task compared with other image segmentation methods listed in the table.

[0097] Table 1 Experimental results of different methods

[0098]

[0099] The present invention experimentally verifies the necessity of the pre-training process of the binocular disparity depth estimation subnetwork and tries another different training method, that is, without pre-training the binocular disparity depth estimation subnetwork, the joint training of the two tasks of image segmentation and disparity estimation is directly performed on the power transmission line segmentation dataset. The experimental results are shown in Table 2. Without pre-training the binocular disparity depth estimation subnetwork in step 3, the segmentation effect of the power transmission line is far different from that after pre-training. Directly conducting joint training of the two branches without pre-training will make the network convergence more difficult, and the two branches will have an adverse effect on each other, because the multi-scale color disparity fusion module needs to fuse the feature pyramid with the disparity map calculated by the binocular disparity depth estimation subnetwork, and guide the power transmission line segmentation task according to the information in the disparity map. When the disparity map is not accurate enough, it will introduce erroneous information, reducing the accuracy of power transmission line segmentation. This experiment shows that it is necessary to pre-train the binocular disparity depth estimation subnetwork.

[0100] Table 2 Comparison of experimental results with and without pre-training of the binocular disparity depth estimation sub-network

[0101]

[0102] The present invention verifies the effectiveness of the color parallax fusion module. Figure 1 The model structure shown in Figure 1 was modified to construct a variant that removes the color-disparity fusion module and directly connects the segmentation head to the feature pyramid in the left image to predict power line segmentation. The results of this variant model on the test set are shown in Table 3 below. Compared with the experimental results of the original model, the IoU of the power line segmentation decreased after removing the color-disparity fusion module. This indicates that the color-disparity fusion module has a significant impact on improving the accuracy of power line segmentation. The disparity map calculated by the binocular disparity depth estimation subnetwork can effectively guide the segmentation task, suppressing the interference of similar textures in the background to a certain extent, and improving the accuracy of the power line segmentation task.

[0103] Table 3 Experimental comparison of whether or not to include the color parallax fusion module

[0104]

[0105] The present invention conducted experiments using different configurations of the spatial information transfer module. The experimental results in Table 4 demonstrate the effectiveness of the spatial information transfer module. When information transfer is performed only along the width of the feature map, the IoU of the transmission line is improved by 1.11%, when information transfer is performed only along the height of the feature map, the IoU is improved by 1.35%, and when information transfer is performed in both the width and height directions, the IoU is improved by 1.68%. This demonstrates that the spatial information transfer module can improve the segmentation accuracy of long objects such as transmission lines, resulting in smoother and more continuous segmentation results.

[0106] Table 4 Experimental results of spatial information transfer module

[0107]

Claims

1. A method for segmenting power lines based on binocular images, characterized in that: The following steps are involved: Step 1: Obtain a binocular power transmission line image segmentation dataset, perform distortion correction and epipolar correction on the images in the dataset, and obtain the corrected left and right images, that is, the updated dataset; Step 2: Construct a binocular image-based power transmission line segmentation network, which includes two backbone networks, a feature pyramid network, a binocular disparity depth estimation subnetwork, a multi-scale disparity color fusion module, and a spatial information enhanced segmentation head; wherein: The two backbone networks are used to extract features from the corrected left and right images respectively. Specifically, the left and right images are input into two parallel backbone networks respectively to obtain four left feature maps of different scales. and right feature map Their sizes are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original image; The binocular disparity depth estimation sub-network is used to calculate the disparity map corresponding to the left feature map and the right feature map extracted by the two backbone networks using the Coarse-to-Fine strategy; The feature pyramid network is used to extract the four left feature maps of different scales from the backbone network. Perform top-down information transfer to obtain the feature pyramid corresponding to the left image The multi-scale color-disparity fusion module is used to perform multi-scale fusion of the feature pyramid and the left disparity map to obtain an RGBD feature pyramid {N2, N3, N4, N5} embedded with depth information; The segmentation head with spatial information enhancement is used to calculate the final transmission line segmentation result based on the RGBD feature pyramid, and contains a spatial information transfer module, which is used to transfer the structural information of the target object in the feature map along the width and height directions; Step 3: Perform unsupervised pre-training of the binocular disparity depth estimation subnetwork on the FlyingThings3D dataset for the disparity estimation task; Step 4: The binocular image-based transmission line segmentation network constructed in step 2 is jointly trained on the disparity estimation and transmission line segmentation tasks on the updated dataset obtained in step 1 to obtain a trained binocular image-based transmission line segmentation network. Step 5: Input the binocular transmission line image to be identified into the trained binocular image-based transmission line segmentation network for identification, and obtain the disparity map and transmission line identification and segmentation results.

2. The method for segmenting a power transmission line based on binocular images according to claim 1, wherein: In step 2, the binocular disparity depth estimation sub-network uses the Coarse-to-Fine strategy to calculate the disparity map for the left feature map and the right feature map extracted by the two backbone networks. The specific process is: the four pairs of left feature maps of different scales extracted by the two backbone networks are combined. and right feature map The number of channels is compressed by the BasicBlock residual unit; then the four pairs of left and right feature maps after channel compression are input into the matching cost volume calculation module of the corresponding scale to obtain two sets of matching cost volumes, namely the left matching cost volume and right matching cost Then, these two sets of matching cost volumes are upsampled and accumulated in the order of scale from small to large to obtain the new left matching cost volume. and right matching cost Finally, the new matching cost volume is subjected to disparity regression calculation to obtain the left disparity map corresponding to the left and right images in the data set obtained in step 1. and right disparity map The sizes of these two sets of disparity maps are (1 / 4, 1 / 8, 1 / 16, 1 / 32) of the original images.

3. The method for segmenting a power transmission line based on binocular images according to claim 1, wherein: In step 2, the multi-scale color disparity fusion module implements the process of: inputting the feature map of the left disparity map output by the binocular disparity depth estimation subnetwork after three downsampling operations into a convolution layer, a batch normalization layer, and an activation layer to obtain a disparity map pyramid, and fusing the feature pyramid with the disparity map pyramid at the corresponding scale to obtain an RGBD feature pyramid embedded with depth information.

4. The method for segmenting a power transmission line based on binocular images according to claim 1, wherein: In step 2, the spatial information enhanced segmentation head increases the size of the RGBD feature pyramid embedded with depth information to 1 / 4 of the size of the original image through several upsampling layers, and then adds and fuses the four upsampled feature maps to obtain a fused feature map; then, through the spatial information transmission module, the structural information of the target object in the fused feature map is sequentially transmitted along the width and height directions in the spatial dimensions to obtain a feature enhanced feature map; and a 1×1 convolution layer and a four-fold bilinear upsampling operation are used on the feature enhanced feature map to output a segmentation probability map with the same length and width as the original input image.

5. The method for segmenting a power transmission line based on binocular images according to claim 2, wherein: The matching cost volume calculation module has two sets of inputs and two sets of outputs. The inputs are the left and right feature maps, and the outputs are the left and right matching cost volumes. The calculation process of the left matching cost volume is as follows: first, the right feature map F with the size h×w×c is right Perform dimension conversion to obtain F with size h×c×w r ' igt , then F r ' igt Input a 1×1 convolution layer to obtain a feature map K of size h×c×w right , and at the same time the left feature map F left Also input another 1×1 convolutional layer to obtain a feature map Q of scale h×w×c left , and finally Q left and K right Perform batch matrix multiplication to calculate the left matching cost volume V of size h×w×w left .

6. The method for segmenting a power transmission line based on binocular images according to claim 2, wherein: The disparity regression calculation refers to using a soft argmax operation to calculate the disparity map corresponding to the binocular image according to the matching cost volume.

7. The method for segmenting a power transmission line based on binocular images according to claim 1, wherein: The loss function used in the network training in step 4 consists of a disparity estimation loss function and an image segmentation loss function, wherein the disparity estimation loss function consists of a photometric loss function, a disparity smoothing loss function, a disparity consistency loss function and an auxiliary disparity loss function, and the image segmentation loss function consists of a Dice loss function and a binary cross entropy loss function.

Images