A method for detecting bird nests in contact lines based on deformable sampling

Abstract

The present invention discloses a method for detecting bird nests in a contact network based on deformable sampling, which relates to the field of image processing technology and includes the following steps: S1, using a deformable sampling method to construct a bird nest detection network model; S2, using a sample set of defective bird nests in the contact network to perform branch auxiliary training on the bird nest detection network model; S3, using a sample set of defective bird nests in the contact network to perform main network independent training on the bird nest detection network model trained with the auxiliary network; S4, inputting the image of the contact network bird nest to be identified into the trained bird nest detection network model to perform bird nest detection. The present invention designs a deformable downsampling model, realizes biased downsampling in space, and outputs coordinate offset features and classification features at the same time, realizing the decoupling of classification prediction and regression prediction; uses a training method of guiding the main network with an auxiliary branch, and realizes the positioning of the local area where the bird nest is located without using an additional network during detection.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, image processing and target detection, and more specifically to a method for detecting bird nests in a contact network based on deformable sampling. Background Art

[0002] The overhead contact network is a high-voltage transmission line that is set up in a zigzag shape above the rails in an electrified railway. It is used by the pantograph to draw current. It is responsible for the important task of directly transmitting the electric energy obtained from the traction substation to the electric locomotive. Therefore, the quality and working condition of the overhead contact network will directly affect the transportation capacity of the electrified railway. In railway transportation, the overhead contact network is required to ensure good supply of electric energy to the electric locomotive under any conditions, and to ensure the safe and high-speed operation of the electric locomotive on the line.

[0003] The hard beams and pillars of the contact network are places where birds frequently build nests. From April to November every year, many birds like to build their nests on the hard beams or pillars of the contact network. Some of the branches brought by the birds are long. When the branches fall on the contact line, it is very easy to break the high-speed pantograph. In addition, the nesting materials brought by the birds are often mixed with conductors such as iron wire. These conductors can easily cause short circuits in the contact network equipment, thereby endangering the safety of the railway. In serious cases, the train can be forced to stop. At the same time, when there are other foreign objects on the contact network, the pantograph will also be broken. Therefore, the presence of bird nests on the contact network poses a serious safety hazard. How to efficiently and accurately detect bird nests is of great practical significance.

[0004] At present, the intelligent detection methods for bird nests in contact networks include methods based on traditional feature extraction and machine learning (such as oriented gradient histograms and support vector machines), two-stage detection methods based on Faster R-CNN, single-stage methods based on the Yolo series, and combined design methods. The limitation of traditional detection methods is that feature design is difficult and it is difficult to adapt to the complex and changeable background environment in contact network images. Methods based on deep neural networks show great superiority in feature extraction. The difficulty of detecting bird nests in contact networks lies in the complex background and the small pixel ratio of bird nests in the entire picture, about 3 / 1000. It is difficult to meet the accuracy requirements by directly using deep learning methods. At present, there is a combination design idea, such as the patent with publication number CN112949634A, which trains two-level Yolo. The first-level Yolo detector screens the area of ​​interest, and the second-level Yolo detects and judges the output of the first-level Yolo. The complexity of the two-level YoLo model is high, which increases the difficulty of deployment.

[0005] At present, there are two main difficulties in the detection of bird nests in the contact network:

[0006] (1) If Figure 1As shown in the figure, in the panoramic image, the background is extremely complex, while the pixel occupied by the target bird's nest is very small, about 3 / 1000, which belongs to the category of small target detection and is a key and difficult problem in the field of target detection.

[0007] (2) If Figure 2 and Figure 3 As shown in the figure, the shapes of bird nests vary greatly, some are very inconspicuous and close to the edge of the image, which increases the difficulty of feature extraction and detection and recognition. Summary of the invention

[0008] In order to overcome the defects existing in the above-mentioned prior art, the present invention discloses a contact network bird nest detection method based on deformable sampling. The purpose of the present invention is to detect bird nests of small size and various shapes in large-scale panoramic images, so as to improve the detection rate of contact network bird nest defects.

[0009] In order to achieve the above objectives, the technical solution adopted by the present invention is:

[0010] A method for detecting bird nests in a contact network based on deformable sampling comprises the following steps:

[0011] 1. Build a network model

[0012] S1. Use deformable sampling method to build a feature extractor containing the main network Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls and regression detection head H Bbox Bird nest detection network model;

[0013] Preferably, in the bird nest detection network model, the main network feature extractor Net sub First classification feature for extracting panoramic images F cls1 and the first regression feature F Bbox1 ;

[0014] The auxiliary network feature extractor Net obj Second classification feature for extracting local images F cls2 and the second regression feature F Bbox2 ;

[0015] The classification detection head H cls For the first classification featureF cls1 and the second classification feature F cls2 Determine whether the target area is a bird's nest or background;

[0016] The regression detection head H Bbox For the first regression feature F Bbox1 and the second regression feature F Bbox2 Compute the coordinates of the bounding box.

[0017] Preferably, the main network feature extractor Net sub and auxiliary network feature extractor Net obj Both include convolution modules and deformable downsampling modules;

[0018] The convolution module and the deformable downsampling module are superimposed as basic units to output coordinate offset features and classification features.

[0019] Preferably, the convolution module includes a convolution layer and a batch normalization layer;

[0020] The deformable downsampling module is used to compress the channel information of the input features, and then generate coordinate offset features through the convolution layer Conv2d and the fully connected FC, and finally select feature values ​​from the original image according to the coordinate offset features and output them.

[0021] Preferably, compressing the channel information of the input feature is specifically: compressing the channel information from H×W×C to H×W×3;

[0022] The size of the coordinate offset feature is H / 2×W / 2×2;

[0023] The feature value is selected from the original image according to the coordinate offset feature and outputted, wherein the output feature size is H / 2×W / 2×C;

[0024] Among them, H represents the height, W represents the width, and C represents the number of channels.

[0025] 2. Branch auxiliary training stage

[0026] S2. Using the image of the contact network bird nest defect sample set, the branch auxiliary training of the bird nest detection network model is performed, including: using the feature similarity loss function and the coordinate regression loss function to update the main network feature extractor Net sub and the auxiliary network feature extractor Net obj, get the bird nest detection network model after auxiliary network training;

[0027] In step S2, the images of the contact network bird nest defect sample set are input into the constructed bird nest detection network model for branch auxiliary training; during the branch auxiliary training process, the panoramic images in the contact network bird nest defect sample set images are input into the main network feature extractor Net sub In the example, the local image is input to the auxiliary network feature extractor Net obj In the main network feature extractor Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls and regression detection head H Bbox training, and using the auxiliary network feature extractor Net obj Guide the main network feature extractor Net sub Learning local region features of bird nests from panoramic images.

[0028] During the branch auxiliary training process in step S2, the main network feature extractor is used Net sub Perform feature extraction on the panoramic image to obtain the first classification feature F cls1 ; Crop the local image where the bird's nest is located in the panoramic image, and use the auxiliary network feature extractor Net obj Extract features from the local image to obtain the second classification feature F cls2 , through the loss function, let the first classification feature F cls1 and the second classification feature F cls2 equal.

[0029] Preferably, the branch auxiliary training comprises the following steps:

[0030] 1. Traverse the picture and process the image

[0031] S21, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image;

[0032] 2. Feature extraction

[0033] S22, inputting the panoramic image into the main network feature extractor Net sub In the first classification feature F cls1 and the first regression feature F Bbox1 ;

[0034] Input the local image into the auxiliary network feature extractor Net obj In the second classification feature F cls2 and the second regression feature F Bbox2 ;

[0035] 3. Auxiliary branch (auxiliary network feature extractor Net obj ) Parameter update

[0036] S23, the second classification feature F cls2 Input to the classification detection head H cls , get the classification prediction result of the local image;

[0037] The second regression feature F Bbox2 Input to the regression detection head H Bbox , get the coordinate prediction result of the local image;

[0038] According to the classification prediction results and coordinate prediction results of the local image, the loss function is calculated and the auxiliary network feature extractor is updated. Net obj Parameters;

[0039] 4. Main network (main network feature extractor Net sub ) Parameter update

[0040] S24, using the first classification feature F cls1 and the second classification feature F cls2 Constructing feature similarity loss function L FSim , guiding the main network feature extractor Net sub Learning local area features where the bird's nest is located from the panoramic image;

[0041] At the same time, construct the coordinate regression loss function L Bbox1 , and use the coordinate regression loss function LBbox1 And feature similarity loss function L Fsim The calculation results are used to update the main network feature extractor Net sub Parameters;

[0042] Preferably, step S24 further includes:

[0043] Using the feature similarity loss function, the first classification feature F cls1 and the second classification feature F cls2 equal.

[0044] 5. Determine whether the auxiliary training phase is over

[0045] S25, determine whether the branch auxiliary training is completed, if so, enter the main network independent training, if not, return to step S21.

[0046] 3. Main network independent training stage

[0047] S3, using the contact network bird nest defect sample set image, the bird nest detection network model trained by the auxiliary network is trained independently by the main network, including: using the classification loss function and the coordinate regression loss function, and updating the main network feature extractor again Net sub , get the trained bird nest detection network model;

[0048] In step S3, the images of the contact network bird nest defect sample set are input into the constructed bird nest detection network model for independent training of the main network; during the independent training of the main network, the panoramic images and local images in the contact network bird nest defect sample set images are input into the main network feature extractor after branch auxiliary training. Net sub In the main network feature extractor Net sub 、 Classification detection head H cls and regression detection head H Bbox training.

[0049] Preferably, the main network independent training comprises the following steps:

[0050] S31, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image;

[0051] S32, inputting the panoramic image and the local image into the main network feature extractor after updating the parameters Netsub In the first classification feature F cls1 and the first regression feature F Bbox1 ;

[0052] S33, the first classification feature F cls1 and the first regression feature F Bbox1 Input to the classification detection head H cls and regression detection head H Bbox In the above example, we obtain the classification prediction and coordinate prediction results of the panoramic image and the local image, and calculate the classification loss function based on the prediction results. L cls1 And the coordinate regression loss function L Box1 , and based on the calculation results, update the main network feature extractor again Net sub Parameters;

[0053] S34, judging whether the independent training of the main network is completed, if so, entering the step of detecting the bird nest of the contact network bird nest to be identified, if not, returning to step S31.

[0054] In steps S2 and S3, the loss function includes:

[0055] The total loss function of feature similarity is:

[0056] ;

[0057] The feature similarity loss function is:

[0058] ;

[0059] The total classification loss function is:

[0060] ;

[0061] The classification loss function is:

[0062] ;

[0063] The classification prediction results are:

[0064] ;

[0065] The coordinate regression loss function is:

[0066] ;

[0067] in, represents the total loss function; Represents the weight of feature similarity loss; represents the feature similarity loss function; represents the coordinate regression loss function; Minimize arrive Mean square error; , and Represents categorical features; represents the total loss function; Represents the weight of classification loss; represents the classification loss function; Represents the classification prediction result; Represents the true category label of the image; represents the classification detection head; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; Represents the diagonal length of the smallest rectangle that contains the predicted box and the true box; Represents the intersection-over-union ratio of the real box and the predicted box, ,

[0068] ; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

[0069] 4. Bird nest detection

[0070] S4. Input the image of the contact network bird nest to be identified into the trained bird nest detection network model to perform bird nest detection.

[0071] Preferably, step S4 comprises:

[0072] Using the main network feature extractor in the trained bird nest detection network model Net sub 、 Classification detection head H cls and regression detection head H Bbox Identify bird nests in the contact network bird nests to be identified image.

[0073] The key points of the present invention are:

[0074] 1. Use the auxiliary network to guide the training of the main network so that the main network can locate the local area where the bird's nest is located during the feature extraction process.

[0075] 2. The method of the present invention mentions a two-stage training method: an auxiliary network guided training stage and a main network independent training stage.

[0076] 3. In step S1, a plug-and-play deformable downsampling module (DDS-Module) was constructed to output coordinate offset features and achieve selective downsampling in space.

[0077] 4. The feature extractor constructed by the "convolution module + deformable downsampling module" in step S1 outputs both coordinate offset features and classification features, so the inputs of the classification detection head and the regression detection head are different.

[0078] 5. Step S2 uses feature similarity and coordinate regression loss to guide the main network feature extractor during the auxiliary training phase Net sub Parameter update. Main network feature extractor Net sub The first classification feature F cls1 No input classification detection head H cls , but with the auxiliary network feature extractor Net obj The second classification feature F cls2 Construct a feature similarity loss function to guide the main network feature extractor Net sub Learn the local area features of the bird's nest from the panoramic image.

[0079] 6. In the independent training phase, the classification loss and coordinate regression loss are used to guide the main network feature extractor. Net sub Parameter update. Auxiliary network feature extractor Net obj No longer guiding training, both panoramic images and local images are input into the main network feature extractor Net sub , feature similarity loss calculation is no longer performed.

[0080] Beneficial effects of the present invention:

[0081] 1. The present invention designs a deformable downsampling model, realizes biased downsampling in space, and outputs coordinate offset features and classification features at the same time, thereby realizing the decoupling of classification prediction and regression prediction.

[0082] 2. The present invention uses an auxiliary branch to guide the training method of the main network, and achieves the positioning of the local area where the bird's nest is located without using an additional network during detection, thereby improving the detection accuracy.

[0083] 3. The present invention designs a single-stage deep network, which uses auxiliary branches for guidance during the training process, so that the area of ​​interest can be located using only one level of network, reducing the difficulty of deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is the image of the bird nest defect in the contact network;

[0085] Figure 2 It is a bird's nest image;

[0086] Figure 3 This is another form of bird's nest image;

[0087] Figure 4 This is a network structure diagram of the present invention;

[0088] Figure 5 This is the branch auxiliary training phase process of the present invention;

[0089] Figure 6 This is the independent training phase process of the main network of the present invention;

[0090] Figure 7 Schematic diagram of the convolution module of the present invention;

[0091] Figure 8 This is a structural schematic diagram of a deformable downsampling module of the present invention;

[0092] Fig. 9 This is a schematic diagram of spatial equal probability downsampling of the present invention;

[0093] Fig.10 A schematic diagram of compressing the features of the coverage area according to the present invention;

[0094] Fig.11 This is a schematic diagram of deformable downsampling positioning output of the present invention;

[0095] Fig.12 It is a structural schematic diagram of the feature extractor of the present invention;

[0096] Fig.13 This is a schematic diagram of image cropping according to the present invention. DETAILED DESCRIPTION

[0097] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings to fully understand the purpose, characteristics and effects of the present invention.

[0098] Example 1

[0099] A method for detecting bird nests in a contact network based on deformable sampling comprises the following steps:

[0100] S1. Use deformable sampling method to build a feature extractor containing the main network Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls and regression detection head H Bbox Bird nest detection network model;

[0101] S2. Using the image of the contact network bird nest defect sample set, the branch auxiliary training of the bird nest detection network model is performed, including: using the feature similarity loss function and the coordinate regression loss function to update the main network feature extractor Net sub and the auxiliary network feature extractor Net obj , get the bird nest detection network model after auxiliary network training;

[0102] S3, using the contact network bird nest defect sample set image, the bird nest detection network model trained by the auxiliary network is trained independently by the main network, including: using the classification loss function and the coordinate regression loss function, and updating the main network feature extractor again Net sub , get the trained bird nest detection network model;

[0103] S4. Input the image of the contact network bird nest to be identified into the trained bird nest detection network model to perform bird nest detection.

[0104] Example 2

[0105] This embodiment is further described on the basis of the embodiment 1. The network structure of the present invention is as follows: Figure 4 As shown in , the network training is divided into two stages: branch auxiliary training stage and main network independent training stage, as shown in Figure 5 and Figure 6 shown.

[0106] The method of the present invention involves 4 modules, namely: main network feature extractor Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls 、Regression detection head H Bbox The functions of each module are shown in Table 1. The final detection network of the method of the present invention consists of the main network feature extractor Net sub , Classification detection head H cls and regression detection head H Bbox Composition, auxiliary network feature extractor Net obj Only participating in model training and not participating in the final detection task will not increase the difficulty of model deployment.

[0107] Table 1 Module function description

[0108]

[0109] The method of the present invention involves four modules: the main network feature extractor Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls and regression detection head H Bbox The main network feature extractor is composed of a convolutional module (ConvModule) and a deformable down-sampling module (DDS-Module). Net sub and auxiliary network feature extractor Net obj The convolution module structure is as follows: Figure 7 As shown in Figure 1, it consists of 2D convolution and batch normalization. The structure of the deformable downsampling module is shown in Figure 1. Figure 8 As shown in the figure, the channel information of the input feature is first compressed from H×W×C to H×W×3, and then the coordinate offset feature is produced through the convolutional layer Conv2d and the fully connected FC with a size of H / 2×W / 2×2. Finally, the feature value is selected from the original image according to the coordinate offset feature, and the output feature size is: H / 2×W / 2×C.

[0110] The role of the deformable downsampling module is to turn spatial equal probability downsampling into a preferred choice. Fig. 9 As shown, the dotted box slides in space with a certain step length to compress the features of the covered area. The result is as follows Fig.10 As shown in . The deformable downsampling is directly positioned to the coordinate offset matrix as shown in Fig. 9The points in the solid line box will output the values ​​of the covered area. The results are as follows Fig.11 shown.

[0111] Main network feature extractor Net sub and auxiliary network feature extractor Net obj The composition of Fig.12 As shown, “ConvModule + DDS-Module” is superimposed as a basic unit, and its output contains two features: coordinate offset feature and classification feature.

[0112] The detection head of the present invention includes a classification detection head H cls and coordinate regression detection head H Bbox , the Decouple-Head module of YoloV8 is adopted.

[0113] Example 3

[0114] This embodiment further elaborates on the basis of embodiment 2. Common methods for solving the problem of small target detection include: improving image resolution, image segmentation, image enhancement, changing annotation boxes, improving feature extraction methods, improving loss functions, etc. In the face of a panoramic image, the human visual system can quickly locate the area where the bird's nest is located. Inspired by this, the present invention designs the following method: Figure 4 The network structure shown, with the main network feature extractor Net sub Perform feature extraction on the panoramic image to obtain the first classification feature F cls1 ; Crop out the local area where the bird's nest is located in the panoramic image and use the auxiliary network feature extractor Net obj Extract features from the local image to obtain the second classification feature F cls2 , through the loss function, let the first classification feature F cls1 and the second classification feature F cls2 equal, thus making the main network feature extractor Net sub In the process of feature extraction, the local area where the bird's nest is located can be located.

[0115] like Figure 5 As shown, branch auxiliary training includes the following steps:

[0116] 1. Traverse the picture and process the image

[0117] S21, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image;

[0118] In the above steps, the training image of the input model is cropped, and the cropping method is as follows Fig.13 As shown in the figure, the local image where the target is located in the panoramic image is cropped out as an auxiliary network feature extractor Net obj The training data is the railway contact network bird nest defect dataset with a resolution of 2560×2048.

[0119] 2. Feature extraction

[0120] S22, inputting the panoramic image into the main network feature extractor Net sub In the first classification feature F cls1 and the first regression feature F Bbox1 ;

[0121] Input the local image into the auxiliary network feature extractor Net obj In the second classification feature F cls2 and the second regression feature F Bbox2 ;

[0122] In the above steps, the panoramic image is input into the main network feature extractor Net sub , get the first classification feature F cls1 and the first regression feature F Bbox1 , input the local image into the auxiliary network feature extractor Net obj , get the second classification feature F cls2 and the second regression feature F Bbox2 .

[0123] ;

[0124] in, represents a panoramic image, represents a local image, and It is the concatenation of all coordinate offset features in the deformable downsampling module.

[0125] 3. Auxiliary branch (auxiliary network feature extractor Netobj ) Parameter update

[0126] S23, the second classification feature F cls2 Input to the classification detection head H cls , get the classification prediction result of the local image;

[0127] The second regression feature F Bbox2 Input to the regression detection head H Bbox , get the coordinate prediction result of the local image;

[0128] According to the classification prediction results and coordinate prediction results of the local image, the loss function is calculated and the auxiliary network feature extractor is updated. Net obj Parameters;

[0129] In the above steps, the second classification feature F cls2 , the second regression feature F Bbox2 Input classification detection head separately H cls 、 Regression detection head H Bbox Get the local image X l Classification prediction and coordinate prediction, calculate the loss function based on the prediction results, minimize the loss function, and update the auxiliary network feature extractor based on the gradient information Net obj Parameters.

[0130] In the step S23, the loss function includes:

[0131] ;

[0132] in, represents the total loss function; Represents the weight of classification loss; represents the classification loss function; represents the coordinate regression loss function; Represents the classification prediction result; Indicates the true category label of the image, where 0 represents a picture without a bird's nest and 1 represents a picture with a bird's nest; represents the classification detection head; Represents categorical features; Represents the intersection-over-union ratio of the real box and the predicted box; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; represents the diagonal length of the smallest rectangle that contains these two boxes; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

[0133] 4. Main network (main network feature extractor Net sub ) Parameter update

[0134] S24, using the first classification feature F cls1 and the second classification feature F cls2 Constructing feature similarity loss function L FSim , guiding the main network feature extractor Net sub Learning local area features where the bird's nest is located from the panoramic image;

[0135] At the same time, construct the coordinate regression loss function L Bbox1 , and use the coordinate regression loss function L Bbox1 And feature similarity loss function L Fsim The calculation results are used to update the main network feature extractor Net sub Parameters;

[0136] The fundamental purpose of the method of the present invention is to use an auxiliary network feature extractor Net obj The main network feature extractor is guided by Net sub It can learn the characteristics of the local area where the bird nest is located, ignoring the background interference. The implementation method is to let the main network feature extractor Net sub and auxiliary network feature extractor Net obj Have the same classification feature, namely the first classification feature F cls1 and the second classification feature F cls2 The implementation method of the present invention is to convert the second classification feature F cls2 As the goal, minimize the first classification feature F cls1 To the second classification feature Fcls2 Mean Square Error (MSE). Therefore, the loss of the main network in the method of the present invention includes feature similarity loss and regression loss, and the specific formula is as follows.

[0137] In the step S24, the loss function includes:

[0138] ;

[0139] in, represents the total loss function; Represents the weight of feature similarity loss; represents the feature similarity loss function; represents the coordinate regression loss function; Minimize arrive Mean square error; and Represents categorical features; Represents the intersection-over-union ratio of the real box and the predicted box; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; represents the diagonal length of the smallest rectangle that contains these two boxes; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

[0140] 5. Determine whether the auxiliary training phase is over

[0141] S25, determine whether the branch auxiliary training is completed, if so, enter the main network independent training, if not, return to step S21.

[0142] The model training process of the present invention is divided into two stages, an auxiliary training stage and a main network independent training stage. The present invention trains 500 rounds in total, the first 490 rounds are auxiliary training, and the last 10 rounds are independent training for the main network.

[0143] If the auxiliary training phase has not ended, return to S21.

[0144] Example 4

[0145] This embodiment is further described on the basis of embodiment 3. Figure 6 As shown, the independent training of the main network includes the following steps:

[0146] S31, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image;

[0147] S32, inputting the panoramic image and the local image into the main network feature extractor after updating the parameters Net sub In the first classification feature F cls1 and the first regression feature F Bbox1 ;

[0148] S33, the first classification feature F cls1 and the first regression feature F Bbox1 Input to the classification detection head H cls and regression detection head H Bbox In the above example, we obtain the classification prediction and coordinate prediction results of the panoramic image and the local image, and calculate the classification loss function based on the prediction results. L cls1 And the coordinate regression loss function L Box1 , and based on the calculation results, update the main network feature extractor again Net sub Parameters;

[0149] S34, judging whether the independent training of the main network is completed, if so, entering the step of detecting the bird nest of the contact network bird nest to be identified, if not, returning to step S31.

[0150] Because the final detection model of the method of the present invention is "main network feature extractor Net sub +Classification detection head H cls +Regression Detection Head H Bbox ", so the last 10 times only train the main network and two detection heads, and the auxiliary network feature extractor Net obj No longer trained, both panoramic images and local images are input into the main network feature extractor Net sub Therefore, the feature similarity loss is no longer calculated. The loss function of the independent training stage is as follows.

[0151] During the independent training of the main network, its loss function includes:

[0152] ;

[0153] in, represents the total loss function; Represents the weight of classification loss; represents the classification loss function; represents the coordinate regression loss function; Represents the classification prediction result; Indicates the true category label of the image, where 0 represents a picture without a bird's nest and 1 represents a picture with a bird's nest; represents the classification detection head; Represents categorical features; Represents the intersection-over-union ratio of the real box and the predicted box; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; represents the diagonal length of the smallest rectangle that contains these two boxes; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

[0154] In summary, in steps S2 and S3, the loss function includes:

[0155] The total loss function of feature similarity is:

[0156] ;

[0157] The feature similarity loss function is:

[0158] ;

[0159] The total classification loss function is:

[0160] ;

[0161] The classification loss function is:

[0162] ;

[0163] The classification prediction results are:

[0164] ;

[0165] The coordinate regression loss function is:

[0166] ;

[0167] in, represents the total loss function; Represents the weight of feature similarity loss; represents the feature similarity loss function; represents the coordinate regression loss function; Minimize arrive Mean square error; , and Represents categorical features; represents the total loss function; Represents the weight of classification loss; represents the classification loss function; Represents the classification prediction result; Represents the true category label of the image; represents the classification detection head; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; Represents the diagonal length of the smallest rectangle that contains the predicted box and the true box; Represents the intersection-over-union ratio of the real box and the predicted box, ,

[0168] ; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

[0169] Example 5

[0170] This embodiment is further described on the basis of the fourth embodiment. In the process of detecting the bird nest in the contact network bird nest to be identified image, the main network feature extractor in the bird nest detection network model is used. Net sub 、 Classification detection head H cls and regression detection head H Bbox Identify bird nests in the contact network bird nests to be identified image.

[0171] The above is a specific description of the implementation mode of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention, and these equivalents or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for detecting bird nests in contact lines based on deformable sampling, characterized in that: The following steps are involved: S1. Use deformable sampling method to build a feature extractor containing the main network Net sub 、 Auxiliary Network Feature Extractor Net obj 、 Classification detection head H cls and regression detection head H Bbox Bird nest detection network model; S2. Using the image of the contact network bird nest defect sample set, the branch auxiliary training of the bird nest detection network model is performed, including: using the feature similarity loss function and the coordinate regression loss function to update the main network feature extractor Net sub and the auxiliary network feature extractor Net obj , get the bird nest detection network model after auxiliary network training; The S2 step includes: S21, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image; S22, inputting the panoramic image into the main network feature extractor Net sub In the first classification feature F cls1 and the first regression feature F Bbox1 ; Input the local image into the auxiliary network feature extractor Net obj In the second classification feature F cls2 and the second regression feature F Bbox2 ; S23, the second classification feature F cls2 Input to the classification detection head H cls , get the classification prediction result of the local image; The second regression feature F Bbox2 Input to the regression detection head H Bbox , get the coordinate prediction result of the local image; According to the classification prediction results and coordinate prediction results of the local image, the loss function is calculated and the auxiliary network feature extractor is updated. Net obj Parameters; S24, using the first classification feature F cls1 and the second classification feature F cls2 Constructing feature similarity loss function L FSim , guiding the main network feature extractor Net sub Learning local area features where the bird's nest is located from the panoramic image; At the same time, construct the coordinate regression loss function L Bbox1 , and use the coordinate regression loss function L Bbox1 And feature similarity loss function L Fsim The calculation results are used to update the main network feature extractor Net sub Parameters; S25, judging whether the branch auxiliary training is finished, if so, entering the main network independent training, if not, returning to step S21; S3, using the contact network bird nest defect sample set image, the bird nest detection network model trained by the auxiliary network is trained independently by the main network, including: using the classification loss function and the coordinate regression loss function, and updating the main network feature extractor again Net sub , get the trained bird nest detection network model; The S3 steps include: S31, traversing the contact network bird nest defect sample set images, cropping to obtain a panoramic image and a local image where the target is located in the panoramic image; S32, inputting the panoramic image and the local image into the main network feature extractor after updating the parameters Net sub In the first classification feature F cls1 and the first regression feature F Bbox1 ; S33, the first classification feature F cls1 and the first regression feature F Bbox1 Input to the classification detection head H cls and regression detection head H Bbox In the above example, we obtain the classification prediction and coordinate prediction results of the panoramic image and the local image, and calculate the classification loss function based on the prediction results. L cls1 And the coordinate regression loss function L Box1 , and based on the calculation results, update the main network feature extractor again Net sub Parameters; S34, judging whether the independent training of the main network is completed, if so, entering the step of detecting the bird nest of the contact network bird nest to be identified, if not, returning to step S31; S4. Input the image of the contact network bird nest to be identified into the trained bird nest detection network model to perform bird nest detection.

2. A method for detecting bird nests in a contact network based on deformable sampling as claimed in claim 1, characterized in that: The main network feature extractor Net sub First classification feature for extracting panoramic images F cls1 and the first regression feature F Bbox1 ; The auxiliary network feature extractor Net obj Second classification feature for extracting local images F cls2 and the second regression feature F Bbox2 ; The classification detection head H cls For the first classification feature F cls1 and the second classification feature F cls2 Determine whether the target area is a bird's nest or background; The regression detection head H Bbox For the first regression feature F Bbox1 and the second regression feature F Bbox2 Compute the coordinates of the bounding box.

3. A method for detecting bird nests in a contact network based on deformable sampling as claimed in claim 1, characterized in that: The main network feature extractor Net sub and auxiliary network feature extractor Net obj Both include convolution modules and deformable downsampling modules; The convolution module and the deformable downsampling module are superimposed as basic units to output coordinate offset features and classification features.

4. A method for detecting bird nests in a contact network based on deformable sampling as claimed in claim 3, characterized in that: The convolution module includes a convolution layer and a batch normalization layer; The deformable downsampling module is used to compress the channel information of the input features, and then generate coordinate offset features through the convolution layer Conv2d and the fully connected FC, and finally select feature values ​​from the original image according to the coordinate offset features and output them.

5. A method for detecting bird nests in a contact network based on deformable sampling as claimed in claim 4, characterized in that: The channel information of the input feature is compressed, specifically: the channel information is compressed from H×W×C to H×W×3; The size of the coordinate offset feature is H / 2×W / 2×2; The feature value is selected from the original image according to the coordinate offset feature and outputted, wherein the output feature size is H / 2×W / 2×C; Among them, H represents the height, W represents the width, and C represents the number of channels.

6. The method for detecting bird nests in contact network based on deformable sampling according to claim 1, characterized in that: Step S24 also includes: Using the feature similarity loss function, the first classification feature F cls1 and the second classification feature F cls2 equal.

7. The method for detecting bird nests in a contact network based on deformable sampling according to claim 1, characterized in that: In steps S2 and S3, the loss function includes: The total loss function of feature similarity is: ； The feature similarity loss function is: ； The total classification loss function is: ； The classification loss function is: ； The classification prediction results are: ； The coordinate regression loss function is: ； in, represents the total loss function; Represents the weight of feature similarity loss; represents the feature similarity loss function; represents the coordinate regression loss function; Minimize arrive Mean square error; , and Represents categorical features; represents the total loss function; Represents the weight of classification loss; represents the classification loss function; Represents the classification prediction result; Represents the true category label of the image; represents the classification detection head; and Represents the center point of the predicted box and the center point of the real box; represents the Euclidean distance; Represents the diagonal length of the smallest rectangle that contains the predicted box and the true box; Represents the intersection-over-union ratio of the real box and the predicted box, , ; Represents the prediction box; represents the real frame; represents the regression detection head; Represents regression features.

8. The method for detecting bird nests in contact network based on deformable sampling as claimed in claim 1, characterized in that: The S4 step includes: Using the main network feature extractor in the trained bird nest detection network model 、 Classification detection head and regression detection head Identify bird nests in the contact network bird nests to be identified image.

Images