Power transmission line unmanned aerial vehicle inspection picture defect identification method and system

Abstract

The invention discloses a power transmission line unmanned aerial vehicle inspection picture defect identification method and system. The method comprises the following steps: carrying out picture resolution amplification processing on a power transmission line unmanned aerial vehicle inspection picture; inputting the processed picture into a large target component identification model, identifying large target component defects and large target components where small target component defects are located, obtaining confidence coefficients corresponding to the identified large target component defects, and retaining each identified large target component defect greater than a confidence coefficient threshold value; cutting an image of the large target part where the identified small target part defects are located, inputting the image into a small target part identification model, identifying the small target part defects, obtaining confidence coefficients corresponding to the small target part defects, and retaining each identified small target part defect greater than a confidence coefficient threshold value; and summarizing the recognized defects of the large target part and the recognized defects of the small target part, and outputting a defect recognition result. According to the method and the device, the problem of defect identification of images with high resolution and concentrated defect distribution can be solved.

Description

Technical Field

[0001] The present application relates to the application field of computer vision and deep learning in image processing, and specifically to a method and system for identifying defects in transmission line drone inspection images. Background Art

[0002] At present, when conducting power transmission line inspections, drones are often used to capture images, and then target detection technology is used to identify and process defects in the images. The currently commonly used target detection defect recognition methods mainly use various target detection networks for recognition. The image resolution is generally small, and a higher recognition rate can be obtained with a smaller input size. For ultra-high resolution images (such as medical images, aerial aerial photography, etc.), the slicing method is often used for defect recognition due to the high degree of defect dispersion. However, the above methods cannot meet the defect recognition requirements of scenes with large resolution and relatively concentrated defect distribution.

[0003] In summary, due to the input resolution limitation, the traditional target detection network has limited recognition ability for large target component defects, and it is difficult to accurately identify small target component defects, such as missing pins, missing nuts, etc. When processing high-resolution images with densely distributed targets, it faces the problems of low detection rate and high false detection rate. Therefore, it is very necessary to develop a defect recognition method that can be applied to high-resolution and densely distributed drone inspection images. Summary of the invention

[0004] In order to solve the problem of low defect recognition accuracy for transmission lines with large resolution and concentrated defect distribution, the present application provides a method and system for defect recognition in transmission line drone inspection images.

[0005] In a first aspect, the present application provides a method for identifying defects in transmission line drone inspection images, comprising: Enlarge the resolution of transmission line drone inspection images; The processed transmission line drone inspection images are input into the large target component recognition model to identify the defects of the large target components and the large target components where the defects of the small target components are located, and the confidence corresponding to the identified defects of the large target components is obtained; wherein, a confidence threshold is set for each large target component defect in the transmission line, and the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the confidence threshold is retained; the large target component recognition model uses a deep learning neural network model; the large target component refers to a component in the transmission line whose actual component size is greater than the preset component size; An image of a large target component where a defect of a small target component is identified is cut out from a transmission line drone inspection picture after self-resolution magnification processing, and the cut image is input into a small target component recognition model to recognize the defect of the small target component and obtain the confidence corresponding to the defect of the small target component; wherein a confidence threshold is set for each small target component defect in the transmission line, and the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, and each identified small target component defect greater than the confidence threshold is retained, and each retained small target component defect is mapped to the original transmission line drone inspection picture that has not been cut; the small target component recognition model uses a deep learning neural network model; the small target component refers to a component in the transmission line whose actual component size is not greater than the preset component size; The identified defects of large target components and small target components are summarized, and the defect recognition results of the transmission line drone inspection images are output.

[0006] By adopting the above scheme, the design uses the image with enlarged resolution to complete the defect recognition of large target components, and adopts a diversion and progressive method to pre-identify the large target components where several small target components have defects, and then perform defect recognition of small target components on the large target components where several small target components have defects, so as to achieve high-resolution defect recognition of large target components and perform defect recognition on small target components with dense targets respectively; based on setting a confidence threshold for each specific category, the target boxes with relatively low confidence are filtered out to reduce the false detection rate of defects in each category of target components and improve the recognition rate of various defects.

[0007] Preferably, the confidence corresponding to the large target component where the identified small target component defect is located and the confidence threshold preset for the large target component where the identified small target component defect is located are obtained, and the confidence corresponding to the large target component where the identified small target component defect is located is compared with the confidence threshold preset for the large target component where the identified small target component defect is located; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is less than the first preset confidence difference threshold, then a preset confidence threshold compensation value is added on the basis of the original confidence threshold set for each identified small target component defect; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is greater than the second preset confidence difference threshold, then the preset confidence threshold compensation value is reduced based on the original confidence threshold set for each identified small target component defect; the second preset confidence difference threshold is greater than the first preset confidence difference threshold.

[0008] By adopting the above scheme, considering that the low recognition confidence of the large target component where the defect of the small target component is located will affect the subsequent defect recognition of the small target component, the small target component recognition model can dynamically adjust its recognition threshold according to the confidence generated by the recognition process of the large target component recognition model; in the case of high confidence in the target recognition of the large component where the defect of the small target component is located, the threshold for the defect recognition of the small target component can be moderately reduced to increase sensitivity, otherwise the threshold can be increased to reduce false alarms.

[0009] Preferably, the deep learning networks selected for the large target component recognition model and the small target component recognition model are both convolutional neural networks, and joint training is performed between the large target component recognition model and the small target component recognition model; the joint training is designed with a joint loss function, which is constructed by weighting the loss function of the large target component model recognition and the loss function of the small target component model recognition; During the joint training process, the joint loss function is calculated in each training iteration, and the iterative training is stopped when the joint loss function reaches a preset joint loss function threshold and the loss function of each model reaches a preset loss function threshold of the corresponding model.

[0010] By adopting the above scheme and a joint training approach, not only the loss function of each model itself but also the collaborative loss function is considered to measure the performance of the two models in collaboratively identifying large and small components, thus achieving a more efficient and accurate transmission line component identification system.

[0011] Preferably, it also includes: Acquire environmental data synchronized with the input transmission line drone inspection images; The inputs of the convolutional neural networks selected for the large target component recognition model and the small target component recognition model both include environmental data synchronized with the input transmission line drone inspection images, and the feature weights and learning frequencies adjusted according to the environmental data in the large target component recognition model are shared with the small target component recognition model.

[0012] By adopting the above scheme, information exchange between large target component models and small target component models is realized, which helps to improve the performance of the recognition model, especially in the task of power transmission line image detection in complex scenes.

[0013] Preferably, NMS processing is performed after the identified large target component defects and small target component defects are summarized.

[0014] By adopting the above solution, when multiple overlapping target frames appear for a specific target component, target frames with relatively low confidence are filtered out according to the degree of overlap, so as to reduce the false detection rate of each category.

[0015] Preferably, in the case where multiple defect types are identified for the same target component, the component defect type with the highest priority is retained according to the priority order of the component defect types set by the user.

[0016] By adopting the above solution, filtering is performed according to the defect priority, and only defects with high priority are output to further reduce the false detection rate.

[0017] Preferably, the deep learning networks selected for the large target component recognition model and the small target component recognition model need to perform balancing processing on their respective training data before training; the balancing processing includes: For the part of the training data in which the number of defects of a type of target components is greater than the first preset number of defects of the target components, reduction processing is performed; for the part of the training data in which the number of defects of a type of target components is less than the second preset number of defects of the target components, incremental processing is performed; for the part of the training data in which the number of target component categories is less than the preset number of target component types, incremental processing is performed.

[0018] By adopting the above scheme, in order to avoid the low recognition rate of some components and some component defects due to the lack of some component categories and some component defect categories in the training data, the recognition rate of each component and component defect category is improved.

[0019] In a second aspect, the present application provides a transmission line drone inspection image defect recognition system, comprising: The input image resolution enlargement processing module is used to enlarge the image resolution of the transmission line UAV inspection image; The input image large component defect recognition module is used to input the processed transmission line drone inspection image into the large target component recognition model, identify the large target component defects and the large target components where the small target component defects are located, and obtain the confidence corresponding to the identified large target component defects; wherein, a confidence threshold is set for each large target component defect in the transmission line, and the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the confidence threshold is retained; the large target component recognition model uses a deep learning neural network model; the large target component refers to a component in the transmission line whose actual component size is greater than the preset component size; The input image small component defect recognition module is used to cut out the image of the large target component where the identified small target component defect is located from the transmission line drone inspection image after self-resolution amplification processing, input the cut image into the small target component recognition model, identify the small target component defect, and obtain the confidence corresponding to the small target component defect; wherein, a confidence threshold is set for each small target component defect in the transmission line, the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, and each identified small target component defect greater than the confidence threshold is retained, and each retained small target component defect is mapped to the original uncut transmission line drone inspection image; the small target component recognition model uses a deep learning neural network model; the small target component refers to a component in the transmission line whose actual component size is not greater than the preset component size; The input image defect summary output module is used to summarize the identified large target component defects and small target component defects, and output the defect recognition results of the transmission line drone inspection images.

[0020] By adopting the above solution, the problem of low defect recognition accuracy for transmission lines with large resolution and concentrated defect distribution is solved, and the accuracy of transmission line defect recognition is improved.

[0021] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0022] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0023] In summary, this application has the following beneficial effects: 1. For large target components in the image, the target detection algorithm model is used to directly identify them by enlarging the resolution; for small target components, the main components of the transmission tower are first detected, and then the target detection algorithm model is used to detect the small target components on the components based on the detected components. Through this shunting and progressive method, on the basis of making full use of video memory resources and fast recognition speed, it not only solves the recognition of large target components under large image resolution and improves the recognition rate of defects of large target components, but also can cut small target components with concentrated defects, and identify the defects of small target components separately, greatly improving the recognition rate of defects of small target components; 2. Set different confidence thresholds for each category of output defects and eliminate detection results below the confidence threshold; and dynamically adjust the confidence threshold of small target component category defects according to the confidence of large target component category defects, reduce the impact of large target component recognition results on small target component recognition results, and reduce the false detection rate of each specific category of defects.

[0024] 3. Jointly train the recognition model of large target parts and the recognition model of small target parts, and interact with features and model parameters to improve the overall recognition rate and model performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a method for identifying defects in transmission line UAV inspection images described in a specific embodiment; Figure 2 It is a schematic diagram of model training in a method for identifying defects in transmission line UAV inspection images described in a specific embodiment; Figure 3 It is a schematic diagram of actual defect identification using the method for identifying defects in transmission line UAV inspection images in a specific embodiment; Figure 4 It is a structural schematic diagram of a power transmission line UAV inspection image defect recognition system described in a specific embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, the embodiment of the present application discloses a method for identifying defects in transmission line UAV inspection images, and the specific method steps are as follows: S1. Input the transmission line drone inspection picture to be identified and enlarge the picture resolution.

[0028] Specifically, according to the preset image resolution value and the preset density value, the transmission line drone inspection image with a larger image resolution and a denser distribution of detection targets is selected as the transmission line drone inspection image to be identified, and the image resolution is enlarged. In this embodiment, the image resolution range is increased from 1280*480~1280*800 to 2100*1280~2100*1800.

[0029] S2. Input the processed transmission line UAV inspection images into the large target component recognition model to identify the large target component defects and the large target component where the small target component defects are located.

[0030] Specifically, the large target component refers to a component in a transmission line whose actual component size is larger than a preset component size, such as a tower, a conductor, an anti-vibration hammer, a suspension clamp, an insulator and other relatively large components.

[0031] The large target component recognition model uses a deep learning neural network model, such as a CNN convolutional neural network, recorded as the first CNN convolutional neural network. The input of the model is a transmission line drone inspection picture with a relatively large resolution (such as a resolution range of 2100*1280~2100*1800), and the output is the defect type of the large target component in the transmission line drone inspection picture (such as foreign matter in the tower, rust on the tower body, foreign matter in the conductor, deformation of the anti-vibration hammer, displacement of the suspension clamp, cracking of the insulator, loose strands of the conductor, broken strands of the conductor, etc.) and the coordinate position, the confidence corresponding to the defect of the large target component, the large target component where the defect of the small target component is located (such as suspension clamps, insulators, etc.) and the confidence corresponding to the large target component where the defect of the small target component is located; wherein, according to the common small target component defects in the transmission line and the hierarchical information of each component, the common defective small target component corresponds to the large target component of the hierarchy, that is, the large target component where the small target defect is located.

[0032] like Figure 2 As shown, the training of the large target component recognition model includes: obtaining a number of historical transmission line drone inspection pictures of the large target component labeled with the name of the large target component, the defect category and coordinates of the large target component, and the location of the small target component defects as training data; after preprocessing the training data such as data cleaning, data balancing, image resolution enlargement, and data normalization, the preprocessed training data is input into the constructed large target component recognition model for training, a loss function is designed, and finally the large target component recognition model is obtained through training; wherein, the data balancing processing can be: reducing the amount of data with a large number of defects and relatively similar features; for categories with a small amount of data, data enhancement is performed by translation, horizontal flipping, scaling, cropping, adjusting contrast, adjusting brightness, etc., to increase the number of minority categories.

[0033] In this embodiment, data balancing processing includes: performing reduction processing on part of the training data in which the number of defects of a type of large target components in the training data is greater than the number of defects of a first preset target component; performing increment processing on part of the training data in which the number of defects of a type of large target components in the training data is less than the number of defects of a second preset target component; and performing increment processing on part of the training data in which the number of large target component categories in the training data is less than the number of preset large target component types.

[0034] The transmission line drone inspection images after image resolution amplification are subjected to data cleaning and data normalization, and then input into the large target component recognition model to identify the defect type of the large target component and the confidence level corresponding to the identified large target component defect, the large target component where the small target component defect is located, and the confidence level corresponding to the large target component where the identified small target component defect is located.

[0035] S3. Screening the identified large target component defects according to a preset confidence threshold for each large target component defect.

[0036] Specifically, a confidence threshold is set for each large target component defect in the transmission line, the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the corresponding confidence threshold is retained.

[0037] S4. Cut out the image of the large target component where the identified small target component defect is located from the transmission line drone inspection image after self-resolution enlargement processing, input the cut image into the small target component recognition model, identify the small target component defect and the confidence level corresponding to each small target component defect.

[0038] Specifically, the small target component refers to a component in the power transmission line whose actual component size is not larger than the preset component size, such as nuts, pins, washers and other larger-sized components.

[0039] The small target component recognition model also uses a deep learning neural network model, such as a CNN convolutional neural network, denoted as the second CNN convolutional neural network. The input of the model is the image of the large target component where the small target component defect is located, and the output is the defect type of the small target component in the transmission line drone inspection image (such as missing pins, missing nuts, pins not installed in place, nuts not installed properly, missing gaskets, etc.) and the coordinate position, and the confidence level corresponding to each small target component defect.

[0040] like Figure 2 As shown, the training of the small target component recognition model includes: obtaining images of each large target component with small target defects cut out from the historical transmission line drone inspection pictures after data cleaning, labeling the images of each large target component with the name of the small target component, the category of the small target component defect and the coordinates, and using the labeled images of each large target component as training data; after preprocessing the training data such as data cleaning, data balancing, image resolution enlargement, and data normalization, the preprocessed training data is input into the constructed small target component recognition model for training, a loss function is designed, and finally the small target component recognition model is obtained through training.

[0041] In this embodiment, data balancing processing includes: performing reduction processing on part of the training data in which the number of defects of a type of small target components is greater than the number of defects of a first preset target component; performing increment processing on part of the training data in which the number of defects of a type of large and small standard components is less than the number of defects of a second preset target component; and performing increment processing on part of the training data in which the number of small target component categories is less than the number of preset small target component types.

[0042] The image of the large target component where the identified small target component defect is located is cut out from the transmission line drone inspection image after self-resolution enlargement processing, and the cut image is input into the small target component recognition model to identify the small target component defects and the confidence level corresponding to each small target component defect; among them, image cutting uses edge-based segmentation technology, which can detect edges on the image and then segment it according to the edges.

[0043] S5. According to the preset confidence threshold of each small target component defect, the identified small target component defects are screened, and the screened small target component defects are mapped to the original uncut transmission line drone inspection image.

[0044] Specifically, a confidence threshold is set for each small target component defect in the transmission line, the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, each identified small target component defect greater than the corresponding confidence threshold is retained, and each retained small target component defect is mapped to the original uncut transmission line drone inspection image, that is, the type label and coordinates of each retained small target component defect are mapped to the original uncut transmission line drone inspection image.

[0045] S6. Summarize the identified defects of large target components and small target components, and output the defect recognition results of the transmission line UAV inspection pictures.

[0046] Specifically, the identified defects of large target components and small target components are summarized in terms of defect categories and coordinates, and the defect recognition results of the transmission line drone inspection images are output.

[0047] By adopting the scheme described in the above embodiment, the image is input and scaled to a size of 2100*1800, and sent to the large target component recognition model to obtain the large target component defects and the large target components where the small target component defects are located, specifically the three large target component defects of the tower foreign matter, tower body corrosion, and suspension wire clamp offset, and the two large target components of the suspension wire clamp and tension wire clamp. For the three large target component defects of the tower foreign matter, tower body corrosion, and suspension wire clamp offset, the confidence levels are 0.5, 0.7, and 0.3, respectively, and the corresponding configured confidence thresholds are 0.4, 0.5, and 0.6, respectively. Obviously, if the confidence levels of the tower foreign matter and tower body corrosion are higher than their corresponding confidence thresholds, the recognition results are retained; while the confidence level of the suspension wire clamp offset is lower than the corresponding confidence threshold, the recognition results are not retained. There are small parts such as nuts, pins and washers on the suspension clamps and tension clamps. The area where the suspension clamps and tension clamps are located is cut out from the original image and sent to the small target component recognition model for recognition. The recognition result shows that there is only one missing pin in the suspension clamp, and there is no defect in the tension clamp. The confidence of the missing pin on the suspension clamp is 0.9, and the corresponding threshold is 0.5. The confidence of the missing pin is higher than the corresponding confidence threshold. The coordinates and labels of the missing pin are mapped to the input image. This image has only a single defect, and finally outputs three types of defects: foreign matter in the tower, rust on the tower body, and missing pins.

[0048] Compared with conventional target detection methods, by adopting the method described in the above embodiment, the defect detection rate of large target parts is increased from 60%~70% to 80%~90%, and the false detection rate is reduced from 3~5 to less than 1; the defect detection rate of small target parts is increased from about 30% to 85%~95%, and the false detection rate is reduced from 15~25 to between 1~3.

[0049] In a specific embodiment, considering that multiple overlapping target frames may appear in the recognition process of some specific target components, target frames with relatively low confidence are filtered out according to the overlap of the target frames to reduce the false detection rate of each category. The method further includes: After the identified large target component defects and small target component defects are summarized, NMS processing is performed to eliminate the identified target component defects with low confidence values ​​whose overlap of the overlapping target frames is greater than a preset overlap threshold.

[0050] By adopting the scheme described in the above embodiment, the image is input and scaled to a size of 2100*1800, and sent to the large target component recognition model to obtain the large target component defects and the large target components where the small target component defects are located, specifically the three large target component defects of the tower foreign body, tower body corrosion, and suspension wire clamp offset, and the two large target components of the suspension wire clamp and tension wire clamp. For the three large target component defects of the tower foreign body, tower body corrosion, and suspension wire clamp offset, the tower foreign body identifies two frames, and the tower body corrosion and suspension wire clamp offset only identify one frame, with confidence levels of 0.9, 0.5, 0.51, and 0.7, respectively, and the corresponding configured confidence thresholds of 0.4, 0.5, and 0.6, respectively. The confidence levels of all identified defects are higher than their corresponding confidence thresholds, and all identification results are retained. There are small parts such as nuts, pins and washers on the suspension clamp and the tension clamp. The area where the suspension clamp and the tension clamp are located is cut out from the original image and sent to the small target component recognition model for recognition. The recognition results show that the suspension clamp has a pin missing defect and a pin installation defect, and the tension clamp has no defects. The confidence of the missing pin on the suspension clamp is 0.1, and the confidence of the pin is not standardized is 0.8. The corresponding confidence thresholds are 0.5 and 0.45 respectively. The confidence of the missing pin is lower than the corresponding confidence threshold, and the confidence of the pin is higher than the corresponding confidence threshold. Only the coordinates and labels of the pin are mapped to the input image. All the recognition results of this image are summarized. After NMS processing, it is found that the overlap of the two foreign objects on the tower is 0.55, which is higher than the set threshold of 0.5. The foreign object frame of the tower with a confidence of 0.5 is deleted, and finally the four defects of foreign objects on the tower, tower corrosion, suspension clamp offset and pin irregularity are output.

[0051] A specific embodiment, based on the above-mentioned embodiment of NMS processing, takes into account the situation that multiple defect types are identified for the same target component. In order to further reduce the false detection rate and retain the more important defect type identification results, the priority setting is used to output the defect types with high priority order, such as Figure 3 As shown, the method also includes: In the case where multiple defect types are identified for the same target component (large target component or small target component), the component defect type with the highest priority is retained according to the component defect type priority order set by the user to ensure that only one defect type is identified for the same target component.

[0052] By adopting the scheme described in the above embodiment, the image is input and scaled to a size of 2100*1800, and sent to the large target component recognition model to obtain the large target component defects and the large target components where the small target component defects are located, specifically the three large target component defects of the tower foreign body, tower body corrosion, and suspension wire clamp offset, and the two large target components of the suspension wire clamp and tension wire clamp. For the three large target defects of the tower foreign body, tower body corrosion, and suspension wire clamp offset, the tower foreign body identifies two frames, and the tower body corrosion and suspension wire clamp offset only identify one frame, with confidence levels of 0.9, 0.51, 0.51, and 0.7, respectively, and the corresponding configured confidence thresholds are 0.4, 0.5, and 0.6, respectively. The confidence levels of all identified defects are higher than their corresponding set values, and all identification results are retained. There are small parts such as nuts, pins and washers on the suspension clamp and the tension clamp. The area where the suspension clamp and the tension clamp are located is cut out from the original image and sent to the small target component recognition model for recognition. The recognition results show that the suspension clamp has a missing nut defect and a missing washer defect, and the tension clamp has no defects. The confidence of the missing nut on the suspension clamp is 0.7, and the confidence of the missing washer is 0.8. The corresponding confidence thresholds are 0.5 and 0.55 respectively. The confidence of the two defects is higher than the set confidence threshold. The coordinates and labels of the two defects are mapped to the input image. All the recognition results of this image are summarized. After NMS processing, it is found that the overlap of the two tower foreign objects is 0.55, which is higher than the set threshold of 0.5. The tower foreign object frame with a confidence of 0.51 is deleted. After priority filtering, it is found that the overlap of the missing nut and the missing washer is higher than the set threshold of 0.5. The priority of the missing nut is higher, so only the defect of the missing nut is retained. The identification results of other defects remain unchanged, and the final output is four defects: foreign matter in the tower, rust on the tower body, offset of the suspension clamp, and missing nuts.

[0053] In a specific embodiment, considering that the recognition accuracy of the large target component will affect the accuracy of the subsequent defect recognition of the corresponding small target component, in order to reduce this influence and improve the accuracy of the defect recognition of the small target component, the method further includes: The confidence threshold set corresponding to each identified small target component defect is the dynamic confidence threshold. The specific dynamic confidence threshold is determined according to the confidence corresponding to the large target component where each identified small target component defect is located. The determination process includes: Obtaining the confidence corresponding to the large target component where the identified small target component defect is located, and a preset confidence threshold for the large target component where the identified small target component defect is located, and comparing the confidence corresponding to the large target component where the identified small target component defect is located with the preset confidence threshold for the large target component where the identified small target component defect is located; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is less than the first preset confidence difference threshold, it indicates that the accuracy of the large target component where the small target component defect is currently identified is average, and a preset confidence threshold compensation value is added to the original confidence threshold set for each identified small target component defect, thereby increasing the threshold to reduce false positives in the identification of small target component defects; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is greater than the second preset confidence difference threshold, it indicates that the accuracy of the large target component where the small target component defect is currently identified is relatively high. On the basis of the original confidence threshold set for each identified small target component defect, the preset confidence threshold compensation value is reduced, and the threshold is appropriately lowered to increase the sensitivity of small target component defect identification; the second preset confidence difference threshold is greater than the first preset confidence difference threshold.

[0054] For example, the confidence levels of the two large target components, the suspension wire clamp and the tension wire clamp, are 0.7 and 0.5 respectively, and the corresponding preset confidence thresholds for the two large target components are 0.88 and 0.5. Then, the recognition results of the two large target components, the suspension wire clamp and the tension wire clamp, are relatively accurate, and the corresponding confidence difference for the suspension wire clamp is 0.18, which is greater than the second preset confidence difference threshold of 0.1. The preset confidence threshold compensation value is added to the original confidence threshold of 0.55 set for the defect recognition of the suspension wire clamp, and the adjusted value is 0.6; the corresponding confidence difference for the tension wire clamp is 0, which is less than the first preset confidence difference threshold of 0.05. The preset confidence threshold compensation value is reduced from the original confidence threshold of 0.55 set for the defect recognition of the tension wire clamp, and the adjusted value is 0.5.

[0055] In a specific embodiment, in order to improve the overall performance of the large target component recognition model and the small target component recognition model and obtain a more accurate defect recognition result, the method further includes: The large target component recognition model and the small target component recognition model are jointly trained in the model process; wherein, the joint training is designed with a joint loss function, and the joint loss function is constructed by weighting the loss function of the large target component model recognition and the loss function of the small target component model recognition; in the joint training process, in each training iteration, in addition to calculating the loss function of each model, when the loss function threshold preset for the corresponding model is reached, it is also necessary to calculate the joint loss function, and when the joint loss function reaches the preset joint loss function threshold, the iterative training is stopped.

[0056] In addition to joint training to improve overall performance, the feature weights and learning frequencies of each model are further interacted to adapt to complex environmental scenarios. The specific steps include: Acquire environmental data synchronized with the input transmission line drone inspection pictures; the environmental data includes: light intensity data and weather data; The inputs of the convolutional neural networks selected for the large target component recognition model and the small target component recognition model both include environmental data synchronized with the input transmission line drone inspection images, and the feature weights and learning frequencies adjusted according to the environmental data in the large target component recognition model are shared with the small target component recognition model.

[0057] Specifically, when the light intensity reaches a first preset light intensity and the rainfall data is less than the first preset rainfall data, that is, corresponding to strong light and clear environmental conditions, the color and texture feature weights in the transmission line UAV inspection picture in the large target component recognition model are increased according to the preset weight adjustment value; when the light intensity is less than a second preset light intensity and the rainfall data is greater than the second preset rainfall data, that is, corresponding to low light and bad weather environmental conditions, the shape and structure feature weights in the transmission line UAV inspection picture in the target component recognition model are increased according to the preset weight adjustment value; wherein the first preset light intensity is greater than the second preset light intensity, and the first preset rainfall data is less than the second preset rainfall data; the feature weights adjusted according to the environmental data in the large target component recognition model are shared with the small target component recognition model.

[0058] Similarly, when the light intensity reaches the first preset light intensity and the rainfall data is less than the first preset rainfall amount, that is, corresponding to strong light exposure and clear environmental conditions, the learning frequency of the large target component recognition model is increased according to the preset learning frequency adjustment value; when the light intensity is less than the second preset light intensity and the rainfall data is greater than the second preset rainfall amount, the learning frequency of the large target component recognition model is reduced according to the preset learning frequency adjustment value; the learning frequency adjusted according to the environmental data in the large target component recognition model is shared with the small target component recognition model.

[0059] like Figure 4 As shown, the embodiment of the present application discloses that the present application provides a transmission line drone inspection image defect recognition system, comprising: The input image resolution enlargement processing module 101 is used to enlarge the image resolution of the transmission line UAV inspection image; The input image large component defect recognition module 102 is used to input the processed transmission line drone inspection image into the large target component recognition model, identify the large target component defects and the large target components where the small target component defects are located, and obtain the confidence corresponding to the identified large target component defects; wherein, a confidence threshold is set for each large target component defect in the transmission line, and the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the confidence threshold is retained; the large target component recognition model uses a deep learning neural network model; the large target component refers to a component in the transmission line whose actual component size is greater than the preset component size; The input image small component defect recognition module 103 is used to cut out the image of the large target component where the identified small target component defect is located from the transmission line drone inspection image after self-resolution amplification processing, input the cut image into the small target component recognition model, identify the small target component defect, and obtain the confidence corresponding to the small target component defect; wherein, a confidence threshold is set for each small target component defect in the transmission line, the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, each identified small target component defect greater than the confidence threshold is retained, and each retained small target component defect is mapped to the original uncut transmission line drone inspection image; the small target component recognition model uses a deep learning neural network model; the small target component refers to a component in the transmission line whose actual component size is not greater than the preset component size; The input image defect summary output module 104 is used to summarize the identified large target component defects and small target component defects, and output the defect recognition results of the transmission line drone inspection image.

[0060] The input image defect summary output module 104 in the system is also used to summarize the identified large target component defects and small target component defects and then perform NMS processing; in the case where multiple defect types are identified for the same target component, the component defect type with the highest priority is retained according to the component defect type priority order set by the user.

[0061] The system further comprises: The component defect recognition model association module 105 is used to obtain environmental data synchronized with the captured input transmission line drone inspection pictures; the inputs of the convolutional neural networks selected for the large target component recognition model and the small target component recognition model both include environmental data synchronized with the captured input transmission line drone inspection pictures, and the feature weights and learning frequencies adjusted according to the environmental data in the large target component recognition model are shared with the small target component recognition model.

[0062] The embodiment of the present application also discloses a computer-readable storage medium.

[0063] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed such as the above-mentioned transmission line drone inspection image defect identification method. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0064] The embodiment of the present application also discloses a computer device.

[0065] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned method for identifying defects in transmission line drone inspection images.

[0066] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A method for identifying defects in transmission line UAV inspection images, characterized in that: include: Enlarge the resolution of transmission line drone inspection images; The processed transmission line drone inspection images are input into the large target component recognition model to identify the defects of the large target components and the large target components where the defects of the small target components are located, and the confidence corresponding to the identified defects of the large target components is obtained; wherein, a confidence threshold is set for each large target component defect in the transmission line, and the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the confidence threshold is retained; the large target component recognition model uses a deep learning neural network model; the large target component refers to a component in the transmission line whose actual component size is greater than the preset component size; An image of a large target component where a defect of a small target component is identified is cut out from a transmission line drone inspection picture after self-resolution magnification processing, and the cut image is input into a small target component recognition model to recognize the defect of the small target component and obtain the confidence corresponding to the defect of the small target component; wherein a confidence threshold is set for each small target component defect in the transmission line, and the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, and each identified small target component defect greater than the confidence threshold is retained, and each retained small target component defect is mapped to the original transmission line drone inspection picture that has not been cut; the small target component recognition model uses a deep learning neural network model; the small target component refers to a component in the transmission line whose actual component size is not greater than the preset component size; The identified defects of large target components and small target components are summarized, and the defect recognition results of the transmission line drone inspection images are output.

2. The method for identifying defects in transmission line UAV inspection images according to claim 1 is characterized in that: The confidence threshold set corresponding to each identified small target component defect is the dynamic confidence threshold. The specific dynamic confidence threshold is determined according to the confidence corresponding to the large target component where each identified small target component defect is located. The determination process includes: Obtaining the confidence corresponding to the large target component where the identified small target component defect is located, and a preset confidence threshold for the large target component where the identified small target component defect is located, and comparing the confidence corresponding to the large target component where the identified small target component defect is located with the preset confidence threshold for the large target component where the identified small target component defect is located; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is less than the first preset confidence difference threshold, then a preset confidence threshold compensation value is added on the basis of the original confidence threshold set for each identified small target component defect; If the confidence corresponding to the large target component where the identified small target component defect is located is greater than the preset confidence threshold of the large target component where the identified small target component defect is located and the difference between the two is greater than the second preset confidence difference threshold, then the preset confidence threshold compensation value is reduced based on the original confidence threshold set for each identified small target component defect; the second preset confidence difference threshold is greater than the first preset confidence difference threshold.

3. The method for identifying defects in transmission line UAV inspection images according to claim 1 is characterized in that: The deep learning networks selected for the large target component recognition model and the small target component recognition model are both convolutional neural networks, and joint training is performed between the large target component recognition model and the small target component recognition model; the joint training is designed with a joint loss function, which is constructed by weighting the loss function of the large target component model recognition and the loss function of the small target component model recognition; During the joint training process, the joint loss function is calculated in each training iteration, and the iterative training is stopped when the joint loss function reaches a preset joint loss function threshold and the loss function of each model reaches a preset loss function threshold of the corresponding model.

4. The method for identifying defects in transmission line UAV inspection images according to claim 3 is characterized in that: Also includes: Acquire environmental data synchronized with the input transmission line drone inspection images; The inputs of the convolutional neural networks selected for the large target component recognition model and the small target component recognition model both include environmental data synchronized with the input transmission line drone inspection images, and the feature weights and learning frequencies adjusted according to the environmental data in the large target component recognition model are shared with the small target component recognition model.

5. The method for identifying defects in transmission line UAV inspection images according to claim 1 is characterized in that: The NMS processing is performed after the identified large target component defects and small target component defects are summarized.

6. The method for identifying defects in transmission line UAV inspection images according to claim 1 is characterized in that: In the case where multiple defect types are identified for the same target component, the component defect type with the highest priority is retained according to the priority order of the target component defect types set by the user.

7. The method for identifying defects in transmission line UAV inspection images according to claim 1 is characterized in that: The deep learning networks selected for the large target component recognition model and the small target component recognition model need to perform balancing processing on their respective training data before training; The balancing process includes: For the part of the training data in which the number of defects of a type of target components is greater than the first preset number of defects of the target components, reduction processing is performed; for the part of the training data in which the number of defects of a type of target components is less than the second preset number of defects of the target components, incremental processing is performed; for the part of the training data in which the number of target component categories is less than the preset number of target component types, incremental processing is performed.

8. A transmission line drone inspection image defect recognition system, characterized in that: include: The input image resolution enlargement processing module is used to enlarge the image resolution of the transmission line UAV inspection image; The input image large component defect recognition module is used to input the processed transmission line drone inspection image into the large target component recognition model, identify the large target component defects and the large target components where the small target component defects are located, and obtain the confidence corresponding to the identified large target component defects; wherein, a confidence threshold is set for each large target component defect in the transmission line, and the confidence corresponding to each identified large target component defect is compared with the corresponding confidence threshold, and each identified large target component defect greater than the confidence threshold is retained; the large target component recognition model uses a deep learning neural network model; the large target component refers to a component in the transmission line whose actual component size is greater than the preset component size; The input image small component defect recognition module is used to cut out the image of the large target component where the identified small target component defect is located from the transmission line drone inspection image after self-resolution magnification processing, input the cut image into the small target component recognition model, identify the small target component defect, and obtain the confidence corresponding to the small target component defect; wherein, a confidence threshold is set for each small target component defect in the transmission line, the confidence corresponding to each identified small target component defect is compared with the corresponding confidence threshold, and each identified small target component defect greater than the confidence threshold is retained, and each retained small target component defect is mapped to the original uncut transmission line drone inspection image; the small target component recognition model uses a deep learning neural network model; the small target component refers to a component in the transmission line whose actual component size is not greater than the preset component size; The input image defect summary output module is used to summarize the identified large target component defects and small target component defects, and output the defect recognition results of the transmission line drone inspection images.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images