Unmanned aerial vehicle inspection defect image recognition system and method

Abstract

The invention discloses an unmanned aerial vehicle inspection defect image recognition system and method, and relates to the technical field of image recognition, and the system comprises an unmanned aerial vehicle platform, an image collection module, an image processing module, a defect detection and recognition module, and a report generation module. The unmanned aerial vehicle platform comprises an unmanned aerial vehicle body, a high-definition camera, an infrared thermal imager, a cloud processing center, a GPS positioning system and a data transmission module. According to the method, the image collected by the unmanned aerial vehicle is subjected to preprocessing such as denoising, then feature extraction is performed, the influence of factors such as environment and weather on the shot image is effectively eliminated, the defect type of the inspection target is accurately identified through the defect identification model, the identification result is more accurate, the influence on defect identification judgment of the inspection target is avoided, and the inspection efficiency is improved. The efficient, accurate and real-time inspection and defect identification of the target area are realized, the model is optimized and updated regularly, and the accuracy and efficiency of identification can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a system and method for unmanned aerial vehicle inspection defect image recognition. Background Art

[0002] With the development of society and the rise of new energy vehicles, people's demand for electric energy has increased year by year. Therefore, it is particularly important to ensure the safety and smooth operation of the power transmission network. Nowadays, with the maturity of ultra-high voltage technology, the density of high-voltage transmission networks continues to increase, resulting in heavy power inspection tasks. With the gradual maturity of drone technology, replacing manual inspections with drone inspections can not only overcome the difficulties of geographical factors, but also ensure the safety and smooth operation of the power transmission network. At the same time, it greatly improves the inspection efficiency. Therefore, a drone inspection defect image recognition system and method has been created.

[0003] The existing technology has the following problems:

[0004] The traditional processing and recognition method for inspection images obtained by inspection drones uses pure convolutional neural networks to recognize and process images. The degree of intelligence is not high, and the adaptability of convolutional neural networks to perceiving multi-scale objects is limited. When drones acquire images, they are affected by many factors, resulting in the inability to accurately identify the inspection targets in the acquired images, which will affect the defect recognition and judgment of the inspection targets. Summary of the invention

[0005] The present invention provides a drone inspection defect image recognition system and method to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In the first aspect, the present invention provides a drone inspection defect image recognition system, which includes: a drone platform, an image acquisition module, an image processing module, a defect detection and recognition module and a report generation module. The drone platform includes a drone body, a high-definition camera, an infrared thermal imager, a cloud processing center, a GPS positioning system, and a data transmission module.

[0008] Preferably, the image processing module is used to preprocess the collected data, including denoising, enhancement, image registration and fusion, and use image processing methods of edge detection and texture analysis to extract key feature information in the image, including edges, textures and colors, and use deep learning algorithms to train and learn images.

[0009] Preferably, the defect detection and identification module is used to input the processed image to be identified into a defect recognition model for defect detection and identification, wherein the defect recognition model includes an offline defect recognition model composed of a small neural network and a typical offline defect database, and an online defect recognition model composed of a large neural network and a real-time updated online defect database.

[0010] Preferably, the construction of the defect recognition model includes the following steps:

[0011] S1: Obtain an annotated image set: Obtain a sample image set of drone inspection, annotate defects on each sample image in the sample image set, and form an annotated image set. The defect annotation includes annotating defect types and defect bounding boxes.

[0012] S3: Image set processing: preprocess each annotated image in the annotated image set, and divide the preprocessed annotated images into a training set and a test set.

[0013] S3: Model construction: Construct a defect recognition model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the skeleton;

[0014] S4: Model training: Use the training set to train the constructed neural network model, and use the test set to test the defect recognition model to obtain a trained defect recognition model.

[0015] Preferably, the report generation module is used to automatically generate an inspection report and a defect analysis report based on the results of defect identification, wherein the defect analysis report includes but is not limited to the category, location, quantity and severity of the defects, and the inspection report includes an inspection summary, basic information of the target area and an image overview.

[0016] In a second aspect, the present invention further provides a method for identifying defect images during drone inspection, the method comprising the following steps:

[0017] Step 1: UAV inspection and image acquisition: Plan the flight route, altitude and shooting strategy of the drone according to the characteristics and needs of the inspection area. Use a high-definition camera and infrared thermal imager to inspect and shoot along the planned route to obtain visible light images and infrared thermal imaging images.

[0018] Step 2: Image processing and transmission: Preprocess the collected images, including denoising, enhancement, image registration and fusion, and use edge detection and texture analysis image processing methods to extract key feature information in the image. The processed image data is transmitted to the cloud processing center in real time through wireless communication technology;

[0019] Step 3: Defect recognition and analysis: Automatically identify and classify defects in the processed image through the trained defect recognition model, and locate the defects based on the recognition results;

[0020] Step 4: Result feedback and decision support: The recognition results are fed back to the inspectors in real time through wireless communication technology. The inspectors locate the defect location based on the feedback results and take corresponding treatment measures. Inspection reports and defect analysis reports are generated according to needs, and decision support is provided to the inspectors based on historical data and current recognition results.

[0021] Step 5: Model optimization and update: Based on new image data and recognition results, the defect recognition model is optimized and updated regularly, and the online defect database is updated in real time.

[0022] Preferably, the step three comprises:

[0023] S31, preliminary recognition: performing preliminary recognition on the received image data through a first defect recognition model to obtain a preliminary recognition result;

[0024] S32, secondary recognition: when the preliminary recognition result is unrecognizable or defective, the preliminary recognition result and the image data are sent to the cloud master station through wireless communication, and the cloud master station performs secondary recognition on the image data using a second defect recognition model;

[0025] S33. Defect classification and location: Based on the identification results, the defects are classified, counted, stored and located.

[0026] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0027] 1. The present invention provides a drone inspection defect image recognition system and method. By performing pre-processing such as denoising on the images collected by the drone inspection and then performing feature extraction, the influence of factors such as environmental weather on the captured images can be effectively eliminated. The defect type of the inspection target can be accurately identified through a defect recognition model. The recognition result is more accurate, avoiding the influence on the defect recognition and judgment of the inspection target, and realizing efficient, accurate, and real-time inspection and defect recognition of the target area. The model can be optimized and updated regularly to improve the accuracy and efficiency of recognition, and the online defect database can be updated in real time to ensure that the defect samples in the database are consistent with the actual situation.

[0028] 2. The present invention provides a drone inspection defect image recognition system and method, which transmits image data and processing results in real time through wireless communication technology, making it convenient for operation and maintenance personnel to promptly understand the status of the target area and take measures to deal with the problem. The inspection report and defect analysis report are automatically generated through the report generation module, which can provide decision support for operation and maintenance personnel, facilitate operation and maintenance personnel to quickly locate problems and handle them, and has a high degree of intelligence.

[0029] 3. The present invention provides a drone inspection defect image recognition system and method. By constructing Swin-Transformer as the backbone network and combining it with a defect recognition model with a SA-FPN structure, it can extract and fuse multi-scale features and has strong adaptability. Compared with the traditional pure convolutional neural network structure, its performance is better.

[0030] 4. The present invention provides a drone inspection defect image recognition system and method, which adopts drone inspection to replace manual inspection, improves the intelligence level of inspection operations, significantly improves inspection efficiency, and drone inspection has fast speed and wide range. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a framework diagram of the drone inspection defect image recognition system of the present invention;

[0032] Figure 2 A schematic diagram of the construction process of the defect recognition model of the present invention;

[0033] Figure 3 It is a schematic diagram of the process of the defect image recognition method for UAV inspection of the present invention;

[0034] Figure 4 It is a schematic diagram of the defect identification and analysis process of the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0036] First, as Figure 1 As shown, the present invention provides a drone inspection defect image recognition system, which includes: a drone platform, an image acquisition module, an image processing module, a defect detection and recognition module and a report generation module. The drone platform includes a drone body, a high-definition camera, an infrared thermal imager, a cloud processing center, a GPS positioning system, and a data transmission module.

[0037] The system, through the cooperation of multiple modules such as drone platform, image acquisition module, image processing module, defect detection and identification module, report generation module, etc., through autonomous flight, obstacle avoidance and other technologies, not only has high-definition image acquisition capabilities, but also can replace manual inspections, realize efficient and safe inspection operations, improve the level of intelligent inspection operations, and significantly improve inspection efficiency. UAV inspections are fast and have a large inspection range.

[0038] The image processing module is used to pre-process the collected data, including denoising, enhancement, image registration and fusion. It uses image processing methods such as edge detection and texture analysis to extract key feature information in the image, including edges, textures and colors, and uses deep learning algorithms to train and learn images.

[0039] By using the image processing module to perform pre-processing such as denoising and enhancement on the images collected by drone inspections, the image quality can be improved, the impact of factors such as environmental weather on the captured images can be effectively eliminated, and subsequent image analysis can be facilitated.

[0040] The defect detection and recognition module is used to input the processed image to be recognized into the defect recognition model for defect detection and recognition, wherein the defect recognition model includes an offline defect recognition model composed of a small neural network and a typical offline defect database, and an online defect recognition model composed of a large neural network and a real-time updated online defect database.

[0041] like Figure 2 As shown in the figure, the construction of the defect recognition model includes the following steps:

[0042] S1: Obtain an annotated image set: Obtain a sample image set of drone inspection, annotate defects on each sample image in the sample image set, and form an annotated image set. The defect annotation includes annotating defect types and defect bounding boxes.

[0043] S3: Image set processing: preprocess each annotated image in the annotated image set, and divide the preprocessed annotated images into a training set and a test set.

[0044] S3: Model construction: Construct a defect recognition model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the skeleton;

[0045] S4: Model training: Use the training set to train the constructed neural network model, and use the test set to test the defect recognition model to obtain a trained defect recognition model.

[0046] By constructing Swin-Transformer as the backbone network and combining it with the defect recognition model of the SA-FPN structure, it is possible to extract and fuse multi-scale features with strong adaptability. Compared with the traditional pure convolutional neural network structure, its performance is better.

[0047] The report generation module is used to automatically generate inspection reports and defect analysis reports based on the results of defect identification. The defect analysis report includes but is not limited to the category, location, quantity and severity of the defects. The inspection report includes an inspection summary, basic information of the target area and an image overview.

[0048] The inspection reports and defect analysis reports automatically generated by the report generation module can provide decision support for operation and maintenance personnel, facilitate them to quickly locate problems and handle them, and have a high degree of intelligence.

[0049] Second, as Figure 3 As shown, the present invention also provides a method for identifying defect images during UAV inspection, and the identification method comprises the following steps:

[0050] Step 1: UAV inspection and image acquisition: Plan the flight route, altitude and shooting strategy of the drone according to the characteristics and needs of the inspection area. Use a high-definition camera and infrared thermal imager to inspect and shoot along the planned route to obtain visible light images and infrared thermal imaging images.

[0051] Step 2: Image processing and transmission: Preprocess the collected images, including denoising, enhancement, image registration and fusion, and use edge detection and texture analysis image processing methods to extract key feature information in the image. The processed image data is transmitted to the cloud processing center in real time through wireless communication technology;

[0052] Step 3: Defect recognition and analysis: Automatically identify and classify defects in the processed image through the trained defect recognition model, and locate the defects based on the recognition results;

[0053] Step 4: Result feedback and decision support: The recognition results are fed back to the inspectors in real time through wireless communication technology. The inspectors locate the defect location based on the feedback results and take corresponding treatment measures. Inspection reports and defect analysis reports are generated according to needs, and decision support is provided to the inspectors based on historical data and current recognition results.

[0054] Step 5: Model optimization and update: According to new image data and recognition results, the defect recognition model is optimized and updated regularly to improve the accuracy and efficiency of system recognition, and the online defect database is updated in real time to ensure that the defect samples in the database are consistent with the actual situation.

[0055] like Figure 4As shown, step three includes:

[0056] S31, preliminary recognition: performing preliminary recognition on the received image data through a first defect recognition model to obtain a preliminary recognition result;

[0057] S32, secondary recognition: when the preliminary recognition result is unrecognizable or defective, the preliminary recognition result and the image data are sent to the cloud master station through wireless communication, and the cloud master station uses the second defect recognition model to perform secondary recognition on the image data to obtain a more accurate recognition result;

[0058] S33. Defect classification and positioning: Based on the recognition results, the defects are classified, counted, stored and positioned. The classification includes temperature defects, structural defects, and judgment of the safety distance of the structure. Positioning is to convert the position of the defect in the image into the position in the actual space.

[0059] In summary, according to the characteristics and needs of the inspection area, the flight route, altitude and shooting strategy of the drone are planned. The high-definition camera and infrared thermal imager are used to conduct inspection and shooting according to the planned route to obtain visible light images and infrared thermal imaging images. The high-definition camera can capture visible light images, and the infrared thermal imager can capture areas with abnormal temperatures, which is particularly important for identifying defects such as hot spot failures of photovoltaic components. The collected images are pre-processed by denoising and enhancement to improve the quality of the images. The influence of environmental weather and other factors on the captured images can be effectively eliminated to avoid affecting the defect recognition and judgment of the inspection targets, which is convenient for subsequent image analysis. The image processing methods of edge detection and texture analysis are used to extract key feature information in the image. The processed image data is transmitted to the cloud processing center in real time through wireless communication technology. The received image data is preliminarily recognized by the first defect recognition model to obtain a preliminary recognition result. When the preliminary recognition result is unrecognizable or defective, the preliminary recognition result and the image data are sent to the cloud master station through wireless communication. The cloud master station uses the second defect recognition model to perform secondary recognition on the image data to obtain a more accurate recognition result. As a result, the defects are classified, counted, stored and located. The classification includes temperature defects, structural defects, and judgment of the safety distance of structures. Positioning is to convert the position of the defect in the image into the position in the actual space. The recognition results are fed back to the inspectors in real time through wireless communication technology. The inspectors locate the defect position according to the feedback results and take corresponding treatment measures. Inspection reports and defect analysis reports are generated according to needs. The inspection report includes the inspection summary, basic information of the target area and image overview. The defect analysis report includes but is not limited to the category, location, quantity and severity of the defects. Decision support is provided to the inspectors based on historical data and current recognition results, such as predicting the potential risk areas of the inspection targets and providing reference for the formulation of inspection plans. According to new image data and recognition results, the defect recognition model is regularly optimized and updated to improve the accuracy and efficiency of system recognition, and the online defect database is updated in real time to ensure that the defect samples in the database are consistent with the actual situation. Through the above steps, efficient, accurate and real-time inspection and defect recognition of the target area can be achieved, providing strong technical support and decision-making basis for operation and maintenance personnel. It has a high degree of intelligence and is built by Swi n-Transformer is used as the backbone network, combined with the defect recognition model of SA-FPN structure, which can extract and fuse multi-scale features and has strong adaptability. Compared with the traditional pure convolutional neural network structure, it has better performance. The use of drone inspection instead of manual inspection has improved the intelligence level of inspection operations and significantly improved inspection efficiency. Drone inspection has fast speed and wide range.

[0060] The above generally describes the present invention in detail, but it is obvious to a person skilled in the art that some modifications or improvements can be made to the present invention. Therefore, modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A drone inspection defect image recognition system, characterized by: The system includes: a drone platform, an image acquisition module, an image processing module, a defect detection and identification module and a report generation module. The drone platform includes a drone body, a high-definition camera, an infrared thermal imager, a cloud processing center, a GPS positioning system, and a data transmission module.

2. The unmanned aerial vehicle inspection defect image recognition system according to claim 1, characterized in that: The image processing module is used to preprocess the collected data, including denoising, enhancement, image registration and fusion, and use image processing methods such as edge detection and texture analysis to extract key feature information in the image, including edge, texture and color, and use deep learning algorithms to train and learn the image.

3. The unmanned aerial vehicle inspection defect image recognition system according to claim 2, characterized in that: The defect detection and identification module is used to input the processed image to be identified into the defect recognition model for defect detection and identification, wherein the defect recognition model includes an offline defect recognition model composed of a small neural network and a typical offline defect database, and an online defect recognition model composed of a large neural network and a real-time updated online defect database.

4. The unmanned aerial vehicle inspection defect image recognition system according to claim 3 is characterized by: The construction of the defect recognition model includes the following steps: S1: Obtain an annotated image set: Obtain a sample image set of drone inspection, annotate defects on each sample image in the sample image set, and form an annotated image set. The defect annotation includes annotating defect types and defect bounding boxes. S3: Image set processing: preprocess each annotated image in the annotated image set, and divide the preprocessed annotated images into a training set and a test set. S3: Model construction: Construct a defect recognition model with a multi-scale attention mechanism network structure Swin-Transformer, a feature pyramid network structure SA-FPN, and a regional feature network structure RPN as the skeleton; S4: Model training: Use the training set to train the constructed neural network model, and use the test set to test the defect recognition model to obtain a trained defect recognition model.

5. The unmanned aerial vehicle inspection defect image recognition system according to claim 1, characterized in that: The report generation module is used to automatically generate an inspection report and a defect analysis report based on the results of defect identification. The defect analysis report includes but is not limited to the category, location, quantity and severity of the defects, and the inspection report includes an inspection summary, basic information of the target area and an image overview.

6. A method for identifying defect images during drone inspection, comprising: a method for identifying defect images during drone inspection according to any one of claims 1 to 5, characterized in that: The identification method comprises the following steps: Step 1: UAV inspection and image acquisition: Plan the flight route, altitude and shooting strategy of the drone according to the characteristics and needs of the inspection area. Use a high-definition camera and infrared thermal imager to inspect and shoot along the planned route to obtain visible light images and infrared thermal imaging images. Step 2: Image processing and transmission: Preprocess the collected images, including denoising, enhancement, image registration and fusion, and use edge detection and texture analysis image processing methods to extract key feature information in the image. The processed image data is transmitted to the cloud processing center in real time through wireless communication technology; Step 3: Defect recognition and analysis: Automatically identify and classify defects in the processed image through the trained defect recognition model, and locate the defects based on the recognition results; Step 4: Result feedback and decision support: The recognition results are fed back to the inspectors in real time through wireless communication technology. The inspectors locate the defect location based on the feedback results and take corresponding treatment measures. Inspection reports and defect analysis reports are generated according to needs, and decision support is provided to the inspectors based on historical data and current recognition results. Step 5: Model optimization and update: Based on new image data and recognition results, the defect recognition model is optimized and updated regularly, and the online defect database is updated in real time.

7. The method for identifying defect images during unmanned aerial vehicle inspection according to claim 6, characterized in that: The step three comprises: S31, preliminary recognition: performing preliminary recognition on the received image data through a first defect recognition model to obtain a preliminary recognition result; S32, secondary recognition: when the preliminary recognition result is unrecognizable or defective, the preliminary recognition result and the image data are sent to the cloud master station through wireless communication, and the cloud master station performs secondary recognition on the image data using a second defect recognition model; S33. Defect classification and location: Based on the identification results, the defects are classified, counted, stored and located.

Images