Unmanned aerial vehicle electric power inspection method and device
By adjusting and compressing the image recognition model, the video frame images collected by the drone are deployed in front-end computing devices in real time, solving the problems of delay and cost of power inspection by the drone, and achieving efficient power inspection results.
Patent Information
- Application Number
- CN202411146504.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
The existing drone power inspection technology has problems such as delay and high requirements for picture clarity and angle, and has high requirements for the loading capacity and battery life of the drone, which is relatively expensive and does not apply to conventional non-conventional non-consistency. Human-machine.
By obtaining the images collected during power inspection, adjusting their light and darkness, training and compressing the image recognition model, deploying them in front-end computing devices to recognize video frame images in real time, improving the efficiency of power inspection.
It realizes a small model with high recognition accuracy, suitable for large light or large shadow images, obtains power inspection results in real time, and improves the efficiency of drone power inspection.
Smart Images

Figure CN119007046A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition, and specifically to a method, device, electronic equipment and computer storage medium for unmanned aerial vehicle power inspection. Background Art
[0002] With the rapid development of drone technology, drones have been widely used in various inspection scenarios. Among them, power inspections using drones can obtain real-time image data of power facilities, greatly improving the efficiency of power inspections.
[0003] Specifically, the technician controls the drone to fly to the target area through the drone flight control terminal, and aims the camera at the target object to shoot. The drone sends the captured pictures to the drone flight control terminal, which then sends the pictures to the server for image recognition and analysis. The staff can perform further processing based on the results of recognition and analysis. This method is affected by the transmission network, has a certain delay, and has high requirements for the clarity of the picture, the shooting angle, etc. If the picture cannot be recognized, it needs to be re-shot, which is time-consuming and laborious.
[0004] In order to improve the real-time performance of power inspection, it is also possible to use drones equipped with image recognition devices for power inspection. The images taken by the camera on the drone can be directly recognized and analyzed by the image recognition device, and then the recognition and analysis results are sent to the drone flight control end, which sends the recognition and analysis results to the service end for further processing by the staff. However, this method has high requirements on the drone's mounting capacity and battery life, and is costly, so it is not suitable for conventional drones.
[0005] Therefore, how to better improve the efficiency of UAV power inspection is an urgent problem that needs to be solved. Summary of the invention
[0006] The present application provides a UAV power inspection method to solve the problem of how to improve the efficiency of UAV power inspection. The present application also provides a UAV power inspection device, electronic equipment and computer storage medium.
[0007] The present application provides a method for electric power inspection using a drone, comprising: obtaining an image collected during an electric power inspection process as an original image, wherein a target scene in the original image includes a target recognition object; adjusting the brightness of the original image to obtain a processed image; obtaining an initial image recognition model, training the initial image recognition model based on the original image and the processed image, and compressing the trained image recognition model to obtain a target image recognition model for detecting the target recognition object; obtaining a current video frame image collected by the drone during the electric power inspection process in real time; and obtaining an electric power inspection result for the current video frame image based on the target image recognition model.
[0008] Optionally, obtaining the images collected during the power inspection process as original images includes: obtaining images collected during the power inspection process for the target identification object at different time periods and different angles as original images.
[0009] Optionally, the adjusting the brightness and darkness of the original image to obtain a processed image includes: converting the original image into a grayscale image and obtaining an image mean of the grayscale image; taking the original image whose image mean is within a preset range as the image to be adjusted, adjusting the brightness and darkness of the image to be adjusted, and obtaining a processed image.
[0010] Optionally, the original image whose image mean is within a preset range is used as the image to be adjusted, and the brightness and darkness of the image to be adjusted are adjusted to obtain a processed image, including: obtaining the maximum illumination distance and the division distance length in the image to be adjusted; obtaining an adjustment amount according to the maximum illumination distance and the division distance length; adjusting the pixels of the image to be adjusted according to the division distance length and the adjustment amount to obtain a processed image.
[0011] Optionally, obtaining the initial image recognition model includes: obtaining an original YOLOv5 network model; adding an SE attention mechanism to the original YOLOv5 network model, and replacing the feature pyramid FPN structure of the original YOLOv5 network model with a weighted bidirectional feature pyramid structure to obtain the initial image recognition model.
[0012] Optionally, the initial image recognition model is trained based on the original image and the processed image, including: taking the original image and the processed image as training samples, annotating the target recognition objects in the training samples, and obtaining annotation results; inputting the training samples into the initial image recognition model, and obtaining image recognition results output by the initial image recognition model; and adjusting the parameters of the initial image recognition model based on the annotation results and image recognition results of the training samples, and obtaining a trained image recognition model.
[0013] Optionally, adjusting the parameters of the initial image recognition model according to the labeling results and image recognition results of the training samples to obtain the trained image recognition model includes: obtaining the value of the image recognition loss function according to the labeling results and image recognition results of the training samples; adjusting the parameters of the initial image recognition model according to the value of the image recognition loss function; and stopping adjusting the parameters of the initial image recognition model until the value of the image recognition loss function reaches a preset condition to obtain the trained image recognition model.
[0014] Optionally, compressing the trained image recognition model to obtain a target image recognition model for detecting a target recognition object includes: compressing the trained image recognition model by weight quantization to obtain a target image recognition model for detecting a target recognition object.
[0015] Optionally, the method further includes: performing image enhancement processing on the current video frame image to obtain the processed current video frame image as the image to be recognized.
[0016] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a first type of target identification object, obtaining a first power inspection result for the current video frame image, the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target identification objects includes at least one of defective power facilities and foreign objects on the power facilities.
[0017] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting a plurality of consecutive images to be recognized within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if, among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be recognized includes a first type of target recognition object exceeds a first preset threshold, then obtaining a first power inspection result for the plurality of current video frame images, the first power inspection result being used to indicate that there are safety hazards in the target scene, and the first type of target recognition objects includes at least one of defective power facilities and foreign objects on the power facilities.
[0018] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a second type of target identification object, obtaining a second power inspection result for the current video frame image, the second power inspection result is used to indicate that the staff in the target scene are not wearing safety protection equipment, and the second type of target identification objects include staff who are located on the power facilities and are not wearing safety protection equipment.
[0019] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting a plurality of consecutive images to be identified within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if, among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be identified includes a second category of target recognition objects exceeds a second preset threshold, then obtaining a second power inspection result for the plurality of current video frame images, the second power inspection result being used to indicate that the staff in the target scene is not wearing safety protection equipment, and the second category of target recognition objects includes staff that are located on power facilities and are not wearing safety protection equipment.
[0020] Optionally, it also includes: sending the power inspection result to a service end.
[0021] The present application also provides a UAV power inspection device, including: an original image acquisition unit, used to obtain an image collected during the power inspection process as the original image, and the target scene in the original image includes a target recognition object; an original image adjustment unit, used to adjust the brightness of the original image to obtain a processed image; a model training unit, used to obtain an initial image recognition model, train the initial image recognition model according to the original image and the processed image, and compress the trained image recognition model to obtain a target image recognition model for detecting the target recognition object; a current image acquisition unit, used to obtain in real time a current video frame image collected by the UAV during the power inspection process; an inspection result acquisition unit, used to obtain a power inspection result for the current video frame image according to the target image recognition model.
[0022] The present application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and after the processor runs the computer program, it executes the above-mentioned UAV power inspection method.
[0023] The present application also provides a computer storage medium, which stores a computer program. After the computer program is run by a processor, the above-mentioned UAV power inspection method is executed.
[0024] Compared with the prior art, this application has the following advantages:
[0025] The present application provides a method for unmanned aerial vehicle power inspection, including: obtaining an image collected during the power inspection process as an original image, wherein the target scene in the original image includes a target recognition object; adjusting the brightness of the original image to obtain a processed image; obtaining an initial image recognition model, training the initial image recognition model according to the original image and the processed image, and compressing the trained image recognition model to obtain a target image recognition model for detecting the target recognition object; obtaining the current video frame image collected by the unmanned aerial vehicle during the power inspection process in real time; obtaining the power inspection result for the current video frame image according to the target image recognition model. The target image recognition model obtained in the above method has high recognition accuracy and small size, and is more applicable to images with large illumination or large shadows in power inspection; the model can be deployed in a front-end computing device connected to the flight control end of the unmanned aerial vehicle, and recognizes the video frame images taken by the unmanned aerial vehicle in real time, and obtains the power inspection result in time, thereby improving the efficiency of the unmanned aerial vehicle power inspection. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a schematic diagram of the scenario of power inspection by drone in this application.
[0027] Figure 2 It is an overall flow chart of the UAV power inspection method provided in this application.
[0028] Figure 3 This is a flow chart of the UAV power inspection method provided in the first embodiment of the present application.
[0029] Figure 4 It is a schematic diagram of a UAV power inspection device provided in the second embodiment of the present application.
[0030] Figure 5 It is a schematic diagram of an electronic device provided in the third embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to make the purpose, advantages and features of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotions without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.
[0032] It should be noted that, in the description of the present application, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance, and a specific order or sequence. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, the term "multiple" refers to two or more. The term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship. The terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0033] As we all know, if there are defects or external damage to the power facilities such as transmission lines, towers, and hardware of overhead transmission lines, they may cause failures and threaten the safety and stable operation of the power grid. In order to understand the operating status of the line and eliminate defects and hidden dangers in time, it is necessary to conduct power inspections regularly, and the development of drone technology provides effective technical support for the inspection of overhead transmission lines.
[0034] Unmanned aircraft is referred to as UAV, and the device that controls the flight of UAV is called UAV flight control terminal, referred to as UAV flight control terminal. Figure 1 As shown, usually, a technician 101 controls a drone 103 to fly to a target area through a drone flight control terminal 102 and directs a camera toward a target object 104 for shooting.
[0035] The drone sends the captured images to the drone flight control end, which then sends the images to the server end for image recognition and analysis to determine whether there are any safety hazards, etc. The staff can perform further processing based on the results of the recognition and analysis. This method is affected by the transmission network, has a certain delay, and has high requirements for the clarity of the image, the shooting angle, etc. If the image cannot be recognized, it needs to be re-photographed, which is time-consuming and laborious. In order to improve the real-time performance of power inspection, the drone can also be equipped with an image recognition device for power inspection. The images taken by the camera on the drone can be directly recognized and analyzed by the image recognition device, and the results of the recognition and analysis are sent to the drone flight control end, which sends the results of the recognition and analysis to the server end for further processing by the staff. However, this method has high requirements for the drone's mounting capacity and battery life, and is costly, so it is not suitable for conventional drones.
[0036] In addition, the images taken by drones during inspections are often affected by lighting, such as strong light or backlight, which can produce large light or large shadows, thus affecting the image recognition effect. Therefore, when performing image recognition, we should focus on adapting to the impact of brightness and darkness on the image.
[0037] In view of the above problems, this application provides a UAV power inspection method, the overall process of which is as follows: Figure 2 As shown, the following steps are included:
[0038] S201, obtaining a target image recognition model.
[0039] The YOLOv5 network model is a commonly used target detection model. This application optimizes and improves the original YOLOv5 network model, trains and compresses it, and finally obtains a target image recognition model that meets the computing power requirements of the front-end computing device and can better adapt to the influence of brightness and darkness.
[0040] Specifically, the SE attention mechanism is added to the original YOLOv5 network model to improve the model accuracy, and the feature pyramid FPN structure of the original YOLOv5 network model is replaced with a weighted bidirectional feature pyramid structure to better capture feature information of different scales, thereby obtaining an initial image recognition model. When training the initial image recognition model, more images with large illumination or large shadows are added as training samples. The trained image recognition model is compressed to reduce the size of the image recognition model, thereby obtaining the target image recognition model.
[0041] S202, deploying the target image recognition model to a front-end computing device.
[0042] The front-end computing device can transmit data with the UAV flight control terminal, and specifically can be connected by wire through the data output interface of the UAV flight control terminal, so as to obtain the image collected by the UAV. The front-end computing device is also configured to have the functions of pre-processing the obtained image, identifying the image to be identified through the target image recognition model, analyzing the image recognition result to obtain the power inspection result, and sending the power inspection result to the service end.
[0043] S203, controlling the flight of the drone and shooting video through the drone flight control terminal.
[0044] The technician controls the drone to take off and fly to the target area through the drone flight control terminal, and the whole process is continuously filmed, that is, video is shot. In the video shot by the drone, the video stream generally contains 24 frames of images per second, that is, 24 video frames are shot per second. According to the flight speed of the drone, it can be basically ensured that the captured video frames contain recognizable images without the need for fixed-point photography. The drone takes pictures during the entire flight process to obtain more information and avoid omissions.
[0045] S204, identifying and analyzing the current video frame image.
[0046] The drone flight control end obtains the video stream shot by the drone in real time and sends the video stream to the front-end computing device. The front-end computing device identifies and analyzes the current video frame image, obtains the power inspection result, and sends the power inspection result to the server. Since the front-end computing device can obtain the video frame image in real time and perform identification and analysis, it can obtain the power inspection result in time, quickly discover and locate risk hazards, and improve the efficiency of drone power inspection.
[0047] The above is a brief description of the technical solution provided by the present application. The technical solution provided by the present application will be described and illustrated in detail through specific embodiments below.
[0048] First embodiment
[0049] The first embodiment of the present application provides a UAV power inspection method, such as Figure 3 As shown, the method comprises the following steps:
[0050] S301, obtaining an image collected during a power inspection process as an original image, wherein a target scene in the original image includes a target recognition object.
[0051] This step is mainly used to obtain training samples for training the initial image recognition model. The images collected during the power inspection process can be images collected by drones, images collected by staff, or images collected by other smart devices. The images can be video frame images or pictures.
[0052] The images collected during the power inspection are obtained as original images. Specifically, images collected during the power inspection for the target recognition object at different time periods and angles can be obtained as original images. When collecting images for the target recognition object, the brightness of the images collected at different time periods and angles may be quite different, which in turn affects the recognition results of the image recognition model. Therefore, more images of different time periods and angles for the target recognition object are obtained, so that the brightness of the training samples is richer, which is conducive to improving the accuracy of the image recognition model.
[0053] The target scene in the original image includes target recognition objects, specifically, the target recognition objects may be foreign objects such as bird nests and plastic bags on power facilities, rusted anti-vibration hammers, power facilities with missing pins, workers on power facilities who do not wear safety helmets, etc. In other words, the target scene in the original image includes objects that may pose safety risks, such as defective power facilities, foreign objects on power facilities, workers on power facilities who do not wear safety protection equipment, etc.
[0054] S302, adjusting the brightness of the original image to obtain a processed image.
[0055] This step is mainly used to obtain more images with strong illumination or strong shadows and expand the training samples so that the trained image recognition model can better recognize images with strong illumination or strong shadows.
[0056] Image brightness is also known as the light and darkness of the image, that is, the brightness of the picture. In this application, the definition of images with large illumination, normal illumination, and large shadows is as follows: According to the three-channel image data of the original image, the original image is transformed into a grayscale image, and the grayscale value of the pixel is gray = 0.299*R + 0.587*G + 0.114*B; the image mean value of the grayscale image is obtained as M = Mean(gray); if M > a1, it is an image with large illumination, if a2 < M < a1, it is an image with normal illumination, if M < a2, it is an image with large shadows, where a1 and a2 are preset values that can be determined through experiments.
[0057] Adjust the light and darkness of the original image to obtain a processed image. Specifically, transform the original image into a grayscale image and obtain the image mean value of the grayscale image; regard the original image with the image mean value within the preset range as the image to be adjusted, and adjust the light and darkness of the image to be adjusted to obtain a processed image. The original image with the image mean value M within the preset range can be an image with a2 < M < a1, that is, an image with normal illumination.
[0058] Regard the original image with the image mean value within the preset range as the image to be adjusted, and adjust the light and darkness of the image to be adjusted to obtain a processed image. Specifically, obtain the maximum illumination distance and the division distance length in the image to be adjusted. The maximum illumination distance is d1 = 255 - M, and the division distance length is s. s is a preset value that can be determined through experiments; according to the maximum illumination distance and the division distance length, obtain the adjustment quantity, and round down the adjustment quantity n = [d1 / s]; according to the division distance length and the adjustment quantity, adjust the pixels of the image to be adjusted to obtain a processed image, that is, perform n image pixel adjustments on the image to be adjusted. Each time the adjusted pixel value is pixel = pixel + s, and the processed image is obtained after n adjustments.
[0059] Through the above steps, the training samples for training the initial image recognition model are obtained.
[0060] S303, obtain the initial image recognition model. According to the original image and the processed image, train the initial image recognition model and perform compression processing on the trained image recognition model to obtain the target image recognition model for detecting the target recognition object.
[0061] This step is mainly used to obtain the target image recognition model for detecting the target recognition object.
[0062] First, obtain the initial image recognition model. In this application, the initial image recognition model is obtained by improving the original YOLOv5 network model. Specifically as follows:
[0063] Obtain an original YOLOv5 network model; add a SE attention mechanism to the original YOLOv5 network model, and replace the feature pyramid FPN structure of the original YOLOv5 network model with a weighted bidirectional feature pyramid structure to obtain the initial image recognition model.
[0064] The SE attention mechanism mainly averages the feature values of each channel through the global pooling operation on the input feature map. Then, through two fully connected layers, the averaged feature values are mapped to a smaller vector, and then the Sigmoid function is used to normalize it to the range of [0, 1] to obtain the weight of each channel. Finally, the weight is multiplied by the input feature map to obtain a weighted feature map to increase the importance of discriminative and representative features. The advantage of the SE attention mechanism is that it can be applied to any type of convolutional layer and can significantly improve the accuracy of the model, while enhancing the relationship between channels in the feature map to better capture global information.
[0065] The feature pyramid FPN structure of the original YOLOv5 network model is replaced with a weighted bidirectional feature pyramid network structure, and multi-level and multi-scale fusion is performed through bidirectional connection and layer-by-layer iteration. The calculation formula for feature fusion of the weighted bidirectional feature pyramid network structure is specifically as follows:
[0066] F out =ω1F1+ω2F2+ω3F3+...+ω n F n
[0067] Among them, F represents the layer feature, ω represents the feature weight of the layer, and F out Represents the output features after feature fusion.
[0068] It can be seen that the above-mentioned weighted bidirectional feature pyramid network structure, as a feature fusion network, has different importance levels for the features connected at each level and scale in the final output features. The weight parameters are used to learn and update the contribution of different input features to distinguish the importance levels. At the same time, the overall network structure shares the learning weight parameters, and then balances the feature information of different scales to obtain better fusion features for the next step of detection, so that the feature information of different scales can be better captured.
[0069] The initial image recognition model is trained based on the original image and the processed image. Specifically, the original image and the processed image are used as training samples, and the target recognition objects in the training samples are labeled to obtain labeling results; the training samples are input into the initial image recognition model to obtain image recognition results output by the initial image recognition model; and the parameters of the initial image recognition model are adjusted according to the labeling results and image recognition results of the training samples to obtain a trained image recognition model.
[0070] Usually, the labeling tool labelimg can be used to label the target recognition objects in the training samples to obtain the labeling results, that is, the labeling boxes and labels created for the target recognition objects in the training samples.
[0071] After the training sample is input into the initial image recognition model, the model will also form a labeling box for the identified target object, that is, the image recognition result. The parameters of the initial image recognition model are adjusted according to the labeling results and image recognition results of the training sample to obtain the trained image recognition model. Specifically, the value of the image recognition loss function is obtained according to the labeling results and image recognition results of the training sample; the parameters of the initial image recognition model are adjusted according to the value of the image recognition loss function; until the value of the image recognition loss function reaches a preset condition, the parameters of the initial image recognition model are stopped to obtain the trained image recognition model.
[0072] When starting training, the batch size is set to 64, which can be adjusted appropriately according to hardware resources and data set size. The number of learning iterations is 50, and the learning rate is 0.0002. After training, the model error tends to be stable, and the trained image recognition model is obtained.
[0073] The trained image recognition model is compressed to obtain a target image recognition model for detecting a target recognition object. Specifically, the trained image recognition model is compressed by weight quantization to obtain a target image recognition model for detecting a target recognition object.
[0074] Weight quantization processing is to obtain the model format in pt file format after model training, and the format of its weights and related parameters is float type. In order to improve recognition efficiency, it needs to be converted into onnx format and bin format file supported by the terminal. When the onnx format model file is converted to bin file, int8 integer quantization is used to map the value range of the network weight and activation value to (-127~128), which reduces the amount of calculation and improves network recognition efficiency.
[0075] At this point, a target image recognition model for detecting target recognition objects has been obtained. The recognition accuracy and detection speed of the model have been relatively improved. The target image recognition model is deployed in a front-end computing device connected to the data output interface of the UAV flight control end, and the front-end computing device is configured to have the functions of preprocessing the obtained image, recognizing the image to be recognized through the target image recognition model, analyzing the image recognition result to obtain the power inspection result, and sending the power inspection result to the server. In this way, the images collected by the UAV can be recognized and analyzed in real time.
[0076] S304, obtaining in real time the current video frame image collected by the drone during the power inspection process.
[0077] The technician controls the drone to take off and fly along the preset route through the drone flight control terminal to conduct power inspections, and continuously shoots video during the entire process. The drone sends the video stream to the drone flight control terminal in real time, and the drone flight control terminal then sends the video stream to the above-mentioned front-end computing device through the data output interface, that is, the current video frame image collected by the drone during the power inspection process is obtained in real time.
[0078] Optionally, after receiving the video stream, the front-end computing device performs image enhancement processing on the current video frame image to obtain the processed current video frame image as the image to be identified.
[0079] The image enhancement processing performed on the current video frame image may be performed on the current video frame image, that is, suppressing the noise of the current video frame image under the condition of retaining the image detail features as much as possible, so that the target image recognition model can better perform image recognition.
[0080] S305: Obtain a power inspection result for the current video frame image according to the target image recognition model.
[0081] The obtaining of the power inspection result for the current video frame image according to the target image recognition model may specifically be any one or more of the following situations:
[0082] 1) Inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a first type of target recognition object, obtaining a first power inspection result for the current video frame image, wherein the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target recognition object includes at least one of defective power facilities and foreign objects on the power facilities.
[0083] For example, the image recognition result of the target image recognition model for the image to be recognized corresponding to the current video frame image is that the target scene in the image to be recognized includes a bird's nest on the power facility, then the first power inspection result for the current video frame image is that there are safety hazards in the target scene.
[0084] 2) Inputting a plurality of continuous images to be identified within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be identified includes a first type of target recognition object exceeds a first preset threshold, obtaining a first power inspection result for the plurality of current video frame images, wherein the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target recognition object includes at least one of defective power facilities and foreign objects on the power facilities.
[0085] For example, according to the flight speed of the drone, the target scenes in the 24 video frame images taken per second are basically the same, so the preset time is determined to be 1 second, and the video frame images taken for the same target scene within 1 second should contain images that can be recognized. The images to be recognized corresponding to the 24 consecutive current video frame images within the preset time of 1 second are sequentially input into the target image recognition model to obtain 24 image recognition results. If among the 24 image recognition results, the number of image recognition results indicating that the target scene in the image to be recognized includes rusted pins exceeds the preset threshold of 3, then the first power inspection result for the 24 current video frame images is that there is a safety hazard in the target scene.
[0086] 3) Inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a second type of target identification object, obtaining a second power inspection result for the current video frame image, wherein the second power inspection result is used to indicate that the staff in the target scene are not wearing safety protection equipment, and the second type of target identification object includes staff located on power facilities and not wearing safety protection equipment.
[0087] For example, the image recognition result of the target image recognition model for the image to be recognized corresponding to the current video frame image is that the target scene in the image to be recognized includes workers who are located on power facilities and are not wearing safety helmets, then the second power inspection result for the current video frame image is that the workers in the target scene are not wearing safety protection equipment.
[0088] 4) Inputting a plurality of continuous images to be identified within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be identified includes a second type of target recognition object exceeds a second preset threshold, obtaining a second power inspection result for the plurality of current video frame images, the second power inspection result being used to indicate that the staff in the target scene is not wearing safety protection equipment, and the second type of target recognition object includes staff located on power facilities and not wearing safety protection equipment.
[0089] For example, the images to be identified corresponding to 24 consecutive current video frame images within a preset time of 1 second are sequentially input into the target image recognition model to obtain 24 image recognition results. If, among the 24 image recognition results, the number of image recognition results indicating that the target scene in the image to be identified includes a worker who is located on a power facility and is not wearing a safety helmet exceeds a preset threshold of 3, then the second power inspection result for the 24 current video frame images is that the worker in the target scene is not wearing safety protection equipment.
[0090] Of course, if the target image recognition model does not recognize that the target scene in the image to be recognized corresponding to the current video frame image includes the first type of target recognition objects and the second type of target recognition objects, then the power inspection result for the current video frame image may not be obtained, or a third power inspection result for the current video frame image may be obtained, and the third power inspection result is used to indicate that there is no safety hazard in the target scene.
[0091] At this point, the image recognition result has been obtained through the target image recognition model, and the power inspection result is obtained in real time according to the image recognition result. The front-end computing device can send the power inspection result to the service end for further processing by the staff.
[0092] Using the above method to conduct UAV power inspection can effectively improve the real-time and accuracy of UAV power inspection.
[0093] Second embodiment
[0094] The second embodiment of the present application provides a UAV power inspection device, such as Figure 4 As shown, the UAV power inspection device 400 includes:
[0095] An original image acquisition unit 401 is used to acquire an image collected during the power inspection process as an original image, wherein the target scene in the original image includes a target recognition object;
[0096] The original image adjustment unit 402 is used to adjust the brightness of the original image to obtain a processed image;
[0097] A model training unit 403 is used to obtain an initial image recognition model, train the initial image recognition model according to the original image and the processed image, and perform compression processing on the trained image recognition model to obtain a target image recognition model for detecting a target recognition object;
[0098] The current image acquisition unit 404 is used to obtain in real time the current video frame image collected by the drone during the power inspection process;
[0099] The inspection result obtaining unit 405 is used to obtain the power inspection result for the current video frame image according to the target image recognition model.
[0100] Optionally, obtaining the images collected during the power inspection process as original images includes: obtaining images collected during the power inspection process for the target identification object at different time periods and different angles as original images.
[0101] Optionally, the adjusting the brightness and darkness of the original image to obtain a processed image includes: converting the original image into a grayscale image and obtaining an image mean of the grayscale image; taking the original image whose image mean is within a preset range as the image to be adjusted, adjusting the brightness and darkness of the image to be adjusted, and obtaining a processed image.
[0102] Optionally, the original image whose image mean is within a preset range is used as the image to be adjusted, and the brightness and darkness of the image to be adjusted are adjusted to obtain a processed image, including: obtaining the maximum illumination distance and the division distance length in the image to be adjusted; obtaining an adjustment amount according to the maximum illumination distance and the division distance length; adjusting the pixels of the image to be adjusted according to the division distance length and the adjustment amount to obtain a processed image.
[0103] Optionally, obtaining the initial image recognition model includes: obtaining an original YOLOv5 network model; adding an SE attention mechanism to the original YOLOv5 network model, and replacing the feature pyramid FPN structure of the original YOLOv5 network model with a weighted bidirectional feature pyramid structure to obtain the initial image recognition model.
[0104] Optionally, the initial image recognition model is trained based on the original image and the processed image, including: taking the original image and the processed image as training samples, annotating the target recognition objects in the training samples, and obtaining annotation results; inputting the training samples into the initial image recognition model, and obtaining image recognition results output by the initial image recognition model; and adjusting the parameters of the initial image recognition model based on the annotation results and image recognition results of the training samples, and obtaining a trained image recognition model.
[0105] Optionally, adjusting the parameters of the initial image recognition model according to the labeling results and image recognition results of the training samples to obtain the trained image recognition model includes: obtaining the value of the image recognition loss function according to the labeling results and image recognition results of the training samples; adjusting the parameters of the initial image recognition model according to the value of the image recognition loss function; and stopping adjusting the parameters of the initial image recognition model until the value of the image recognition loss function reaches a preset condition to obtain the trained image recognition model.
[0106] Optionally, compressing the trained image recognition model to obtain a target image recognition model for detecting a target recognition object includes: compressing the trained image recognition model by weight quantization to obtain a target image recognition model for detecting a target recognition object.
[0107] Optionally, the device further includes: an image preprocessing unit, configured to perform image enhancement processing on the current video frame image to obtain the processed current video frame image as the image to be identified.
[0108] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a first type of target identification object, obtaining a first power inspection result for the current video frame image, the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target identification objects includes at least one of defective power facilities and foreign objects on the power facilities.
[0109] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting a plurality of consecutive images to be recognized within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if, among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be recognized includes a first type of target recognition object exceeds a first preset threshold, then obtaining a first power inspection result for the plurality of current video frame images, the first power inspection result being used to indicate that there are safety hazards in the target scene, and the first type of target recognition objects includes at least one of defective power facilities and foreign objects on the power facilities.
[0110] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting the image to be identified into the target image recognition model to obtain an image recognition result; if the image recognition result indicates that the target scene in the image to be identified includes a second type of target identification object, obtaining a second power inspection result for the current video frame image, the second power inspection result is used to indicate that the staff in the target scene are not wearing safety protection equipment, and the second type of target identification objects include staff who are located on the power facilities and are not wearing safety protection equipment.
[0111] Optionally, obtaining the power inspection result for the current video frame image based on the target image recognition model includes: inputting a plurality of consecutive images to be identified within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; if, among the plurality of image recognition results, the number of image recognition results indicating that the target scene in the image to be identified includes a second category of target recognition objects exceeds a second preset threshold, then obtaining a second power inspection result for the plurality of current video frame images, the second power inspection result being used to indicate that the staff in the target scene is not wearing safety protection equipment, and the second category of target recognition objects includes staff that are located on power facilities and are not wearing safety protection equipment.
[0112] Optionally, the device further includes: a sending unit, configured to send the power inspection result to a service end.
[0113] The above-mentioned device corresponds to the UAV power inspection method provided in the first embodiment. Since the device embodiment is similar to the method embodiment, it is described here in a relatively simple manner. For relevant matters, please refer to the content of the first embodiment, and no further details will be given here.
[0114] Third embodiment
[0115] The third embodiment of the present application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and after the processor runs the computer program, it executes the drone power inspection method provided in the first embodiment of the present application.
[0116] See also Figure 5 As shown, Figure 5 It is a structural diagram of an electronic device embodiment provided by the present application, and the electronic device includes: at least one processor 501, at least one memory 502, at least one communication interface 503 and at least one communication bus 504. Optionally, the processor 501 may be a processor CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The memory 502 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication interface 503 can be an interface of a communication module, such as an interface of a GSM module. Among them, the memory 502 stores computer programs and data, and the processor 501 calls the computer program stored in the memory 502 to execute the above-mentioned UAV power inspection method.
[0117] Fourth embodiment
[0118] The fourth embodiment of the present application provides a computer storage medium, which stores a computer program. After the computer program is run by a processor, the drone power inspection method provided in the first embodiment of the present application is executed.
[0119] Although several modules or units for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the specific implementation of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being concretized by multiple modules or units.
[0120] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0121] It should be noted that the embodiments of the present application can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present application can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0122] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by any technician familiar with the technical field within the technical scope disclosed in the present application and within the spirit and principle of the present application shall be covered by the protection scope of the present application.
Claims
1. A UAV power inspection method, characterized in that: include: Obtaining an image collected during the power inspection process as an original image, wherein the target scene in the original image includes a target recognition object; Adjusting the brightness and darkness of the original image to obtain a processed image; Obtaining an initial image recognition model, training the initial image recognition model according to the original image and the processed image, and compressing the trained image recognition model to obtain a target image recognition model for detecting a target recognition object; Obtain the current video frame image collected by the drone during the power inspection process in real time; According to the target image recognition model, a power inspection result for the current video frame image is obtained.
2. The UAV power inspection method according to claim 1 is characterized in that: The obtaining of the image collected during the power inspection as the original image includes: Images of the target recognition object at different time periods and angles collected during the power inspection process are obtained as original images.
3. The UAV power inspection method according to claim 1, characterized in that: The step of adjusting the brightness of the original image to obtain a processed image includes: Convert the original image into a grayscale image and obtain an image mean of the grayscale image; An original image whose image mean is within a preset range is used as an image to be adjusted, and the brightness of the image to be adjusted is adjusted to obtain a processed image.
4. The UAV power inspection method according to claim 3 is characterized in that: The method of taking an original image whose image mean is within a preset range as an image to be adjusted, adjusting the brightness of the image to be adjusted, and obtaining a processed image includes: Obtaining the maximum illumination distance and the length of the division distance in the image to be adjusted; Obtaining an adjustment amount according to the maximum illumination distance and the length of the divided distance; According to the division distance length and the adjustment amount, the pixels of the image to be adjusted are adjusted to obtain a processed image.
5. The UAV power inspection method according to claim 1, characterized in that: The obtaining of the initial image recognition model comprises: Get the original YOLOv5 network model; The SE attention mechanism is added to the original YOLOv5 network model, and the feature pyramid FPN structure of the original YOLOv5 network model is replaced with a weighted bidirectional feature pyramid structure to obtain the initial image recognition model.
6. The UAV power inspection method according to claim 1, characterized in that: The training of the initial image recognition model according to the original image and the processed image includes: Using the original image and the processed image as training samples, marking target recognition objects in the training samples to obtain marking results; Inputting the training sample into the initial image recognition model to obtain an image recognition result output by the initial image recognition model; According to the labeling results and image recognition results of the training samples, the parameters of the initial image recognition model are adjusted to obtain a trained image recognition model.
7. The UAV power inspection method according to claim 6, characterized in that: The step of adjusting the parameters of the initial image recognition model according to the labeling results and the image recognition results of the training samples to obtain a trained image recognition model includes: Obtaining a value of an image recognition loss function according to the labeling results and image recognition results for the training samples; Adjusting the parameters of the initial image recognition model according to the value of the image recognition loss function; Until the value of the image recognition loss function reaches a preset condition, stop adjusting the parameters of the initial image recognition model to obtain a trained image recognition model.
8. The UAV power inspection method according to claim 1, characterized in that: The method of compressing the trained image recognition model to obtain a target image recognition model for detecting a target recognition object includes: The trained image recognition model is compressed by weight quantization to obtain a target image recognition model for detecting the target recognition object.
9. The UAV power inspection method according to claim 1, characterized in that: Also includes: Perform image enhancement processing on the current video frame image to obtain the processed current video frame image as the image to be recognized.
10. The UAV power inspection method according to claim 1, characterized in that: The step of obtaining a power inspection result for the current video frame image according to the target image recognition model includes: Inputting the image to be recognized into the target image recognition model to obtain an image recognition result; If the image recognition result indicates that the target scene in the image to be identified includes a first type of target identification object, a first power inspection result for the current video frame image is obtained, and the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target identification object includes at least one of defective power facilities and foreign objects on the power facilities.
11. The UAV power inspection method according to claim 1, characterized in that: The step of obtaining a power inspection result for the current video frame image according to the target image recognition model includes: Inputting a plurality of consecutive images to be recognized within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; If, among the multiple image recognition results, the number of image recognition results indicating that the target scene in the image to be recognized includes a first type of target recognition object exceeds a first preset threshold, a first power inspection result for the multiple current video frame images is obtained, and the first power inspection result is used to indicate that there are safety hazards in the target scene, and the first type of target recognition objects includes at least one of defective power facilities and foreign objects on the power facilities.
12. The UAV power inspection method according to claim 1, characterized in that: The step of obtaining a power inspection result for the current video frame image according to the target image recognition model includes: Inputting the image to be recognized into the target image recognition model to obtain an image recognition result; If the image recognition result indicates that the target scene in the image to be identified includes a second type of target identification object, a second power inspection result for the current video frame image is obtained, and the second power inspection result is used to indicate that the staff in the target scene is not wearing safety protection equipment, and the second type of target identification object includes staff who are located on the power facilities and are not wearing safety protection equipment.
13. The UAV power inspection method according to claim 1, characterized in that: The step of obtaining a power inspection result for the current video frame image according to the target image recognition model includes: Inputting a plurality of consecutive images to be recognized within a preset time into the target image recognition model in sequence to obtain a plurality of image recognition results; If, among the multiple image recognition results, the number of image recognition results indicating that the target scene in the image to be recognized includes a second category of target recognition objects exceeds a second preset threshold, a second power inspection result for the multiple current video frame images is obtained, and the second power inspection result is used to indicate that the staff in the target scene is not wearing safety protection equipment, and the second category of target recognition objects includes staff who are located on the power facilities and are not wearing safety protection equipment.
14. The UAV power inspection method according to claim 1, characterized in that: Also includes: The power inspection result is sent to the service end.
15. An unmanned aerial vehicle power inspection device, characterized in that: include: An original image acquisition unit, used to acquire an image collected during the power inspection process as an original image, wherein the target scene in the original image includes a target recognition object; An original image adjustment unit, used to adjust the brightness of the original image to obtain a processed image; A model training unit, used to obtain an initial image recognition model, train the initial image recognition model according to the original image and the processed image, and perform compression processing on the trained image recognition model to obtain a target image recognition model for detecting a target recognition object; A current image acquisition unit is used to obtain in real time the current video frame image collected by the drone during the power inspection process; The inspection result obtaining unit is used to obtain the power inspection result for the current video frame image according to the target image recognition model.
16. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and after the processor runs the computer program, it executes the UAV power inspection method described in any one of claims 1-14.
17. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and after the computer program is run by the processor, the unmanned aerial vehicle power inspection method described in any one of claims 1-14 is executed.
Citation Information
Patent Citations
Power line inspection method and inspection platform
CN115393273A
Image foreign matter detection method and device, electronic equipment and storage medium
CN116453096A
Lightweight YOLOv5-based insulator defect detection method
CN117132531A
Power transmission line unmanned aerial vehicle autonomous inspection method and device based on artificial intelligence
CN117173447A
Image recognition model training method, image recognition method and related product
WO2024083033A1
