A method and system for unmanned aerial vehicle video inspection

Abstract

The application relates to the technical field of intelligent inspection, in particular to a UAV video inspection method and system, which comprises the following steps: obtaining target data of a region to be inspected, and generating a plurality of inspection photo sets based on the target data; sequentially obtaining the inspection photos, and judging whether the inspection photos belong to the to-be-reused ones based on a use mark; if the inspection photos belong to the to-be-reused ones, obtaining a facility to be marked based on the inspection photos; performing target detection on the facility to be marked based on a target detection technology to obtain a detection result, and judging whether the facility to be marked meets the inspection requirements based on the detection result; repeatedly marking the facility to be marked, and updating the use mark based on the repeated marking to obtain an updated inspection photo set. The application reduces the data amount of image processing of the target data, continuously inspects the region to be inspected according to the updated inspection photos, improves the inspection efficiency of the region to be inspected, and improves the use rate of the inspection photos of the target data.

Description

Technical Field

[0001] This application relates to the technical field of intelligent inspection, and in particular to a method and system for video inspection based on unmanned aerial vehicles (UAVs). Background Technology

[0002] As cities develop and expand, municipal roads become longer and longer, and related municipal road traffic facilities become more and more numerous. However, the number of road facility management personnel and the area under their supervision are limited. This leads to problems such as difficulty in detecting defects in municipal road traffic facilities, slow repair response, troublesome daily maintenance operations, and difficulty in collecting facility information. In some cases, maintenance personnel cannot even detect serious defects in time, and pedestrians who discover defects do not have effective means to report them, resulting in long-term damage to traffic facilities and serious consequences such as traffic accidents that endanger life safety.

[0003] Currently, intelligent inspection methods are being developed that utilize drones equipped with various sensors (such as high-definition cameras, temperature and humidity sensors, vibration sensors, etc.) to conduct real-time detection, data collection, and fault diagnosis of target transportation facilities.

[0004] Currently, the multimodal data processing methods collected during drone inspections are too simplistic, making it difficult to effectively integrate image, sensor, and location information. This results in an incomplete analysis of equipment status and limited fault detection capabilities. In particular, it is often difficult to accurately identify complex fault modes, lacks multiple uses of image data, and cannot efficiently utilize the detected image data. Summary of the Invention

[0005] In order to reduce the amount of image data processing and improve the utilization efficiency of acquired image data, this application provides a method and system for video inspection based on unmanned aerial vehicles (UAVs).

[0006] Firstly, this application provides a method for unmanned aerial vehicle (UAV) video inspection, employing the following technical solution: A drone-based video inspection method includes the following steps: Acquire target data for the area to be tested, and generate several sets of inspection photos based on the target data. Each set of inspection photos includes at least one set of inspection photos and corresponding inspection marks. The inspection marks include usage marks corresponding to the inspection photos. The inspection photos are acquired sequentially, and it is determined whether the inspection photos are to be reused based on the usage marker. If the inspection photos are to be reused, then the facilities to be marked are obtained based on the inspection photos; The facility to be marked is detected using target detection technology to obtain the detection results, and the facility to be marked is judged to meet the inspection requirements based on the detection results. If the facility to be marked is determined to meet the inspection requirements, the facility to be marked is marked again, and the usage mark is updated based on the repeated marking to obtain an updated set of inspection photos.

[0007] By adopting the above technical solution, the target data of the area to be tested is divided to generate a set of inspection photos. Based on the usage markers, it is determined whether the inspection photos are to be reused. If the inspection photos are to be reused, facilities to be marked are obtained based on the inspection photos. Target detection technology is used to detect the facilities to be marked to obtain detection results, and based on the detection results, it is determined whether the facilities to be marked meet the inspection requirements. If the facilities to be marked meet the inspection requirements, they are marked again, and the usage markers are updated based on the repeated markings. This reduces the amount of image processing data for the target data, improving the inspection efficiency of the area to be tested by continuously inspecting the area according to the updated inspection photos. Furthermore, updating the usage markers based on the repeated markings to obtain an updated set of inspection photos allows for the selection of reused photos from the already acquired target data, improving the utilization rate of the inspection photos for the target data.

[0008] In some embodiments, the determination of whether the inspection photo is to be reused is based on the usage tag, wherein the method of obtaining the usage tag includes the following steps: The inspection photos are used to perform target detection based on target detection technology to obtain several groups of facilities to be inspected. The corresponding confidence level is obtained based on the facility to be tested, and a usage tag is obtained based on the confidence level and a preset confidence value; When the confidence level of the facility to be inspected is greater than the preset confidence value, the usage mark corresponding to the inspection photo is incremented by 1.

[0009] By adopting the above technical solution, target detection is performed on the inspection photos based on target detection technology to obtain the facility to be detected and the confidence detection degree corresponding to the facility to be detected. The confidence detection degree is compared with a preset confidence value. When the confidence detection degree corresponding to the facility to be detected is greater than the preset confidence value, the usage mark corresponding to the inspection photo is incremented by 1, thereby obtaining the usage mark corresponding to the inspection photo. This can accurately determine whether the inspection photo belongs to the category of reusable inspection photos, thereby reducing the number of reusable inspection photos and improving the utilization efficiency of the acquired image data.

[0010] In some embodiments, determining whether an inspection photo is to be reused based on the usage marker includes the following steps: The usage marker is compared with a preset marker value, and it is determined whether the usage marker exceeds the preset marker value; If the usage mark exceeds the preset mark value, the inspection photo is determined to be reusable; If the usage mark does not exceed the preset mark value, then the inspection photo is determined not to be reused.

[0011] By adopting the above technical solution, a marker is used to indicate the facilities to be inspected that meet the inspection requirements in the inspection photo. If the number of markers exceeds the preset marker value, the inspection photo is determined to be reusable; if the number of markers does not exceed the preset marker value, the inspection photo is determined not to be reusable. Therefore, by using markers to determine whether an inspection photo is reusable, it is possible to make a judgment based on the actual facilities to be inspected contained in the inspection photo, and reuse the existing inspection photos, which facilitates rapid and efficient inspection operations of the area to be tested in the future.

[0012] In some embodiments, after determining whether the inspection photo is to be reused based on the usage marker, the following steps are also included: If it is determined that the usage mark corresponding to the inspection photo does not belong to the category of photos to be reused, then it is determined whether the inspection photo belongs to the category of photos to be used alone based on the usage mark. If the inspection photos are intended for individual use, then facilities to be inspected are generated based on the inspection photos; The facilities to be inspected are screened among the facilities to be marked to obtain the screening results, and it is determined whether there are duplicate markings in the screening results; If the screening results contain duplicate markers, a rejection signal is generated, and the inspection photos are rejected based on the rejection signal to update the target data.

[0013] By adopting the above technical solution, if the usage mark corresponding to the inspection photo is not to be reused, it is necessary to determine whether the inspection photo is to be used alone. If so, it is necessary to filter among the facilities to be marked to determine whether it can be found. If it can be found, it means that the facility to be inspected is already included in the facilities to be marked. Therefore, the inspection photo corresponding to the facility to be marked needs to be deleted to reduce the repeated operation on the facilities to be inspected and improve the overall inspection efficiency of the area to be tested.

[0014] In some embodiments, determining whether an inspection photo is to be reused based on the usage marker includes the following steps: The usage marker is compared with a preset marker value, and it is determined whether the usage marker exceeds the preset marker value; If the usage mark exceeds the preset mark value, the inspection photo is determined to be reusable; If the usage mark does not exceed the preset mark value, then the inspection photo is determined not to be reused.

[0015] By adopting the above technical solution, a marker is used to indicate the facilities to be inspected that meet the inspection requirements in the inspection photo. If the number of markers exceeds the preset marker value, the inspection photo is determined to be reusable; if the number of markers does not exceed the preset marker value, the inspection photo is determined not to be reusable. Therefore, by using markers to determine whether an inspection photo is reusable, it is possible to make a judgment based on the actual facilities to be inspected contained in the inspection photo, and reuse the existing inspection photos, which facilitates rapid and efficient inspection operations of the area to be tested in the future.

[0016] In some embodiments, after generating the facility to be inspected based on the inspection photos, the following steps are also included: Based on the facility to be inspected, obtain the corresponding candidate facility in the target data, generate an estimated signal based on the candidate facility, and determine whether the candidate facility and the facility to be inspected are in the same frame based on the estimated signal. If the candidate facility and the facility to be inspected can be in the same frame, a shooting angle is generated based on the candidate facility and the facility to be inspected, and the inspection photo is marked with a different label based on the shooting angle.

[0017] By adopting the above technical solution, the corresponding candidate facilities are obtained from the target data based on the facility to be inspected, a prediction signal is generated based on the candidate facilities, and it is determined whether the candidate facilities and the facility to be inspected are in the same frame based on the prediction signal. If the candidate facilities and the facility to be inspected can be in the same frame, a shooting angle is generated based on the candidate facilities and the facility to be inspected, and the inspection photo is marked with a new label based on the shooting angle. By filtering the facility to be inspected in the target data, it is determined whether the candidate facilities and the facility to be inspected are in the same frame based on the prediction signal. If so, the inspection photo is marked with a new label based on the shooting angle, which facilitates the subsequent updating of the inspection photos and can reduce the number of inspection photos in the area to be tested, thereby improving the inspection efficiency of the area to be tested.

[0018] In some embodiments, after changing the markings on the inspection photos based on the shooting angle, the following steps are also included: Based on the area to be tested, obtain the corresponding set of inspection photos, and determine whether there are replacement marks in the inspection photos; If the inspection photos have replacement marks, the facilities to be collected are obtained based on the area to be tested, and the corresponding standard shooting points are obtained from the historical database based on the facilities to be collected. The facilities to be collected are divided based on the standard shooting points to obtain a set of shooting areas, which includes several groups of shooting areas. An inspection operation is performed based on the set of shooting areas to obtain the target data corresponding to the set of shooting areas.

[0019] By adopting the above technical solution, a set of corresponding inspection photos is obtained based on the area to be tested, and it is determined whether the inspection photos have replacement marks. If the inspection photos have replacement marks, it means that the number of inspection photos in the area to be tested is not optimal. The number of inspection photos in the area to be tested can be reduced by updating the inspection photos. Therefore, it is necessary to obtain the standard shooting points of the facilities to be collected from the historical database of the area to be tested, and obtain the shooting area set by analyzing several standard shooting points. This ensures that all facilities to be collected in the area to be tested are inspected, and the number of inspection photos to be collected is minimized, thereby reducing the amount of image data processing and improving the inspection efficiency of the area to be tested.

[0020] In some embodiments, the facility to be captured is divided based on the standard shooting locations to obtain a set of shooting areas, including the following steps: Based on the facilities to be collected, several groups of key facilities are selected, and these key facilities serve as the main inspection facilities within the shooting area. Based on the key facilities and corresponding standard shooting locations as the basic values, and based on the minimum number of elements corresponding to the shooting area set as the constraint condition; Based on the aforementioned basic values ​​and constraints, the shooting angles corresponding to the key facilities are adaptively adjusted to obtain the shooting area and the corresponding shooting angles.

[0021] By adopting the above technical solution, key facilities are selected based on the facilities to be collected, and the shooting angle is adaptively adjusted based on the key facilities, standard shooting points, and the number of elements in the shooting area set, thereby obtaining the shooting area and the corresponding shooting angle. Based on the shooting angle, inspection photos of the area under test are collected, resulting in the fewest number of inspection photos, which can cover all the facilities to be collected in the area under test, thereby improving the inspection efficiency of the area under test.

[0022] In some embodiments, an inspection operation is performed based on the set of shooting areas to obtain target data corresponding to the set of shooting areas, including the following steps: Fault prediction is performed based on the historical fault data corresponding to the area to be tested to generate an estimated fault time, and the inspection frequency corresponding to the set of shooting areas is generated based on the estimated fault time. A shooting route is generated based on the inspection frequency and shooting angle corresponding to the set of shooting areas. The shooting route is sent to the drone, and the target data corresponding to the shooting area set is obtained based on the drone's inspection.

[0023] By adopting the above technical solution, fault prediction is performed based on historical fault data of the area under test, thereby generating an estimated fault time. Based on the estimated fault time, an inspection frequency corresponding to the set of shooting areas is generated. Based on the inspection frequency and shooting angle of the set of shooting areas, a shooting route is generated and sent to the drone. The drone inspects and obtains target data corresponding to the set of shooting areas. By adjusting the drone's shooting route through the inspection frequency and shooting angle, the frequency of obtaining inspection photos of the area under test can be improved. Furthermore, by filtering the video obtained from the target data captured by the drone according to the shooting angle, inspection photos can be accurately obtained, thereby improving the inspection efficiency of the area under test.

[0024] Secondly, this application provides a drone-based video inspection system, employing the following technical solution: A drone-based video inspection system, performing the drone-based video inspection method described in the first aspect, includes: The data acquisition module is used to acquire target data of the area to be tested and generate several sets of inspection photos based on the target data. The set of inspection photos includes at least one set of inspection photos and corresponding inspection marks. The inspection marks include usage marks corresponding to the inspection photos. The data judgment module is used to sequentially acquire the inspection photos and determine whether the inspection photos belong to those to be reused based on the usage marker. The data processing module, if the inspection photo is to be reused, is used to obtain the facility to be marked based on the inspection photo; The facility to be marked is detected using target detection technology to obtain the detection results, and the facility to be marked is judged to meet the inspection requirements based on the detection results. If the detection result determines that the facility to be marked meets the inspection requirements, the data processing module will repeatedly mark the facility to be marked and update the usage mark based on the repeated mark to obtain an updated set of inspection photos.

[0025] In some embodiments, a maintenance storage module is also included, which is used to store inspection photos of multiple traffic facilities, wherein the inspection photos correspond to inspection times, and to generate inspection signals based on the inspection photos and to generate work schedules based on the inspection signals.

[0026] In summary, this application includes at least one of the following beneficial technical effects: 1. The target data of the area to be tested is divided to generate a set of inspection photos. Based on usage tags, it is determined whether the inspection photos are to be reused. If the inspection photos are to be reused, facilities to be marked are obtained based on the inspection photos. Target detection technology is used to detect the facilities to be marked to obtain detection results. Based on the detection results, it is determined whether the facilities to be marked meet the inspection requirements. If the facilities to be marked meet the inspection requirements, they are marked again, and the usage tags are updated based on the repeated markings. This reduces the amount of image processing data for the target data, improving the inspection efficiency of the area to be tested by continuously inspecting the area according to the updated inspection photos. Furthermore, updating the usage tags based on the repeated markings to obtain an updated set of inspection photos allows for the selection of reused photos from the already acquired target data, improving the utilization rate of the inspection photos for the target data. 2. Use markers to indicate the facilities to be inspected that meet the inspection requirements in the inspection photos. If the number of markers used exceeds the preset marker value, the inspection photos are determined to be reusable; if the number of markers used does not exceed the preset marker value, the inspection photos are determined not to be reusable. Therefore, by using markers to determine whether an inspection photo is reusable, it is possible to make a judgment based on the actual facilities to be inspected contained in the inspection photos, and reuse existing inspection photos, which facilitates rapid and efficient inspection operations of the area to be tested in the future. Attached Figure Description

[0027] Figure 1 This is a block diagram of the UAV video inspection method provided in the embodiments of this application; Figure 2 This is a block diagram of a method for updating target data provided in an embodiment of this application; Figure 3 This is a block diagram of a better marker generation method provided in the embodiments of this application; Figure 4 This is a block diagram of the target data acquisition method provided in the embodiments of this application; Figure 5 This is a block diagram of a method for obtaining a set of shooting areas provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of the UAV-based video inspection system provided in this embodiment.

[0028] Explanation of reference numerals in the attached diagram: 10, Data acquisition module; 20, Data judgment module; 30, Data processing module; 40, Maintenance and storage module. Detailed Implementation

[0029] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0030] This application discloses a method for video inspection based on unmanned aerial vehicles (UAVs).

[0031] like Figure 1 As shown, the drone-based video inspection method includes the following steps: S100: Acquire target data for the area to be tested, and generate several sets of inspection photos based on the target data.

[0032] The area to be tested represents the area that needs to be inspected. This area can be divided based on the number of facilities that need to be inspected, specifically the number of facilities that can be inspected in a single operation. "Single operation" can refer to the number of facilities inspected in a single drone flight or the number of facilities inspected by staff in a single operation.

[0033] It's important to note that the number of inspection facilities can be set based on their specific locations within the area. Furthermore, the number of inspection facilities can be adjusted when dividing the testing area. A floating value can be set for the number of facilities, ensuring they are within a preset range of the center of the testing area and on the inspection route. This guarantees that inspections are carried out on the designated route without needing to start new routes, thus maximizing the number of facilities monitored in a single inspection, improving monitoring efficiency, and enabling rapid response and maintenance when problems arise.

[0034] The target data is inspection data obtained based on the area to be inspected. This target data can be photos and / or videos taken by personnel and / or drones of the inspected facilities. The collection of inspection photos includes at least one set of inspection photos and corresponding inspection marks, including usage marks corresponding to the inspection photos. The inspection marks include usage marks, which characterize the facilities to be inspected in the inspection photos that meet the photo clarity requirements.

[0035] It's important to note that when the target data is photographs, it is directly used as the inspection photo set. However, if the target data includes video data, photo extraction is required to obtain the corresponding inspection photo set. To obtain the inspection photo set based on video data, VLC media player can be used. Manually open the video file and play it to the desired frame. Then, take a screenshot of the frame, automatically save the location, and store the screenshot in the image folder, automatically setting the appropriate processing format. The processing format simply needs to be converted to a format that can be read later; no specific restrictions are imposed here.

[0036] Of course, PotPlayer can also be used to extract photos from video data. Pause the video to the target frame, save the target frame using a keyboard shortcut, and then select the appropriate photo format. Alternatively, command-line tools such as FFmpeg can be used to batch extract video data, setting a preset frame rate to identify the playing video and obtain the corresponding inspection photos.

[0037] Since this embodiment acquires multiple inspection photos from video data, a command-line tool is used, which is suitable for batch extraction. The specific steps are as follows: retrieve and start playing the video data, and identify the playing video data according to a preset frame rate to obtain the corresponding inspection photos.

[0038] It should be noted that the preset frame rate can be adjusted based on the identified inspection facilities. Since the video data is continuous, the video data is processed according to the playback order based on the actual number of video frames processed. First, the first target photo is acquired, target detection is performed on the target photo, the corresponding target facilities are extracted, and the target facilities are marked to obtain a marked facility set. Next, the next frame of the target photo is acquired, and the above processing method is used to obtain the corresponding comparison facility set. The marked facility set and the comparison facility set are compared for similarity. When the comparison facility set contains the marked facility set, and the confidence level of the comparison facilities meets the preset sharpness, the comparison facility set is used as the marked facility set, and the comparison is performed on the next frame of the target photo.

[0039] Only when the comparison facility set does not include the marker facility set and / or the confidence level of the comparison facility does not meet the preset sharpness, will the target photo corresponding to the marker facility set be used as an inspection photo. Since the inspection of the area under test is a periodic and repetitive operation, to facilitate rapid completion of the video data for the area under test during subsequent video data processing, the number of comparisons required to obtain the inspection photo and the actual number of video frames processed can be used as the preset frame number to quickly obtain inspection photos during subsequent inspections.

[0040] Of course, the video data acquisition method must be the same as the original video data. Using the same acquisition method ensures that the corresponding inspection photos can be accurately acquired when using the preset frame rate, thus guaranteeing the accuracy of the inspection within the area to be tested. The preset resolution mentioned above refers to the resolution of the target facilities in the target photos, and should be set according to the actual situation.

[0041] In one embodiment, determining whether an inspection photo is to be reused is based on the use of tags, wherein the method of obtaining the tags includes the following steps: S110 uses target detection technology to perform target detection on inspection photos to obtain several sets of facilities to be inspected.

[0042] S120: Obtain the corresponding confidence level based on the facility to be tested, and obtain the usage mark based on the confidence level and the preset confidence value.

[0043] S130, when the confidence level of the facility to be inspected is greater than the preset confidence value, the usage mark corresponding to the inspection photo is incremented by 1.

[0044] The inspection photos contain several facilities to be inspected. Therefore, target detection needs to be performed on the inspection photos to identify the corresponding facilities. It should be noted that target detection based on target detection technology is mainly performed using existing techniques, which will not be elaborated upon here. The confidence level is the level of clarity of the detected facility, while the preset confidence value represents the standard value for meeting the photo's clarity requirements; this is selected based on the actual situation. When the confidence level of the facility to be inspected is greater than the preset confidence value, the usage flag for the inspection photo is incremented by 1. Similarly, if the confidence level is equal to the preset confidence value, the usage flag is also incremented by 1. If the confidence level is less than the preset confidence value, no action is taken on the usage flag, and its initial value is set to 0.

[0045] S200: Sequentially acquire inspection photos and determine whether the inspection photos are to be reused based on the usage tags.

[0046] The inspection facilities in the area to be tested are inspected regularly, and the inspection photos need to be uploaded to the inspection system. The inspection system is set up with different inspection modules according to different inspection facilities. The inspection photos can be uploaded in chronological order and stored in the corresponding inspection module.

[0047] Because the target data acquired from the area under test contains a large number of inspection photos, manually uploading each photo to the corresponding inspection storage system would be extremely labor-intensive. If staff directly upload the photos taken on-site without review, the quality of the photos would be compromised. Reusable inspection photos still require manual review to determine which storage system to upload to, a time-consuming process, and the accuracy of determining compliance with inspection standards is not always guaranteed by human eyes.

[0048] Therefore, in one embodiment, using markers to characterize facilities in inspection photos that meet the inspection requirements, and determining whether an inspection photo is to be reused based on the use of markers, includes the following steps: S210, compare the used tag with the preset tag value, and determine whether the used tag exceeds the preset tag value.

[0049] S220: If the number of markers used exceeds the preset marker value, the inspection photo is determined to be reusable.

[0050] S230: If the number of markers used does not exceed the preset marker value, the inspection photo is determined not to be reused.

[0051] The preset marker value represents the minimum standard value for determining whether something is reusable. In this embodiment, the preset marker value is set to 2. A preset marker value of 2 indicates that there are two facilities in the inspection photo that meet the inspection requirements.

[0052] S300: If the inspection photos are to be reused, then the facilities to be marked are obtained based on the inspection photos.

[0053] The facilities to be marked represent facilities that can be inspected using inspection photos. When the number of usage marks in an inspection photo exceeds a preset mark value, the inspection photo is determined to be reusable, and the corresponding facilities to be marked are identified based on the inspection photo.

[0054] It should be noted that, in order to facilitate the acquisition of facilities to be marked, when performing target detection on inspection photos and acquiring usage marks, the usage mark can be directly incremented by 1, and the inspection photos can be marked with facilities based on the corresponding facilities to be detected. When it is determined that the inspection photo belongs to the category of facilities to be reused, the corresponding facilities to be marked can be directly obtained based on the facility marks of the inspection photo.

[0055] The S400 uses target detection technology to perform target detection on the facilities to be marked in order to obtain the detection results, and then determines whether the facilities to be marked meet the inspection requirements based on the detection results.

[0056] The test results characterize whether the facility to be labeled meets the inspection requirements of the system. The results include compliant and non-compliant inspections. A compliant inspection means the confidence level of the facility exceeds the inspection standard value, while a non-compliant inspection means the confidence level does not exceed the inspection standard value. This inspection standard value is set based on the requirements of different facilities to be labeled or different inspection systems. Unlike the preset confidence level, which represents the standard value for meeting photographic sharpness requirements, this standard value is different.

[0057] S500: If it is determined that the facility to be marked meets the inspection requirements, the facility to be marked is marked again, and the marking is updated based on the repeated marking to obtain an updated set of inspection photos.

[0058] Among them, the repeated mark indicates that there are marks that have been used multiple times in the inspection photo. The inspection marks include the use mark and the repeated mark. The use mark indicates that there are inspection facilities in the inspection photo that meet the clarity requirements, while the repeated mark indicates that there are inspection facilities in the inspection photo that meet the inspection standards.

[0059] It's important to clarify that the process of marking facilities repeatedly involves checking inspection photos for duplicate markings. If duplicate markings exist, the facility to be marked is added to the duplicate marking list of the inspection photo. Conversely, if no duplicate markings exist, the inspection photo is marked again, and the facility to be marked is added to the duplicate marking list. The process involves updating the usage markers based on duplicate markings to obtain an updated set of inspection photos. Specifically, unprocessed inspection photos are first obtained from the set, along with the facilities to be marked based on duplicate markings. These unprocessed photos are then filtered to determine if a matching facility is found. If a matching facility is found, the usage marker for that photo is decremented by 1. If no matching facility is found, no further processing is required for the unprocessed photos.

[0060] Reference Figure 2 In one embodiment, after determining whether an inspection photo is to be reused based on the usage marker, the following steps are also included: S310, if it is determined that the usage mark corresponding to the inspection photo does not belong to the category of photos to be reused, then the inspection photo is determined to be used alone based on the usage mark.

[0061] S320: If the inspection photo is to be used separately, then generate the facility to be inspected based on the inspection photo.

[0062] S330: The facilities to be inspected are screened among the facilities to be marked in order to obtain the screening results and determine whether there are duplicate markings in the screening results.

[0063] S340, if there are duplicate markers in the screening results, a rejection signal is generated, and the inspection photos are rejected based on the rejection signal to update the target data.

[0064] If a usage marker in an inspection photo does not indicate a facility to be inspected, it means that the photo may or may not contain such a facility. If an inspection photo is for single use, it means that the facility to be inspected in that photo is unique. In this case, facilities to be inspected need to be generated based on the inspection photo. These facilities are then filtered against those to be marked, and the results are checked for duplicate markers. If duplicate markers are found, a rejection signal is generated, and the inspection photo is removed based on this signal to update the target data. If no duplicate markers are found, it means that the inspection photo contains only one unique facility to be inspected.

[0065] It should be noted that the rejection signal is mainly generated by the processor. The rejection signal is mainly used to perform rejection operations on the inspection photos, thereby reducing the number of inspection photos in the target data and reducing the amount of target data in the area to be tested.

[0066] Reference Figure 3 In one embodiment, after generating the facility to be inspected based on the inspection photos, the following steps are also included: S350 obtains the corresponding candidate facilities from the target data based on the facility to be inspected, generates a prediction signal based on the candidate facilities, and determines whether the candidate facilities and the facility to be inspected are in the same frame based on the prediction signal.

[0067] S360: If the candidate facility and the facility to be inspected can be in the same frame, a shooting angle is generated based on the candidate facility and the facility to be inspected, and the inspection photo is marked with a different label based on the shooting angle.

[0068] Among them, the facilities to be inspected represent those that are not in the inspection photos to be reused, while the facilities to be selected are the facilities that exist around the facility to be inspected. The predicted signal represents the signal that determines whether the facility to be inspected and its surrounding facilities may be in the same frame.

[0069] The method for generating the prediction signal is as follows: First, the historical shooting data of the facility to be inspected is obtained, and then the historical shooting data is filtered to see if there are other inspection facilities. If there are, a prediction signal that there is a possibility of being in the same frame is generated. If there are no other inspection facilities in the same frame in the historical shooting data, then a simulated shooting angle is selected based on the facility to be inspected to obtain simulated photos. Then, based on the process of filtering the simulated photos, it is determined whether there are any inspection facilities in the same frame. If there are, a prediction signal that there is a possibility of being in the same frame is generated. If there are no, a prediction signal that there is no possibility of being in the same frame is generated.

[0070] It should be noted that the simulated photos are obtained by modeling the facility to be inspected and the facility itself according to an actual model, and by selecting the shooting angle. The modeling process uses existing technology, which will not be elaborated upon here.

[0071] Changing the marker indicates that the facility to be inspected is being inspected. By changing the shooting angle, the resulting inspection photo can include multiple facilities to be inspected. Therefore, when inspecting the area to be tested in the future, the inspection efficiency can be improved by changing the shooting angle.

[0072] Reference Figure 4 In one embodiment, after changing the markings on the inspection photos based on the shooting angle, the following steps are also included: S600 acquires the corresponding set of inspection photos based on the area to be tested and determines whether there are replacement marks in the inspection photos.

[0073] If the inspection photos have replacement marks, the S700 will obtain the facilities to be collected based on the area to be tested, and obtain the corresponding standard shooting points from the historical database based on the facilities to be collected.

[0074] The S800 divides the facilities to be captured based on standard shooting points to obtain a set of shooting areas.

[0075] The S900 performs inspection operations based on a set of shooting areas to obtain target data corresponding to the set of shooting areas.

[0076] The "facilities to be sampled" refers to those facilities in the area to be tested that require a change in shooting angle. Standard shooting points represent the shooting position and angle when shooting each facility individually. The shooting area set includes several groups of shooting areas, divided based on the standard shooting points. This division can be achieved by having shooting locations within the same preset area, thus obtaining the shooting area set. The selection of preset areas can be set based on the completion of a single inspection operation; the specific values ​​for the preset areas will not be elaborated upon here.

[0077] Reference Figure 5 In one embodiment, the data acquisition facility is divided based on standard shooting locations to obtain a set of shooting areas, including the following steps: S810 selects several key facilities based on the facilities to be collected, and these key facilities serve as the main inspection facilities within the shooting area.

[0078] S820 uses key facilities and corresponding standard shooting locations as basic values, and the minimum number of elements corresponding to the shooting area set as a constraint.

[0079] The S830 adaptively adjusts the shooting angle of key facilities based on basic values ​​and constraints to obtain the shooting area and corresponding shooting angle.

[0080] In this context, "elements" refers to the photographs taken during the inspection, while "key facilities" represents the facilities used for the primary inspection tasks. Key facilities can be those that are frequently damaged, those that have been in use for a long time, or those located in important areas. Key locations can be intersections, corners, etc. Using key facilities and standard shooting points as basic values, and the elements within the shooting area set as constraints, simulation modeling is performed to obtain the shooting angles corresponding to the shooting areas.

[0081] The specific operation involves, based on the above understanding, first defining the coverage relationship, then constructing the coverage set, solving the coverage problem, and adaptively adjusting. First, several shooting points are randomly generated. For each shooting point, the areas that can be covered under different shooting angles are calculated. This calculation uses geometric methods, taking into account factors such as the camera's field of view (FOV) and occlusion. Next, for each shooting point and possible shooting angles, a "coverage set" is generated, representing the list of areas that the point can cover at that angle. Since the angle can be continuously adjusted, discretization may be necessary, for example, calculating every certain angle. Then, all coverage sets are used as input, and a set-based coverage algorithm is used to select the minimum set—the minimum combination of shooting points and angles—to cover all areas. The set-based coverage problem is NP-hard, so approximation algorithms such as greedy algorithms can be used. Finally, after the initial selection, the shooting angles can be further fine-tuned to optimize coverage or satisfy other constraints, such as minimizing overlap and improving shooting quality.

[0082] The specific solution steps include selecting key facilities, standard shooting locations, setting shooting areas, and establishing coverage relationships. For example, key facilities include facility A, facility B, and facility C. Standard shooting locations include facilities A, B, and C. Facility A includes points A1 and A2, with A1 having a default angle α1 and A2 a default angle α2. Facility B includes point B1, with B1 having a default angle β1. Facility C includes points C1 and C2, with C1 having a default angle γ1 and C2 a default angle γ2. Shooting areas include area 1, area 2, area 3, and area 4. The coverage relationships are as follows: point A1 at angle α1 covers area 1 and area 2; point A1 at angle α1 + Δα covers area 1, area 2, and area 3; point B1 at angle β1 covers area 3; and point C1 at angle γ1 covers area 4.

[0083] For example, we can list all possible combinations of shooting points and angles and the areas they cover, constructing a set coverage problem: the entire set is {region 1, region 2, region 3, region 4}. The subset corresponding to point A1 (angle α1) is set S1, which is {region 1, region 2}. The subset corresponding to point A1 (angle α1 + Δα) is set S2, which is {region 1, region 2, region 3}. The subset corresponding to point B1 (angle β1) is set S3, which is {region 3}. The subset corresponding to point C1 (angle γ1) is set S4, which is {region 4}. We select the subset with the fewest elements to cover the entire set; for example, we can choose S2 and S4 to cover all regions ({region 1, region 2, region 3} + {region 4}).

[0084] In one embodiment, an inspection operation is performed based on a set of shooting areas to obtain target data corresponding to the set of shooting areas, including the following steps: S910 performs fault prediction based on historical fault data corresponding to the area under test to generate an estimated fault time, and generates an inspection frequency corresponding to the set of shooting areas based on the estimated fault time.

[0085] The S920 generates shooting routes based on the inspection frequency and shooting angle corresponding to the set of shooting areas.

[0086] The S930 sends the shooting route to the drone, and the drone's inspection data is used to obtain the target data corresponding to the shooting area set.

[0087] The estimated failure time is obtained by predicting failures based on historical failure data, while the inspection frequency is the interval between required inspections. The drone takes pictures of the area under test along its shooting route, resulting in a minimal set of inspection photos, reducing the amount of image data processing and improving the efficiency of utilizing the acquired image data.

[0088] It's important to note that different inspection facilities have varying estimated downtimes. Therefore, the inspection frequency can be generated based on the shortest estimated downtime. However, if there are too many vehicles on the road during low-altitude drone inspections, it will interfere with the drone's inspection effectiveness. Therefore, when considering inspection frequency, it's advisable to update the frequency based on road usage conditions.

[0089] Specifically, to update the inspection frequency based on road usage status, first, traffic flow in the area to be tested is estimated based on historical traffic flow data to obtain the minimum traffic flow time. Then, it is determined whether the current traffic flow time is suitable for low-altitude drone flight. If suitable, and provided it meets the estimated failure time of each inspection facility, the drone is controlled to patrol the area to be tested based on the shooting route to obtain target data. If the current traffic flow time is not suitable for low-altitude drone flight, manual facility patrol can be initiated based on the shooting route, provided it meets the estimated failure time of each inspection facility.

[0090] This application also discloses a drone-based video inspection system.

[0091] Reference Figure 6 The UAV-based video inspection system includes a data acquisition module, a data judgment module, and a data processing module. The data acquisition module acquires target data for the area to be inspected and generates several sets of inspection photos based on this data. Each photo set includes at least one set of photos and corresponding inspection tags, including usage tags for the photos. The data judgment module acquires the photos sequentially and determines whether a photo is intended for reuse based on the usage tags. If a photo is intended for reuse, the data processing module acquires facilities to be marked based on the photos. It then performs target detection on the facilities to be marked using target detection technology to obtain detection results and determines whether the facilities meet the inspection requirements. If the detection results indicate that the facilities meet the inspection requirements, the data processing module re-marks the facilities and updates the usage tags based on the re-marks to obtain an updated set of inspection photos.

[0092] The other functions performed in the data acquisition module, data judgment module, and data processing module, as well as the technical details of each function, are the same as or similar to the corresponding features in the UAV-based video inspection method described above, so they will not be repeated here.

[0093] In one embodiment, the system further includes a maintenance storage module for storing inspection photos of multiple traffic facilities, with each inspection photo corresponding to an inspection time. The system generates inspection signals based on the inspection photos and generates work schedules based on the inspection signals.

[0094] The implementation principle is as follows: Acquire target data for the area to be tested, and generate several sets of inspection photos based on the target data. Each set of inspection photos includes at least one set of inspection photos and corresponding inspection marks. The inspection marks include usage marks corresponding to the inspection photos.

[0095] The inspection photos are acquired sequentially, and the use of tags determines whether the inspection photos are to be reused.

[0096] If the inspection photos are to be reused, then the facilities to be marked are obtained based on the inspection photos.

[0097] The target detection technology is used to detect the facilities to be marked in order to obtain the detection results, and to determine whether the facilities to be marked meet the inspection requirements based on the detection results.

[0098] If the facility to be marked is determined to meet the inspection requirements, the facility to be marked is marked again, and the marking is updated based on the repeated marking to obtain an updated set of inspection photos.

[0099] If it is determined that the usage mark corresponding to the inspection photo does not belong to the category of photos to be reused, then the inspection photo is determined to be to be used alone based on the usage mark.

[0100] If the inspection photos are intended for separate use, then facilities to be inspected will be generated based on the inspection photos.

[0101] The facilities to be inspected are filtered among the facilities to be marked in order to obtain the screening results and determine whether there are duplicate markings in the screening results.

[0102] If duplicate markers are found in the screening results, a rejection signal is generated, and the inspection photos are rejected based on the rejection signal to update the target data.

[0103] It should be understood that although the steps in the flowcharts in the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders.

[0104] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for video inspection based on unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Acquire target data for the area to be tested, and generate several sets of inspection photos based on the target data. Each set of inspection photos includes at least one set of inspection photos and corresponding inspection markers. The inspection markers include usage markers corresponding to the inspection photos, and the usage markers represent the number of facilities to be inspected that meet the photo clarity requirements in the inspection photos. The method for acquiring the usage markers includes the following steps: The inspection photos are used to perform target detection based on target detection technology to obtain several groups of facilities to be inspected. The corresponding confidence level is obtained based on the facility to be tested, and a usage tag is obtained based on the confidence level and a preset confidence value; When the confidence level of the facility to be inspected is greater than the preset confidence value, the usage mark corresponding to the inspection photo is incremented by 1; The inspection photos are acquired sequentially, and it is determined whether the inspection photos are to be reused based on the usage marker. The used marker is compared with the preset marker value, and it is determined whether the used marker exceeds the preset marker value. If the used marker exceeds the preset marker value, the inspection photo is determined to be reusable. If the used marker does not exceed the preset marker value, the inspection photo is determined not to be reusable. If the inspection photos are to be reused, then the facilities to be marked are obtained based on the inspection photos; The facility to be marked is detected using target detection technology to obtain the detection results, and the facility to be marked is judged to meet the inspection requirements based on the detection results. If it is determined that the facility to be marked meets the inspection requirements, the facility to be marked is marked again, and the usage mark is updated based on the repeated marking to obtain an updated set of inspection photos; Among them, the process of marking the facility to be marked repeatedly refers to judging the inspection photos to determine whether there are duplicate marks in the inspection photos. If there are duplicate marks in the inspection photos, the facility to be marked will be added to the duplicate marks in the inspection photos. If there are no duplicate marks in the inspection photos, the inspection photos need to be marked repeatedly and the facility to be marked will be added to the duplicate marks in the inspection photos. To update the usage tags based on duplicate tags and obtain an updated set of inspection photos, the following steps are included: Unprocessed inspection photos are obtained from the inspection photo set, and facilities to be marked are obtained based on repeated markings; The facilities to be marked are filtered through the unprocessed inspection photos to determine whether the corresponding facilities to be marked are found. If they are found, the usage mark of the selected inspection photos is decremented by 1. If they are not found, no corresponding processing is required for the unprocessed inspection photos.

2. The UAV-based video inspection method according to claim 1, characterized in that, After determining whether the inspection photo belongs to the category of photos to be reused based on the usage marker, the following steps are also included: If it is determined that the usage mark corresponding to the inspection photo does not belong to the category of photos to be reused, then it is determined whether the inspection photo belongs to the category of photos to be used alone based on the usage mark. If the inspection photos are intended for individual use, then facilities to be inspected are generated based on the inspection photos; The facilities to be inspected are screened among the facilities to be marked to obtain the screening results, and it is determined whether there are duplicate markings in the screening results; If the screening results contain duplicate markers, a rejection signal is generated, and the inspection photos are rejected based on the rejection signal to update the target data.

3. The UAV-based video inspection method according to claim 2, characterized in that, After generating the facilities to be inspected based on the inspection photos, the following steps are also included: Based on the facility to be inspected, obtain the corresponding candidate facility in the target data, generate a prediction signal based on the candidate facility, and determine whether the candidate facility and the facility to be inspected are in the same frame based on the prediction signal; If the candidate facility and the facility to be inspected can be in the same frame, a shooting angle is generated based on the candidate facility and the facility to be inspected, and the inspection photo is marked with a different label based on the shooting angle.

4. The UAV-based video inspection method according to claim 3, characterized in that, After changing the markings on the inspection photos based on the shooting angle, the following steps are also included: Based on the area to be tested, obtain the corresponding set of inspection photos, and determine whether there are replacement marks in the inspection photos; If the inspection photos have replacement marks, the facilities to be collected are obtained based on the area to be tested, and the corresponding standard shooting points are obtained from the historical database based on the facilities to be collected. The facilities to be collected are divided based on the standard shooting points to obtain a set of shooting areas, which includes several groups of shooting areas. An inspection operation is performed based on the set of shooting areas to obtain the target data corresponding to the set of shooting areas.

5. The UAV-based video inspection method according to claim 4, characterized in that, The facilities to be captured are divided based on the standard shooting locations to obtain a set of shooting areas, including the following steps: Based on the facilities to be collected, several groups of key facilities are selected, and these key facilities serve as the main inspection facilities within the shooting area. Based on the key facilities and corresponding standard shooting locations as the basic values, and based on the minimum number of elements corresponding to the shooting area set as the constraint condition; Based on the aforementioned basic values ​​and constraints, the shooting angles corresponding to the key facilities are adaptively adjusted to obtain the shooting area and the corresponding shooting angles.

6. The UAV-based video inspection method according to claim 4, characterized in that, The inspection operation is performed based on the set of shooting areas to obtain the target data corresponding to the set of shooting areas, including the following steps: Fault prediction is performed based on the historical fault data corresponding to the area to be tested to generate an estimated fault time, and the inspection frequency corresponding to the set of shooting areas is generated based on the estimated fault time. A shooting route is generated based on the inspection frequency and shooting angle corresponding to the set of shooting areas. The shooting route is sent to the drone, and the target data corresponding to the shooting area set is obtained based on the drone's inspection.

7. A drone-based video inspection system, characterized in that, The method for unmanned aerial vehicle (UAV) video inspection according to any one of claims 1-6 includes: The data acquisition module is used to acquire target data of the area to be tested and generate several sets of inspection photos based on the target data. The set of inspection photos includes at least one set of inspection photos and corresponding inspection marks. The inspection marks include usage marks corresponding to the inspection photos. The data judgment module is used to sequentially acquire the inspection photos and determine whether the inspection photos belong to those to be reused based on the usage marker. The data processing module, if the inspection photo is to be reused, is used to obtain the facility to be marked based on the inspection photo; The facility to be marked is detected using target detection technology to obtain the detection results, and the facility to be marked is judged to meet the inspection requirements based on the detection results. If the detection result determines that the facility to be marked meets the inspection requirements, the data processing module will repeatedly mark the facility to be marked and update the usage mark based on the repeated mark to obtain an updated set of inspection photos.

8. The UAV-based video inspection system according to claim 7, characterized in that, It also includes a maintenance storage module, which is used to store inspection photos of multiple traffic facilities, with each inspection photo corresponding to an inspection time, and to generate inspection signals based on the inspection photos and to generate work schedules based on the inspection signals.

Images