Intelligent unmanned aerial vehicle inspection system and method for street lamps based on multi-source video fusion

Abstract

The application discloses an unmanned aerial vehicle inspection system and method for intelligent street lamps based on multi-source video fusion, relates to the technical field of unmanned aerial vehicle inspection, and comprises an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module and an inspection task dynamic evaluation module; the inspection data acquisition module, the inspection image analysis module, the inspection mode switching module and the inspection task dynamic evaluation module are connected in communication with each other; wherein the inspection data acquisition module is used for flying inspection of street lamps in a target inspection area one by one by means of an unmanned aerial vehicle provided with a visible light camera, and setting image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used for processing and analyzing collected image data; and the application solves the problem that the existing multi-source image fusion processing mode greatly increases the inspection time and data processing burden.

Description

Technical Field

[0001] This invention relates to the field of drone inspection technology, specifically to an intelligent street light drone inspection system and method based on multi-source video fusion. Background Technology

[0002] With the continuous advancement of smart city construction, urban lighting management urgently needs to transform towards digitalization and intelligence. As an important component of urban basic lighting infrastructure, streetlights directly affect traffic safety and urban energy consumption management. To improve the efficiency of streetlight inspections and the timeliness of fault detection, more and more cities are beginning to try using drones to replace traditional manual methods for aerial inspections of streetlight operation.

[0003] In existing technologies, drone inspection systems mostly adopt a "multi-source video fusion" approach. This involves integrating multiple image acquisition modules, such as visible light cameras, infrared thermal imagers, and depth cameras, onto the drone. Through multimodal image fusion analysis, the accuracy of image recognition and fault detection is enhanced in complex environments such as nighttime and strong light interference. This type of solution has a significant advantage in accuracy and is particularly suitable for scenarios where high precision in inspection results is required.

[0004] However, practical application has revealed key issues. While employing full multi-source image fusion processing for every streetlight enhances recognition capabilities, it significantly increases inspection time and data processing burden. This is particularly problematic in high-density lighting areas (such as urban main roads and park roads), where drones need to hover over each streetlight for extended periods to collect and process multimodal images. This drastically reduces the number of streetlights a single drone can inspect with limited battery power, resulting in overall decreased inspection efficiency. Furthermore, in multi-drone collaborative operation modes, the uneven distribution of potentially complex environments encountered by drones leads to much higher-than-expected power consumption. Relying solely on statically preset task allocations and streetlight quantity targets often results in some drones running out of power before completing their inspections, or setting excessively low task targets to avoid power depletion, leading to low drone utilization and significant resource waste. Therefore, designing a practical, efficient, and intelligent intelligent streetlight drone inspection system and method based on multi-source video fusion is essential. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent street light inspection system and method based on multi-source video fusion, so as to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: an intelligent street light UAV inspection system and method based on multi-source video fusion, comprising an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module, and an inspection task dynamic evaluation module; the inspection data acquisition module, inspection image analysis module, inspection mode switching module, and inspection task dynamic evaluation module are interconnected; wherein, the inspection data acquisition module is used to perform flight inspections of street lights in the target inspection area one by one using a UAV equipped with a visible light camera, and sets image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used to process and analyze the acquired image data; the inspection mode switching module is used to switch between a single video source recognition mode and a multi-source video fusion mode according to the inspection image analysis results; the inspection task dynamic evaluation module is used to dynamically adjust the actual total inspection task of the UAV based on the current task completion rate, remaining battery power, and the estimated number of street lights in the current path that need to enter the multi-source fusion mode.

[0007] According to the above technical solution, the inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographical coordinates of the street light, the preset inspection path and the image calibration array template. The real-time image acquisition module acquires visible light images when the UAV hovers over the target street light and associates them with the current street light location information.

[0008] According to the above technical solution, the inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module, and a multi-level feature matching module. The image processing module is used to perform grayscale processing on the visible light image. The image fitting module is used to extract the image contour and fit the boundary features. The multi-level feature generation module is used to extract multi-level feature information of the image in sequence to provide a comprehensive data foundation for accurate matching. The multi-level feature matching module performs accurate matching operations in stages based on the generated feature information.

[0009] According to the above technical solution, the inspection task dynamic evaluation module includes an inspection path division module, an environmental information collection module, a drone data acquisition module, and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of streetlights and related data in each stage. The environmental information collection module is suitable for collecting environmental information related to wind speed and wind direction encountered by the drone during the inspection process. The drone data acquisition module is used to obtain the performance parameters and remaining power of the drone. The energy consumption prediction module is used to calculate the total number of streetlights that the drone can complete based on the collected data and the performance parameters of the drone, and dynamically adjust the total amount of actual inspection tasks.

[0010] According to the above technical solution, the energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of streetlights that the UAV can complete, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction result.

[0011] A method for intelligent street light inspection using drones based on multi-source video fusion includes the following steps:

[0012] Step S1: Use a drone equipped with a visible light camera to conduct aerial inspections of the streetlights in the target inspection area one by one. Set at least one fixed image calibration array point at each streetlight to match the accuracy of the inspection image acquisition.

[0013] Step S2: When the drone flies to the target street light, it acquires the image data of the target street light through the real-time image acquisition module, and performs image alignment matching based on the image calibration array points, including matching whether the number, spatial position, height, and direction of the calibration points captured in the image are consistent with the set image feature template.

[0014] Step S3: If all image matching signals are successful, it indicates that the currently acquired image meets the requirements of clarity and orientation integrity. Then, the intelligent recognition algorithm based on a single video source is executed to identify and determine the status of the target street light.

[0015] Step S4: If an image matching failure signal occurs, the system will automatically switch to multi-source video fusion mode, triggering the use of signals from visible light camera, infrared camera, and depth camera, and performing data fusion processing.

[0016] Step S5: After binding the recognition results with the current street light geographic information, upload them to the ground control center to realize remote street light status management;

[0017] Step S6: During the above inspection process, monitor the changes in the drone's battery level in real time, and dynamically adjust the total amount of actual inspection tasks of the drone based on the current task completion rate, remaining battery level, and the estimated number of streetlights that need to enter the multi-source fusion mode in the current path. Reallocate the remaining uninspected streetlight targets to maximize the energy utilization rate and inspection completion rate of the entire system.

[0018] According to the above technical solution, the specific steps of image alignment matching based on the image calibration array points in step S2 include:

[0019] Step S21: Perform grayscale processing on the visible light image currently acquired by the UAV;

[0020] Step S22: Fit the boundary contour using the image fitting module;

[0021] Step S23: The multi-level feature generation module and the multi-level feature matching module are started together, and feature extraction and feature matching are performed sequentially from the contour line angle feature, contour line length ratio feature, and contour line density ratio feature, respectively. That is, the feature extraction task of the next instruction will only be performed after the previous feature is successfully matched.

[0022] Step S24: Once the "contour angle feature, contour length ratio feature, and contour density ratio feature" in step S23 are all matched, the single image calibration point is output as successfully matched.

[0023] Step S25: Repeat steps S23-S24. After all the single image calibration points have been successfully matched and output, the multi-level feature generation module will perform the overall feature extraction step again, extracting the number of image calibration points, spatial location features, and image array point size comparison features in sequence. That is, the feature extraction task of the next instruction will only be performed after the previous feature is successfully matched.

[0024] Step S26: When the number of image calibration points extracted matches the preset image calibration array points of the inspection target, spatial position feature extraction and matching are performed. During this process, by comparing information and based on the inspection scheme database, the height deviation of the inspection position and the image acquisition angle deviation under the reverse reasoning and correction are fed back to the UAV inspection route control signal to adjust the inspection route. The adjusted spatial position features are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output. The inspection image corresponding to the matching success signal is then overlapped with the image calibration array points in the inspection scheme database. The image recognition area of ​​the two calibration points is compared. If the average deviation of the area of ​​a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

[0025] According to the above technical solution, the specific method for dynamically adjusting the total number of actual inspection tasks of the UAV in step S6 includes:

[0026] Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, determine the total number of streetlights to be inspected corresponding to that path. The path is divided into two phases, with the first phase comprising the first third of the route. The number of streetlights in the first phase is denoted as [missing information]. The second stage is the last two-thirds of the path, and the number of streetlights in the second stage is denoted as... During the first phase of execution, information was collected and recorded from the following data sources for each street light: an indication of whether multi-source fusion was triggered for each street light. If the i-th lamp needs to enter multi-source fusion, then ,otherwise Several wind speed sampling values ​​and wind speed sampling sequences and relative wind direction sampling value ; and location information used to estimate the average spacing between adjacent lamps. ;

[0027] Step S62: Calculate the proportion of multi-source fusion observations based on the first-stage observation data. And calculate the average ambient wind speed. With wind speed variance ;

[0028] Step S63: Obtain performance parameters for the UAV performing the current task, including cruise power. hovering power Average cruising speed Reference hovering time Additional hovering time caused by multi-source fusion Additional energy for single-cycle fusion computing and transmission and the system's reserved safety energy Simultaneously, the proportion of observations through multi-source fusion was... The first-stage estimate of the multi-source proportion is obtained by weighting and smoothing with historical priors. ;

[0029] Step S64: Obtain the current available remaining battery power of the drone. Calculate the total number of streetlights that the drone can complete this task. And output it as a preliminary estimate;

[0030] Step S65: When the output estimated value Then the remaining number of tasks to be assigned And key predicted parameter values ​​are transmitted back to the ground control center;

[0031] Step S66: The ground control center triggers manual intervention to supplement resources, and at the same time uses the significant deviation information observed in the first stage to update the model parameters to improve the accuracy of subsequent inspection route prediction.

[0032] According to the above technical solution, in step S64, the total number of streetlights that the drone can complete this time is calculated. The calculation formula is:

[0033] ;

[0034] in, This indicates the drone's current remaining battery power. Reserve safety energy for the system; Cruise power; This represents the average distance between two adjacent lights. This refers to the average cruising speed. Environmental wind correction factor; For drone hovering power; The baseline hovering time of the drone at the streetlight; This is an estimate of the multi-source fusion trigger probability. This refers to hovering power; The additional hovering time caused by multi-source fusion; Additional energy for single-cycle fusion computing and transmission;

[0035] In the formula, the denominator is the estimated average energy consumption of each street lamp in the second stage;

[0036] in, ;

[0037] In this formula, This represents the average wind speed obtained from the first phase of sampling. The variance of wind speed samples; The average cosine value obtained from the first stage of sampling. ,in The angle between the wind direction and the flight direction; , These are control coefficients, all of which are constants greater than 0; Used only when the average is headwind The impact of energy consumption is amplified when the wind is strong, but the denominator is not reduced when the wind is tailwind to avoid overly optimistic results.

[0038] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention utilizes a drone equipped with a visible light camera, combined with image calibration array points to improve the accuracy of image acquisition, and ensures the reliability of street light status recognition through multi-level image feature extraction and matching. Addressing potential issues such as light interference and occlusion in the inspection environment, the system can intelligently switch between single-source video recognition and multi-source video fusion modes, significantly improving inspection recognition accuracy while ensuring inspection efficiency. Furthermore, based on real-time collected environmental wind speed, wind direction, and drone performance parameters, the system dynamically assesses and predicts the energy consumption and completion rate of inspection tasks, rationally adjusting the remaining workload to maximize energy utilization and inspection efficiency. Through preliminary path division and data collection, the system scientifically predicts the workload of subsequent inspection stages, ensuring dynamic scheduling and accurate completion of inspection tasks, thereby improving the intelligence level and practical value of drone inspections, significantly reducing inspection costs, and enhancing the safety and reliability of street light maintenance. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0040] Figure 1This is a schematic diagram of the system module composition of the present invention;

[0041] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 This invention provides a technical solution: an intelligent street light UAV inspection system and method based on multi-source video fusion, including an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module, and an inspection task dynamic evaluation module; the inspection data acquisition module, the inspection image analysis module, the inspection mode switching module, and the inspection task dynamic evaluation module are interconnected; wherein, the inspection data acquisition module is used to conduct flight inspections of street lights in the target inspection area one by one using a UAV equipped with a visible light camera, and sets image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used to process and analyze the acquired image data; the inspection mode switching module is used to switch between single video source recognition mode and multi-source video fusion mode according to the inspection image analysis results; the inspection task dynamic evaluation module is used to dynamically adjust the actual total inspection task of the UAV based on the current task completion rate, remaining power, and the estimated number of street lights that need to enter the multi-source fusion mode in the current path; by using a UAV equipped with a visible light camera, combined with image calibration array points, the accuracy of image acquisition is improved, and the reliability of street light status recognition is ensured through multi-level image feature extraction and matching. To address potential issues such as lighting interference and occlusion in the inspection environment, the system can intelligently switch between single-source video recognition and multi-source video fusion modes, significantly improving the robustness and recognition accuracy of the inspection. Furthermore, based on real-time collected environmental wind speed, wind direction, and drone performance parameters, the system dynamically assesses and predicts the energy consumption and completion rate of inspection tasks, rationally adjusting the remaining workload to maximize energy utilization and inspection efficiency. Through preliminary path planning and data collection, the system scientifically predicts the workload in subsequent inspection phases, ensuring dynamic scheduling and accurate completion of inspection tasks. This enhances the intelligence and practical value of drone inspections, significantly reduces inspection costs, and improves the safety and reliability of street light maintenance.

[0044] The inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographical coordinates of streetlights, preset inspection paths, and image calibration array templates. The real-time image acquisition module acquires visible light images when the drone hovers over the target streetlight and associates them with the current streetlight location information.

[0045] The inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module, and a multi-level feature matching module. The image processing module is used to perform grayscale processing on visible light images. The image fitting module is used to extract image contours and fit boundary features. The multi-level feature generation module is used to extract multi-level feature information of the image in sequence, providing a comprehensive data foundation for accurate matching. The multi-level feature matching module performs accurate matching operations in stages based on the generated feature information.

[0046] The dynamic evaluation module for inspection tasks includes an inspection path division module, an environmental information collection module, a drone data acquisition module, and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of streetlights and related data in each stage. The environmental information collection module is used to collect environmental information related to wind speed and wind direction encountered by the drone during the inspection process. The drone data acquisition module is used to obtain the drone's performance parameters and remaining power. The energy consumption prediction module is used to calculate the total number of streetlights that the drone can complete based on the collected data and the drone's performance parameters, and dynamically adjust the total amount of actual inspection tasks.

[0047] The energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of streetlights that the drone can complete, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction results.

[0048] A method for intelligent street light inspection using drones based on multi-source video fusion includes the following steps:

[0049] Step S1: Use a drone equipped with a visible light camera to conduct aerial inspections of the streetlights in the target inspection area one by one. Set at least one fixed image calibration array point at each streetlight to match the accuracy of the inspection image acquisition.

[0050] Step S2: When the drone flies to the target street light, it acquires the image data of the target street light through the real-time image acquisition module, and performs image alignment matching based on the image calibration array points, including whether the number, spatial position, height, and direction of the calibration points captured in the matching image are consistent with the set image feature template.

[0051] Step S3: If all image matching signals are successful, it indicates that the currently acquired image meets the requirements of clarity and orientation integrity. Then, the intelligent recognition algorithm based on a single video source is executed to identify and determine the status of the target street light. This simplifies the traditional process of directly identifying and analyzing street lights and then determining whether to switch inspection modes by matching image calibration array points. Only a small amount of computing power is needed to quickly analyze relatively simple image calibration array points and determine the appropriate inspection mode for the target street light. This greatly reduces unnecessary waiting caused by preliminary calculations during the inspection process and improves inspection efficiency.

[0052] Step S4: If an image matching failure signal appears, it means that the inspection process may be affected by external interference (including but not limited to strong light from vehicle headlights at night, rainy or foggy weather, obstructions, etc.) causing the target street light to be unrecognizable in the current view. Then, the system will automatically switch to multi-source video fusion mode, trigger the call of visible light camera, infrared and depth camera signals, and perform data fusion processing to enhance the image information dimension and ensure the reliability of the inspection image recognition.

[0053] Step S5: After binding the recognition results with the current street light geographic information, upload them to the ground control center to realize remote street light status management;

[0054] Step S6: During the above inspection process, monitor the changes in the drone's battery level in real time, and dynamically adjust the total amount of actual inspection tasks of the drone based on the current task completion rate, remaining battery level, and the estimated number of streetlights that need to enter the multi-source fusion mode in the current path. Reallocate the remaining uninspected streetlight targets to maximize the energy utilization rate and inspection completion rate of the entire system.

[0055] The specific steps of image alignment matching based on image calibration array points in step S2 include:

[0056] Step S21: Perform grayscale processing on the visible light image currently acquired by the UAV;

[0057] Step S22: Fit the boundary contour using the image fitting module;

[0058] Step S23: The multi-level feature generation module and the multi-level feature matching module are started together. They extract and match features sequentially from the contour line angle feature, the contour line length ratio feature, and the contour line density ratio feature. That is, the feature extraction task of the next instruction will only be carried out after the previous feature is successfully matched. This avoids continuing the subsequent feature extraction and matching steps when the previous feature is not matched, thus avoiding unnecessary time and computing power wasted. This achieves the function of efficient inspection.

[0059] Step S24: Once the "contour angle feature, contour length ratio feature, and contour density ratio feature" in step S23 are all matched, the single image calibration point is output as successfully matched.

[0060] Step S25: Repeat steps S23-S24. After all the single image calibration points have been successfully matched and output, the multi-level feature generation module will perform the overall feature extraction step again, extracting the number of image calibration points, spatial location features, and image array point size comparison features in sequence. That is, the feature extraction task of the next instruction will only be performed after the previous feature is successfully matched.

[0061] Step S26: When the number of image calibration points extracted matches the preset image calibration array points of the inspection target, spatial position feature extraction and matching are performed. During this process, by comparing information and based on the inspection scheme database, the height deviation of the inspection position and the image acquisition angle deviation under the reverse reasoning and correction are fed back to the UAV inspection route control signal to adjust the inspection route. The adjusted spatial position features are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output. The inspection image corresponding to the matching success signal is then overlapped with the image calibration array points in the inspection scheme database. The image recognition area of ​​the two calibration points is compared. If the average deviation of the area of ​​a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

[0062] The specific methods for dynamically adjusting the total number of actual inspection tasks of the UAV in step S6 include:

[0063] Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, determine the total number of streetlights to be inspected corresponding to that path. The path is divided into two phases, with the first phase comprising the first third of the route. The number of streetlights in the first phase is denoted as [missing information]. The second stage is the last two-thirds of the path, and the number of streetlights in the second stage is denoted as... During the first phase of execution, information was collected and recorded from the following data sources for each street light: an indication of whether multi-source fusion was triggered for each street light. If the i-th lamp needs to enter multi-source fusion, then ,otherwise Several wind speed sampling values ​​and wind speed sampling sequences and relative wind direction sampling value ; and location information used to estimate the average spacing between adjacent lamps. ;

[0064] Step S62: Calculate the proportion of multi-source fusion observations based on the first-stage observation data. And calculate the average ambient wind speed. With wind speed variance ;

[0065] Step S63: Obtain performance parameters for the UAV performing the current task, including cruise power. hovering power Average cruising speed Reference hovering time Additional hovering time caused by multi-source fusion Additional energy for single-cycle fusion computing and transmission and the system's reserved safety energy Simultaneously, the proportion of observations through multi-source fusion was... The first-stage estimate of the multi-source proportion is obtained by weighting and smoothing with historical priors. ;

[0066] In this embodiment of the invention, historical prior refers to the multi-source fusion triggering statistics generated by the UAV in the early inspection missions. Specifically, it includes: the multi-source fusion triggering rate of streetlights recorded in similar inspection areas, similar time periods, and similar environmental conditions; statistical data of each inspection mission summarized by the ground control center, including the ratio of the actual number of multi-source fusion triggers on different path segments to the total number of inspections; and historical correlation data such as weather, illumination, and wind speed.

[0067] To obtain the multi-source fusion trigger probability estimate for the first stage This invention employs a weighted smoothing method based on observed data and historical priors. The calculation formula is as follows:

[0068]

[0069] in, This represents the percentage of multi-source fusion data observed in real-time during the first phase of the current inspection. This refers to the multi-source fusion prior probability of path segments corresponding to similar tasks extracted from the historical inspection database. This is a smoothing coefficient, with a range of values. This can be adaptively adjusted based on system configuration or historical prediction errors; in this step, when the amount of observation data in the first stage is small, Mainly depends on This avoids prediction fluctuations caused by insufficient samples; and as the amount of observation data in the first stage gradually increases... It will be more influenced by real-time observation data, thus having better adaptability in the event of local anomalies.

[0070] Step S64: Obtain the current available remaining battery power of the drone. Calculate the total number of streetlights that the drone can complete this task. And output it as a preliminary estimate;

[0071] Step S65: When the output estimated value Then the remaining number of tasks to be assigned And key predicted parameter values ​​are transmitted back to the ground control center;

[0072] Step S66: The ground control center triggers manual intervention to supplement resources, and at the same time uses the significant deviation information observed in the first stage to update the model parameters to improve the accuracy of subsequent inspection route prediction.

[0073] In step S64, the total number of streetlights that the drone can complete in this operation is calculated. The calculation formula is:

[0074] ;

[0075] in, This indicates the drone's current remaining battery power. Reserve safety energy for the system; Cruise power; This represents the average distance between two adjacent lights. This refers to the average cruising speed. Environmental wind correction factor; For drone hovering power; The baseline hovering time of the drone at the streetlight; This is an estimate of the multi-source fusion trigger probability. This refers to hovering power; The additional hovering time caused by multi-source fusion; Additional energy for single-cycle fusion computing and transmission;

[0076] In the formula, the denominator is the estimated average energy consumption of each street lamp in the second stage;

[0077] in, ;

[0078] In this formula, This represents the average wind speed obtained from the first phase of sampling. The variance of wind speed samples; The average cosine value obtained from the first stage of sampling. ,in The angle between the wind direction and the flight direction; , These are control coefficients, all of which are constants greater than 0; Used only when the average is headwind When the wind is strong, the energy consumption effect is amplified; when the wind is tailwind, the denominator is not reduced to avoid overly optimistic results.

[0079] Through the above steps, the total path of a single drone's preset inspection task can be cleverly divided into two parts: the first third and the last two-thirds, to achieve dynamic and precise adjustment of the total inspection task. The first third of the path mainly undertakes the function of predicting data source acquisition. During the drone's inspection of this segment, the system monitors and collects various key data sources that have a significant impact on power consumption in real time and accurately. These include: First, the proportion of streetlights that need to be inspected in multi-source video fusion mode. This proportion directly reflects the complexity of the inspection; the larger the proportion, the more time, computing power, and final power consumption will increase. Second, the angle between the drone's flight path and the wind direction. When the angle is 180°, it indicates that the drone is flying against the wind, which will significantly increase power consumption. If it is 0°, it indicates a tailwind, with relatively less power consumption. The wind speed also affects power consumption. Third, wind stability, measured by the variance of wind speed fluctuations. Large wind speed fluctuations mean that when the drone is inspecting and aligning the image calibration array at the streetlight, it needs to overcome greater wind changes and consume more power to maintain a stable state.

[0080] Based on the rich and crucial data collected in the first third of the path, the system can scientifically and reasonably predict the increase in power consumption of the drone in the remaining two-thirds of the journey. Combined with the drone's current actual battery level, and using specific analytical and predictive calculation formulas, the system accurately predicts the final number of streetlights that can be inspected. For example, in adverse environments, such as headwinds with high wind speeds, and when the inspection image analysis module determines that a large number of streetlights require multi-source video fusion inspection, it is predicted that only 60% of the original work can be completed in the remaining two-thirds of the path. Therefore, the final number of streetlights that can be inspected is the sum of the number of streetlights calculated with 100% completion in the first third of the path and the number calculated with 60% completion in the remaining two-thirds. This final prediction result will serve as an important basis for dynamically adjusting the total amount of actual inspection tasks for the drone, ensuring that inspection tasks can be adjusted reasonably and efficiently according to actual conditions.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for intelligent street light inspection using unmanned aerial vehicles (UAVs) based on multi-source video fusion, characterized in that: Includes the following steps: Step S1: Use a drone equipped with a visible light camera to conduct aerial inspections of the streetlights in the target inspection area one by one. Set at least one fixed image calibration array point at each streetlight to match the accuracy of the inspection image acquisition. Step S2: When the drone flies to the target street light, it acquires the image data of the target street light through the real-time image acquisition module, and performs image alignment matching based on the image calibration array points, including matching whether the number, spatial position, height, and direction of the calibration points captured in the image are consistent with the set image feature template. Step S3: If all image matching signals are successful, it indicates that the currently acquired image meets the requirements of clarity and orientation integrity. Then, the intelligent recognition algorithm based on a single video source is executed to identify and determine the status of the target street light. Step S4: If an image matching failure signal occurs, the system will automatically switch to multi-source video fusion mode, triggering the use of signals from visible light camera, infrared camera, and depth camera, and performing data fusion processing. Step S5: After binding the recognition results with the current street light geographic information, upload them to the ground control center to realize remote street light status management; Step S6: During the inspection process, monitor the changes in the drone's battery level in real time, and dynamically adjust the total amount of actual inspection tasks of the drone based on the current task completion rate, remaining battery level, and the estimated number of streetlights that need to enter the multi-source fusion mode in the current path. Reallocate the remaining uninspected streetlight targets to maximize the energy utilization rate and inspection completion rate of the entire system. The specific method for dynamically adjusting the total number of actual inspection tasks of the UAV in step S6 includes: Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, determine the total number of streetlights to be inspected corresponding to that path. The path is divided into two phases, with the first phase comprising the first third of the route. The number of streetlights in the first phase is denoted as [missing information]. The second stage is the last two-thirds of the path, and the number of streetlights in the second stage is denoted as... ; During the first phase of execution, information was collected and recorded from the following data sources for each street light: an indication of whether multi-source fusion was triggered for each street light. If the i-th lamp needs to enter multi-source fusion, then ,otherwise Several wind speed sampling values ​​and wind speed sampling sequences and relative wind direction sampling value ; and location information used to estimate the average spacing between adjacent lamps. ; Step S62: Calculate the proportion of multi-source fusion observations based on the first-stage observation data. And calculate the average ambient wind speed. With wind speed variance ; Step S63: Obtain performance parameters for the UAV performing the current task, including cruise power. hovering power Average cruising speed Reference hovering time Additional hovering time caused by multi-source fusion Additional energy for single-cycle fusion computing and transmission and the system's reserved safety energy Simultaneously, the proportion of observations through multi-source fusion was... The first-stage estimate of the multi-source proportion is obtained by weighting and smoothing with historical priors. ; Step S64: Obtain the current available remaining battery power of the drone. Calculate the total number of streetlights that the drone can complete this task. And output it as a preliminary estimate; Step S65: When the output estimated value Then the remaining number of tasks to be assigned And key predicted parameter values ​​are transmitted back to the ground control center; Step S66: The ground control center triggers manual intervention to supplement resources, and at the same time uses the significant deviation information observed in the first stage to update the model parameters to improve the accuracy of subsequent inspection route prediction.

2. The intelligent street light inspection method using unmanned aerial vehicles (UAVs) based on multi-source video fusion as described in claim 1, characterized in that: The specific steps for image alignment matching based on the image calibration array points in step S2 include: Step S21: Perform grayscale processing on the visible light image currently acquired by the UAV; Step S22: Fit the boundary contour using the image fitting module; Step S23: The multi-level feature generation module and the multi-level feature matching module are started together, and feature extraction and feature matching are performed sequentially from the contour line angle feature, contour line length ratio feature, and contour line density ratio feature, respectively. That is, the feature extraction task of the next instruction will only be performed after the previous feature is successfully matched. Step S24: Once the "contour angle feature, contour length ratio feature, and contour density ratio feature" in step S23 are all matched, the single image calibration point is output as successfully matched. Step S25: Repeat steps S23-S24. After all the single image calibration points have been successfully matched and output, the multi-level feature generation module will perform the overall feature extraction step again, extracting the number of image calibration points, spatial location features, and image array point size comparison features in sequence. That is, the feature extraction task of the next instruction will only be performed after the previous feature is successfully matched. Step S26: After the extracted image calibration point quantity feature matches the preset image calibration array points of the inspection target, spatial position feature extraction and matching are performed. During this process, by comparing information and based on the inspection scheme database, the height deviation of the inspection position and the image acquisition angle deviation under the reverse reasoning and correction are fed back to the UAV inspection route control signal to adjust the inspection route. The adjusted spatial position features are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output. The inspection image corresponding to the matching success signal is then overlapped with the image calibration array points in the inspection scheme database. The image recognition area of ​​the two calibration points is compared. If the average deviation of the area of ​​a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

3. The intelligent street light inspection method using unmanned aerial vehicles (UAVs) based on multi-source video fusion according to claim 1, characterized in that: In step S64, the total number of streetlights that the drone can complete in this operation is calculated. The calculation formula is: ； in, This indicates the drone's current remaining battery power. Reserve safety energy for the system; Cruise power; This represents the average distance between two adjacent lights. This refers to the average cruising speed. Environmental wind correction factor; For drone hovering power; The baseline hovering time of the drone at the streetlight; This is an estimate of the multi-source fusion trigger probability. This refers to hovering power; The additional hovering time caused by multi-source fusion; Additional energy for single-cycle fusion computing and transmission; In the formula, the denominator is the estimated average energy consumption of each street lamp in the second stage; in, ; In this formula, This represents the average wind speed obtained from the first phase of sampling. The variance of wind speed samples; The average cosine value obtained from the first stage of sampling. ,in The angle between the wind direction and the flight direction; , These are control coefficients, all of which are constants greater than 0; Used only when the average is headwind The impact of energy consumption is amplified when the wind is strong, but the denominator is not reduced when the wind is tailwind to avoid overly optimistic results.

4. An intelligent street light inspection system based on multi-source video fusion for implementing the method of claim 1, characterized in that, The system includes an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module, and an inspection task dynamic evaluation module. These modules are interconnected. The inspection data acquisition module uses a drone equipped with a visible light camera to conduct aerial inspections of streetlights within a target inspection area, setting image calibration array points to match the accuracy of the acquired images. The inspection image analysis module processes and analyzes the acquired image data. The inspection mode switching module switches between a single video source recognition mode and a multi-source video fusion mode based on the inspection image analysis results. The inspection task dynamic evaluation module dynamically adjusts the total inspection workload of the drone based on the current task completion rate, remaining battery power, and the estimated number of streetlights requiring multi-source fusion mode in the current path.

5. The intelligent street light UAV inspection system based on multi-source video fusion according to claim 4, characterized in that: The inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographical coordinates of the streetlights, preset inspection paths, and image calibration array templates. The real-time image acquisition module acquires visible light images when the drone hovers over the target streetlight and associates them with the current streetlight location information.

6. The intelligent street light UAV inspection system based on multi-source video fusion according to claim 4, characterized in that: The inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module, and a multi-level feature matching module. The image processing module is used to perform grayscale processing on the visible light image. The image fitting module is used to extract the image contour and fit the boundary features. The multi-level feature generation module is used to extract multi-level feature information of the image in sequence to provide a comprehensive data foundation for accurate matching. The multi-level feature matching module performs accurate matching operations in stages based on the generated feature information.

7. The intelligent street light UAV inspection system based on multi-source video fusion according to claim 4, characterized in that: The dynamic evaluation module for inspection tasks includes an inspection path division module, an environmental information collection module, a drone data acquisition module, and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of streetlights and related data in each stage. The environmental information collection module is used to collect environmental information related to wind speed and wind direction encountered by the drone during the inspection process. The drone data acquisition module is used to obtain the performance parameters and remaining power of the drone. The energy consumption prediction module is used to calculate the total number of streetlights that the drone can complete based on the collected data and the performance parameters of the drone, and dynamically adjust the total amount of actual inspection tasks.

8. The intelligent street light UAV inspection system based on multi-source video fusion according to claim 7, characterized in that: The energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of streetlights that the UAV can complete, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction results.

Images