Unmanned aerial vehicle inspection system and method for intelligent street lamp based on multi-source video fusion

Abstract

The invention discloses an unmanned aerial vehicle inspection system and method for an intelligent street lamp based on multi-source video fusion, and relates to the technical field of unmanned aerial vehicle inspection, and the system comprises an inspection data collection module, an inspection image analysis module, an inspection mode switching module and an inspection task dynamic evaluation module. The inspection data acquisition module, the inspection image analysis module, the inspection mode switching module and the inspection task dynamic evaluation module are in communication connection with one another; wherein the inspection data acquisition module is used for performing flight inspection on street lamps in a target inspection area one by one through an unmanned aerial vehicle with a visible light camera, and setting image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used for processing and analyzing the acquired image data; according to the invention, the problem that the inspection time consumption and the data processing burden are greatly increased in the existing multi-source image fusion processing mode is solved.

Description

Technical Field

[0001] The present invention relates to the field of drone inspection technology, and in particular to an intelligent street lamp drone inspection system and method based on multi-source video fusion. Background Art

[0002] With the continuous advancement of smart city construction, urban lighting management urgently needs to transform towards digitalization and intelligence. As a vital component of urban lighting infrastructure, the operating status of streetlights directly impacts traffic safety and urban energy management. To improve the efficiency of streetlight inspections and the timeliness of fault detection, more and more cities are experimenting with drones, replacing traditional manual methods, for aerial inspections of streetlight operations.

[0003] Existing drone inspection systems often use a "multi-source video fusion" approach. This involves integrating multiple image acquisition modules, such as visible light cameras, infrared thermal imagers, and depth cameras, onto drones. Through multimodal image fusion analysis, this approach enhances image recognition accuracy and fault detection in complex environments, such as nighttime, fog and haze, and strong light interference. This approach offers significant advantages in accuracy and is particularly suitable for scenarios requiring high-precision inspection results.

[0004] However, practice has exposed a key issue: while employing full multi-source image fusion processing at every streetlight, while this approach enhances recognition capabilities, it also significantly increases inspection time and data processing burdens. This is especially true in densely lit areas (such as urban arterials and park roads). Drones must hover for extended periods at each streetlight to capture and process multimodal images. This significantly reduces the number of streetlights a single drone can inspect within its limited battery capacity, reducing overall inspection efficiency. Furthermore, when multiple drones operate collaboratively, the uneven distribution of complex environments they may encounter can lead to significantly higher power consumption than anticipated. Relying solely on statically pre-set task allocation and streetlight target counts often results in individual drones running out of battery power without completing their inspection missions, or setting low task targets to mitigate the risk of exhaustion, resulting in low drone utilization and significant resource waste. Therefore, it is imperative to design a practical, efficient, and intelligent drone inspection system and method for streetlights based on multi-source video fusion. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent street lamp drone inspection system and method based on multi-source video fusion to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: an intelligent drone inspection system and method for street lamps based on multi-source video fusion, including an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module and an inspection task dynamic evaluation module; the inspection data acquisition module, the inspection image analysis module, the inspection mode switching module and the inspection task dynamic evaluation module are interconnected for communication; wherein, the inspection data acquisition module is used to perform flight inspections of the street lamps in the target inspection area one by one through a drone equipped with a visible light camera, and set image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used to process and analyze the collected image data; the inspection mode switching module is used to switch between a single video source recognition mode and a multi-source video fusion mode according to the inspection image analysis results; the inspection task dynamic evaluation module is used to dynamically adjust the actual total amount of inspection tasks of the drone according to the current task completion degree, the remaining power and the estimated number of street lamps in the current path that need to enter the multi-source fusion mode.

[0007] According to the above technical solution, the inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographical coordinates of street lights, preset inspection paths and image calibration array templates. The real-time image acquisition module collects visible light images when the drone hovers over the target street light and associates them with the current street light location information.

[0008] According to the above technical solution, the inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module and a multi-level feature matching module. The image processing module is used to grayscale the visible light image, the image fitting module is used to extract the image contour and fit the boundary features, the multi-level feature generation module is used to sequentially extract the multi-level feature information of the image to provide a comprehensive data basis for precise matching, and the multi-level feature matching module performs precise matching operations in stages based on the generated feature information.

[0009] According to the above technical solution, the inspection task dynamic evaluation module includes an inspection path division module, an environmental information collection module, a drone collection and acquisition module and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of street lights and related data in each stage. The environmental information collection module is suitable for collecting environmental information related to wind speed and wind direction during the drone inspection process. The drone collection and acquisition module is used to obtain the performance parameters and remaining power of the drone. The energy consumption prediction module is used to calculate the total number of street lights that the drone can complete based on the collected data and the performance parameters of the drone, and dynamically adjust the total amount of actual inspection tasks.

[0010] According to the above technical solution, the energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of street lights that the drone can complete, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction result.

[0011] An intelligent street lamp drone inspection method based on multi-source video fusion includes the following steps: Step S1: Using a drone equipped with a visible light camera to perform a flight inspection of the streetlights within the target inspection area, one by one, and setting at least one fixed-position image calibration array point at each streetlight to match the accuracy of inspection image acquisition; Step S2: When the drone flies to the target streetlight, the real-time image acquisition module acquires image data of the target streetlight and performs image alignment matching based on the image calibration array points, including whether the number, spatial position, height, and direction of the calibration points captured in the matching image are consistent with the set image feature template; Step S3: If the image matching is successful, indicating that the current captured image meets the clarity and directional integrity requirements, the intelligent recognition algorithm based on a single video source is executed to identify and determine the state of the target street lamp; Step S4: If an image matching failure signal appears, the system automatically switches to multi-source video fusion mode, triggering the call of visible light camera, infrared camera, and depth camera signals, and performs data fusion processing; Step S5: Bind the recognition result with the current street light geographic information and upload it to the ground control center to achieve remote street light status management; Step S6: During the above inspection process, the power changes of the drone are monitored in real time, and according to the current task completion, remaining power and the estimated number of street lights that need to enter the multi-source fusion mode in the current path, the actual total number of inspection tasks of the drone is dynamically adjusted, and the remaining uninspected street light targets are reallocated to maximize the energy utilization and inspection completion of the entire system.

[0012] According to the above technical solution, the specific steps of performing image alignment matching based on the image calibration array points in step S2 include: Step S21: grayscale processing is performed on the visible light image currently collected by the drone; Step S22: fitting the boundary contour through the image fitting module; Step S23: The multi-level feature generation module and the multi-level feature matching module are jointly started to extract and match features from the contour angle feature, the contour length ratio feature, and the contour density ratio feature respectively. That is, the feature extraction task of the next instruction will be carried out only after the previous feature matching is successful. Step S24: When the “contour angle feature, contour length ratio feature, and contour density ratio feature” in step S23 are all matched, a single image calibration point is output as successfully matched; Step S25: Repeat steps S23-S24. After all single image calibration points have successfully matched, the multi-level feature generation module will perform the overall feature extraction step again, sequentially extracting the image calibration point quantity feature, spatial position feature, and image array point size comparison feature. That is, the feature extraction task of the next instruction will be performed only after the previous feature match is successful. Step S26: When the number characteristics of the extracted image calibration points are consistent with the preset image calibration array points of the inspection target, spatial position feature extraction and matching are performed. During this period, by comparing information and based on the inspection plan database, the height deviation of the inspection position under reverse reasoning and correction conditions and the image acquisition angle deviation are fed back to the UAV inspection route control signal to adjust the inspection route, and the adjusted spatial position characteristics are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output, and the inspection image corresponding to the matching success signal is further overlapped with the image calibration array points in the inspection plan database, and the image recognition areas of the calibration points of the two are compared. If the average area deviation of a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

[0013] According to the above technical solution, the specific method of dynamically adjusting the total amount of actual inspection tasks of the drone in step S6 includes: Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, the total number of street lamps to be inspected corresponding to the path is calculated. It is divided into the first and second stages, where the first stage is the first third of the path, and the number of street lights in the first stage is recorded as The second stage is the last two-thirds of the path, and the number of street lights in the second stage is recorded as During the first phase, the following data sources are collected and recorded: an indication of whether each streetlight triggers multi-source fusion; If the i-th lamp needs to enter multi-source fusion, then ,otherwise ; Several wind speed sampling values ​​wind speed sampling sequence and relative wind direction sampling values ; and location information for estimating the average distance between adjacent lamps ; Step S62: Calculate the proportion of multi-source fusion observations based on the first phase observation data , and calculate the mean ambient wind speed and wind speed variance ; Step S63: Obtain the performance parameters of the UAV for the current mission, including cruise power , hovering power , average cruising speed , baseline hovering time , additional hovering time caused by multi-source fusion , single fusion calculation and transmission of additional energy , and the system reserves safety energy ; At the same time, the proportion of multi-source fusion observation The multi-source proportion estimated in the first stage is obtained by weighted smoothing with historical priors ; Step S64: Obtain the current remaining power of the drone , calculate the total number of street lights that the drone can complete this time , and output as an estimated value; Step S65: When the estimated value output , then the remaining number of tasks to be assigned and key prediction parameter values ​​are transmitted back to the ground control center; Step S66: The ground control center triggers manual intervention to supplement resources, and uses the significant deviation information observed in the first phase to update the model parameters to improve the accuracy of subsequent inspection route predictions.

[0014] According to the above technical solution, in step S64, the total number of street lights that the drone can complete is calculated. The calculation formula is: ; in, The current remaining available power of the drone; Reserve safety energy for the system; is the cruising power; The average horizontal distance between two adjacent lights; is the average cruising speed; is the ambient wind correction factor; The hovering power of the drone; is the benchmark hovering time of the UAV at the street light; is the estimated value of multi-source fusion trigger probability; is the hovering power; Additional hovering time caused by multi-source fusion; Calculate and transmit additional energy for a single fusion; Where, the denominator is the estimated average energy consumption of each streetlight in the second phase; in, ; In this formula, is the mean wind speed obtained by sampling in the first stage; is the wind speed sample variance; is the average cosine value obtained by sampling in the first stage, ,in is the angle between wind direction and flight direction; 、 is the control coefficient, which is a constant greater than 0; Used only in average headwinds When the wind is favorable, the denominator is not reduced to avoid over-optimism.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention utilizes a drone equipped with a visible light camera, combined with image calibration array points to improve the accuracy of image acquisition, and ensures the reliability of street lamp status recognition through multi-level image feature extraction and matching. In response to problems such as light interference and occlusion that may occur in the inspection environment, the system can intelligently switch between single video source recognition and multi-source video fusion mode, significantly improving inspection recognition accuracy while also ensuring inspection efficiency. In addition, based on the real-time collection of environmental wind speed, wind direction and drone performance parameters, the energy consumption and completion of inspection tasks are dynamically evaluated and predicted, the remaining task volume is reasonably adjusted, and energy utilization and inspection efficiency are maximized. Through early path division and data collection, the system scientifically predicts the task volume of the subsequent inspection stage, ensuring the dynamic scheduling and precise completion of inspection tasks, thereby improving the intelligence level and practical value of drone inspections, significantly reducing inspection costs, and improving the safety and reliability of street lamp maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a schematic diagram of the system module composition of the present invention; Figure 2 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] See also Figure 1The present invention provides a technical solution: an intelligent street lamp drone inspection system and method based on multi-source video fusion, including an inspection data acquisition module, an inspection image analysis module, an inspection mode switching module and an inspection task dynamic evaluation module; the inspection data acquisition module, the inspection image analysis module, the inspection mode switching module and the inspection task dynamic evaluation module are interconnected and communicated with each other; wherein, the inspection data acquisition module is used to perform flight inspections of the street lamps in the target inspection area one by one through a drone equipped with a visible light camera, and set image calibration array points to match the accuracy of inspection image acquisition; the inspection image analysis module is used to process and analyze the collected image data; the inspection mode switching module is used to switch between a single video source recognition mode and a multi-source video fusion mode according to the inspection image analysis results; the inspection task dynamic evaluation module is used to dynamically adjust the actual total amount of inspection tasks of the drone according to the current task completion degree, the remaining power and the estimated number of street lamps in the current path that need to enter the multi-source fusion mode; the drone equipped with a visible light camera is used in combination with the image calibration array points to improve the accuracy of image acquisition, and the reliability of street lamp status recognition is ensured through multi-level image feature extraction and matching. To address issues such as lighting interference and occlusion that may arise in inspection environments, the system intelligently switches between single-source video recognition and multi-source video fusion modes, significantly improving inspection robustness and recognition accuracy. Furthermore, based on real-time acquisition of ambient wind speed, direction, and drone performance parameters, it dynamically evaluates and predicts the energy consumption and completion rate of inspection tasks, rationally adjusting the remaining workload to maximize energy utilization and inspection efficiency. Through preliminary path division and data collection, the system scientifically predicts the workload of subsequent inspection phases, ensuring the dynamic scheduling and precise completion of inspection tasks. This enhances the intelligence and practical value of drone inspections, significantly reduces inspection costs, and improves the safety and reliability of streetlight maintenance.

[0019] The inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographic coordinates of street lights, preset inspection paths, and image calibration array templates. The real-time image acquisition module collects visible light images when the drone hovers over the target street light and associates them with the current street light location information.

[0020] The inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module and a multi-level feature matching module. The image processing module is used to grayscale the visible light image. The image fitting module is used to extract the image contour and fit the boundary features. The multi-level feature generation module is used to sequentially extract the multi-level feature information of the image to provide a comprehensive data basis for accurate matching. The multi-level feature matching module performs accurate matching operations in stages based on the generated feature information.

[0021] The dynamic evaluation module of the inspection task includes an inspection path division module, an environmental information collection module, a drone collection and acquisition module and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of street lights and related data in each stage. The environmental information collection module is suitable for collecting environmental information related to wind speed and wind direction during the drone inspection process. The drone collection and acquisition module is used to obtain the performance parameters and remaining power of the drone. The energy consumption prediction module is used to calculate the total number of street lights that the drone can complete based on the collected data and the performance parameters of the drone, and dynamically adjust the total amount of actual inspection tasks.

[0022] The energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of street lights that can be completed by the drone, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction result.

[0023] An intelligent street lamp drone inspection method based on multi-source video fusion includes the following steps: Step S1: Using a drone equipped with a visible light camera to perform a flight inspection of the streetlights within the target inspection area, one by one, and setting at least one fixed-position image calibration array point at each streetlight to match the accuracy of inspection image acquisition; Step S2: When the drone flies to the target streetlight, it acquires image data of the target streetlight through the real-time image acquisition module and performs image alignment matching based on the image calibration array points, including whether the number, spatial position, height, and direction of the calibration points captured in the matching image are consistent with the set image feature template; Step S3: If all image matches are successful, indicating that the currently captured image meets the clarity and directional integrity requirements, an intelligent recognition algorithm based on a single video source is executed to identify and determine the state of the target street lamp. This simplifies the traditional process of directly identifying and analyzing the street lamp and then determining whether to switch to the inspection mode by matching the image calibration array points. Only relatively simple image calibration array points can be quickly analyzed with relatively low computing power resources to determine whether they are suitable for the inspection mode of the target street lamp, greatly reducing unnecessary waiting caused by preliminary calculations during the inspection process and improving inspection efficiency. Step S4: If an image matching failure signal appears, indicating that the inspection process may be affected by external interference (including but not limited to strong light from vehicle lights at night, rainy and foggy weather, obstructions, etc.), resulting in the target street lamp being unrecognizable in the current view, the system automatically switches to multi-source video fusion mode, triggering the call of visible light camera, infrared camera, and depth camera signals, and performs data fusion processing to enhance the image information dimension and ensure the recognition reliability of the inspection image; Step S5: Bind the recognition result with the current street light geographic information and upload it to the ground control center to achieve remote street light status management; Step S6: During the above inspection process, the power changes of the drone are monitored in real time, and according to the current task completion, remaining power and the estimated number of street lights that need to enter the multi-source fusion mode in the current path, the actual total number of inspection tasks of the drone is dynamically adjusted, and the remaining uninspected street light targets are reallocated to maximize the energy utilization and inspection completion of the entire system.

[0024] The specific steps of performing image alignment matching based on the image calibration array points in step S2 include: Step S21: grayscale processing is performed on the visible light image currently collected by the drone; Step S22: fitting the boundary contour through the image fitting module; Step S23: The multi-level feature generation module and the multi-level feature matching module are jointly started to extract and match features from the contour angle feature, the contour length ratio feature, and the contour density ratio feature in sequence. That is, the feature extraction task of the next instruction will be carried out only after the previous feature match is successful, thereby avoiding the waste of unnecessary time and computing power by continuing the subsequent feature extraction and matching steps when the previous feature match fails, thereby achieving the effect of efficient inspection; Step S24: When the “contour angle feature, contour length ratio feature, and contour density ratio feature” in step S23 are all matched, a single image calibration point is output as successfully matched; Step S25: Repeat steps S23-S24. After all single image calibration points have successfully matched, the multi-level feature generation module will perform the overall feature extraction step again, sequentially extracting the image calibration point quantity feature, spatial position feature, and image array point size comparison feature. That is, the feature extraction task of the next instruction will be performed only after the previous feature match is successful. Step S26: When the number characteristics of the extracted image calibration points are consistent with the preset image calibration array points of the inspection target, spatial position feature extraction and matching are performed. During this period, by comparing information and based on the inspection plan database, the height deviation of the inspection position under reverse reasoning and correction conditions and the image acquisition angle deviation are fed back to the UAV inspection route control signal to adjust the inspection route, and the adjusted spatial position characteristics are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output, and the inspection image corresponding to the matching success signal is further overlapped with the image calibration array points in the inspection plan database, and the image recognition areas of the calibration points of the two are compared. If the average area deviation of a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

[0025] The specific method for dynamically adjusting the total amount of actual inspection tasks of the drone in step S6 includes: Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, the total number of street lamps to be inspected corresponding to the path is calculated. It is divided into the first and second stages, where the first stage is the first third of the path, and the number of street lights in the first stage is recorded as The second stage is the last two-thirds of the path, and the number of street lights in the second stage is recorded as During the first phase, the following data sources are collected and recorded: an indication of whether each streetlight triggers multi-source fusion; If the i-th lamp needs to enter multi-source fusion, then ,otherwise ; Several wind speed sampling values ​​wind speed sampling sequence and relative wind direction sampling values ; and location information for estimating the average distance between adjacent lamps ; Step S62: Calculate the proportion of multi-source fusion observations based on the first phase observation data , and calculate the mean ambient wind speed and wind speed variance ; Step S63: Obtain the performance parameters of the UAV for the current mission, including cruise power , hovering power , average cruising speed , baseline hovering time , additional hovering time caused by multi-source fusion , single fusion calculation and transmission of additional energy , and the system reserves safety energy ; At the same time, the proportion of multi-source fusion observation The multi-source proportion estimated in the first stage is obtained by weighted smoothing with historical priors ; In an embodiment of the present invention, historical prior refers to the multi-source fusion trigger statistics generated by the drone in the previous inspection mission, which specifically includes: the multi-source fusion trigger rate of street lights recorded in similar inspection areas, similar time periods, and similar environmental conditions; the statistical data of previous inspection missions summarized by the ground control center, including the ratio of the actual number of multi-source fusion triggers on different path segments to the total number of inspections; and historical related data such as weather, light, and wind speed.

[0026] To obtain the first stage multi-source fusion trigger probability estimate , the present invention adopts a weighted smoothing method based on observed data and historical priors. The calculation formula is as follows: ; in, is the proportion of multi-source fusion observed in real time during the first phase of the current inspection. is the multi-source fusion prior probability of the corresponding path segments of similar tasks extracted from the historical inspection database, is the smoothing coefficient, the value range is , can be adjusted adaptively according to the system configuration or based on historical prediction errors; in this step, when the amount of observation data in the first stage is small, Mainly depends on , to avoid the prediction fluctuation caused by insufficient samples; and when the amount of observation data in the first stage gradually increases, It will be more affected by real-time observation data, thus having better adaptability in local abnormal situations.

[0027] Step S64: Obtain the current remaining power of the drone , calculate the total number of street lights that the drone can complete this time , and output as an estimated value; Step S65: When the estimated value output , then the remaining number of tasks to be assigned and key prediction parameter values ​​are transmitted back to the ground control center; Step S66: The ground control center triggers manual intervention to supplement resources, and uses the significant deviation information observed in the first phase to update the model parameters to improve the accuracy of subsequent inspection route predictions.

[0028] In step S64, the total number of street lights that the drone can complete is calculated. The calculation formula is: ; in, The current remaining available power of the drone; Reserve safety energy for the system; is the cruising power; The average horizontal distance between two adjacent lights; is the average cruising speed; is the ambient wind correction factor; The hovering power of the drone; is the benchmark hovering time of the UAV at the street light; is the estimated value of multi-source fusion trigger probability; is the hovering power; Additional hovering time caused by multi-source fusion; Calculate and transmit additional energy for a single fusion; Where, the denominator is the estimated average energy consumption of each streetlight in the second phase; in, ; In this formula, is the mean wind speed obtained by sampling in the first stage; is the wind speed sample variance; is the average cosine value obtained by sampling in the first stage, ,in is the angle between wind direction and flight direction; 、 is the control coefficient, which is a constant greater than 0; Used only in average headwinds When the wind is favorable, the denominator is not reduced to avoid over-optimism.

[0029] Through the above steps, a single drone's pre-set inspection mission can be cleverly divided into the first third and the last two-thirds to achieve dynamic and precise adjustment of the total inspection volume. The first third of the path primarily handles predictive data source acquisition. During this inspection, the system accurately monitors and collects key data sources that significantly impact power consumption in real time. This includes the percentage of streetlights requiring multi-source video fusion mode for inspection. This percentage directly reflects the complexity of the inspection; a higher percentage means a significant increase in inspection time, computing power, and ultimately power consumption. Secondly, the angle between the drone's flight path and the wind direction. A 180° angle indicates a headwind, significantly increasing power consumption. A 0° angle indicates a tailwind, resulting in relatively low power consumption. Wind speed also influences power consumption. Thirdly, wind stability is measured by the variance of wind speed fluctuations. Larger wind speed fluctuations mean the drone must overcome greater wind fluctuations when calibrating the image calibration array at the streetlights, consuming more power to maintain stability.

[0030] Based on this rich and critical data collected during the first third of the route, the system can scientifically and rationally predict the increased power consumption of the drone during the remaining two-thirds of the route. Combined with the drone's current actual power consumption, a specific analytical and predictive calculation formula is used to accurately predict the number of streetlights that can ultimately be inspected. For example, in adverse conditions, such as high headwinds and headwinds, and if the inspection image analysis module determines that a large number of streetlights require multi-source video fusion inspection, the system predicts that only 60% of the remaining two-thirds of the route will actually be completed. The final number of streetlights that can be inspected is the sum of the number of streetlights calculated for 100% completion in the first third plus the number of streetlights calculated for 60% completion in the remaining two-thirds. This final prediction output serves as an important basis for dynamically adjusting the drone's total number of inspection tasks, ensuring that inspections can be rationally and efficiently adjusted based on actual conditions.

[0031] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0032] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0033] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0034] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. An intelligent streetlight drone inspection method based on multi-source video fusion, characterized by: The following steps are involved: Step S1: Using a drone equipped with a visible light camera to perform a flight inspection of the streetlights within the target inspection area, one by one, and setting at least one fixed-position image calibration array point at each streetlight to match the accuracy of inspection image acquisition; Step S2: When the drone flies to the target streetlight, the real-time image acquisition module acquires image data of the target streetlight and performs image alignment matching based on the image calibration array points, including whether the number, spatial position, height, and direction of the calibration points captured in the matching image are consistent with the set image feature template; Step S3: If the image matching is successful, indicating that the current captured image meets the clarity and directional integrity requirements, the intelligent recognition algorithm based on a single video source is executed to identify and determine the state of the target street lamp; Step S4: If an image matching failure signal appears, the system automatically switches to multi-source video fusion mode, triggering the call of visible light camera, infrared camera, and depth camera signals, and performs data fusion processing; Step S5: Bind the recognition result with the current street light geographic information and upload it to the ground control center to achieve remote street light status management; Step S6: During the inspection process, the power changes of the drone are monitored in real time. Based on the current task completion, remaining power, and the estimated number of street lights in the current path that need to enter the multi-source fusion mode, the actual total number of inspection tasks of the drone is dynamically adjusted, and the remaining uninspected street light targets are reallocated to maximize the energy utilization and inspection completion of the entire system.

2. The intelligent streetlight drone inspection method based on multi-source video fusion according to claim 1 is characterized by: The specific steps of performing image alignment matching based on the image calibration array points in step S2 include: Step S21: grayscale processing is performed on the visible light image currently collected by the drone; Step S22: fitting the boundary contour through the image fitting module; Step S23: The multi-level feature generation module and the multi-level feature matching module are jointly started to extract and match features from the contour angle feature, the contour length ratio feature, and the contour density ratio feature respectively. That is, the feature extraction task of the next instruction will be carried out only after the previous feature matching is successful. Step S24: When the “contour angle feature, contour length ratio feature, and contour density ratio feature” in step S23 are all matched, a single image calibration point is output as successfully matched; Step S25: Repeat steps S23-S24. After all single image calibration points have successfully matched, the multi-level feature generation module will perform the overall feature extraction step again, sequentially extracting the image calibration point quantity feature, spatial position feature, and image array point size comparison feature. That is, the feature extraction task of the next instruction will be performed only after the previous feature match is successful. Step S26: When the number characteristics of the extracted image calibration points are consistent with the image calibration array points preset for the inspection target, spatial position feature extraction and matching are performed. During this period, by comparing information and based on the inspection plan database, the height deviation of the inspection position under reverse reasoning and correction conditions and the image acquisition angle deviation are fed back to the UAV inspection route control signal to adjust the inspection route, and the adjusted spatial position characteristics are further extracted. When the matching degree with the image calibration array points exceeds 90%, a spatial position feature matching success signal is output, and the inspection image corresponding to the matching success signal is further overlapped with the image calibration array points in the inspection plan database, and the image recognition areas of the calibration points of the two are compared. If the average area deviation of a single calibration point is less than 10%, the multi-level feature matching module outputs an image array point size comparison matching success signal.

3. The intelligent streetlight drone inspection method based on multi-source video fusion according to claim 1 is characterized by: The specific method of dynamically adjusting the total amount of actual inspection tasks of the drone in step S6 includes: Step S61: Before the inspection task begins, based on the preset inspection path of a single drone, the total number of street lamps to be inspected corresponding to the path is calculated. It is divided into the first and second stages, where the first stage is the first third of the path, and the number of street lights in the first stage is recorded as The second stage is the last two-thirds of the path, and the number of street lights in the second stage is recorded as During the first phase, the following data sources are collected and recorded: an indication of whether each streetlight triggers multi-source fusion; If the i-th lamp needs to enter multi-source fusion, then ,otherwise ; Several wind speed sampling values ​​wind speed sampling sequence and relative wind direction sampling values ; and location information for estimating the average distance between adjacent lamps ; Step S62: Calculate the proportion of multi-source fusion observations based on the first phase observation data , and calculate the mean ambient wind speed and wind speed variance ; Step S63: Obtain the performance parameters of the UAV for the current mission, including cruise power , hovering power , average cruising speed , baseline hovering time , additional hovering time caused by multi-source fusion , single fusion calculation and transmission of additional energy , and the system reserves safety energy ; At the same time, the proportion of multi-source fusion observation The multi-source proportion estimated in the first stage is obtained by weighted smoothing with historical priors ; Step S64: Obtain the current remaining power of the drone , calculate the total number of street lights that the drone can complete this time , and output as an estimated value; Step S65: When the estimated value output , then the remaining number of tasks to be assigned and key prediction parameter values ​​are transmitted back to the ground control center; Step S66: The ground control center triggers manual intervention to supplement resources, and uses the significant deviation information observed in the first phase to update the model parameters to improve the accuracy of subsequent inspection route predictions.

4. The intelligent streetlight drone inspection method based on multi-source video fusion according to claim 3 is characterized by: In step S64, the total number of street lights that the drone can complete is calculated. The calculation formula is: ； in, The current remaining available power of the drone; Reserve safety energy for the system; is the cruising power; The average horizontal distance between two adjacent lights; is the average cruising speed; is the ambient wind correction factor; The hovering power of the drone; is the benchmark hovering time of the UAV at the street light; is the estimated value of multi-source fusion trigger probability; is the hovering power; Additional hovering time caused by multi-source fusion; Calculate and transmit additional energy for a single fusion; Where, the denominator is the estimated average energy consumption of each streetlight in the second phase; in, ; In this formula, is the mean wind speed obtained by sampling in the first stage; is the wind speed sample variance; is the average cosine value obtained by sampling in the first stage, ,in is the angle between wind direction and flight direction; 、 is the control coefficient, which is a constant greater than 0; Used only in average headwinds When the wind is favorable, the denominator is not reduced to avoid over-optimism.

5. An intelligent streetlight drone inspection system based on multi-source video fusion for implementing the method of claim 1, characterized in that: It includes a patrol data acquisition module, a patrol image analysis module, a patrol mode switching module and a patrol task dynamic evaluation module; the patrol data acquisition module, the patrol image analysis module, the patrol mode switching module and the patrol task dynamic evaluation module are interconnected and communicated with each other; wherein, the patrol data acquisition module is used to perform flight inspections of street lights in the target patrol area one by one through a drone equipped with a visible light camera, and set image calibration array points to match the accuracy of patrol image acquisition; the patrol image analysis module is used to process and analyze the collected image data; the patrol mode switching module is used to switch to a single video source recognition mode or a multi-source video fusion mode according to the patrol image analysis results; the patrol task dynamic evaluation module is used to dynamically adjust the actual total amount of patrol tasks of the drone according to the current task completion, the remaining power and the estimated number of street lights in the current path that need to enter the multi-source fusion mode.

6. The intelligent streetlight drone inspection system based on multi-source video fusion according to claim 5 is characterized by: The inspection data acquisition module includes an inspection plan database and a real-time image acquisition module. The inspection plan database is used to store the geographical coordinates of street lights, preset inspection paths and image calibration array templates. The real-time image acquisition module collects visible light images when the drone hovers over the target street light and associates them with the current street light location information.

7. The intelligent streetlight drone inspection system based on multi-source video fusion according to claim 5 is characterized by: The inspection image analysis module includes an image processing module, an image fitting module, a multi-level feature generation module and a multi-level feature matching module. The image processing module is used to grayscale the visible light image, the image fitting module is used to extract the image contour and fit the boundary features, the multi-level feature generation module is used to sequentially extract multi-level feature information of the image to provide a comprehensive data basis for accurate matching, and the multi-level feature matching module performs accurate matching operations in stages based on the generated feature information.

8. The intelligent streetlight drone inspection system based on multi-source video fusion according to claim 5 is characterized by: The inspection task dynamic evaluation module includes an inspection path division module, an environmental information collection module, a drone collection and acquisition module and an energy consumption prediction module. The inspection path division module is used to divide the preset inspection path into different stages and record the number of street lights and related data in each stage. The environmental information collection module is suitable for collecting environmental information related to wind speed and wind direction during the drone inspection process. The drone collection and acquisition module is used to obtain the performance parameters and remaining power of the drone. The energy consumption prediction module is used to calculate the total number of street lights that the drone can complete based on the collected data and the performance parameters of the drone, and dynamically adjust the total amount of actual inspection tasks.

9. The intelligent streetlight drone inspection system based on multi-source video fusion according to claim 8 is characterized by: The energy consumption prediction module further includes a calculation unit and an environmental wind correction factor calculation unit. The calculation unit is used to calculate the total number of street lights that can be completed by the drone, and the environmental wind correction factor calculation unit is used to calculate the environmental wind correction factor to correct the energy consumption prediction result.

Images