A multi-source data fusion method, system and unmanned aerial vehicle for unmanned aerial vehicle inspection

Abstract

The application provides a multi-source data fusion method and system for unmanned aerial vehicle inspection and an unmanned aerial vehicle, and relates to the technical field of data processing.The method comprises the following steps: collecting original data of an inspection object by a multi-modal sensor carried on an unmanned aerial vehicle platform, wherein the multi-modal sensor at least comprises a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit; and pre-processing the original data, including noise filtering by using a sliding window filter and dynamic threshold detection, and marking abnormal data, so as to obtain pre-processed multi-source data.The application realizes safe and stable data transmission and supports ground monitoring and operation decision-making.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-source data fusion method and system for unmanned aerial vehicle (UAV) inspection and an unmanned aerial vehicle. BACKGROUND

[0002] In the winter operation scene of power lines, line short circuit and tower collapse accidents caused by icing seriously threaten power supply safety. Traditional unmanned aerial vehicle inspection relies on single laser radar or visual sensor to collect icing data, which is difficult to integrate environmental interference and flight state information synchronously, resulting in large deviation in ice thickness measurement and lagging in ice removal decision.

[0003] Taking the winter inspection of a 500kV power transmission line in a certain province as an example, the operation team uses an unmanned aerial vehicle equipped with only a visual sensor to carry out icing monitoring. Due to low temperature, the sensor lens is foggy, and the collected ice layer image is blurred. At the same time, the unmanned aerial vehicle is affected by 6-level gust, and the flight attitude is unstable. The state data such as wind speed and flight height are not recorded synchronously. The error of the finally calculated ice thickness is up to 15%, and it is impossible to judge whether the ice layer is in a continuous thickening state. The line appears partial conductor breakage failure due to icing overload. The core of the technical defect is that a multi-source data set integrating ice thickness, environment and flight state is not constructed. Single sensor data is easily affected by weather interference, and there is lack of flight state data to correct the ice layer measurement results. It is difficult to meet the technical requirements of unmanned aerial vehicle auxiliary control intelligent ice measurement and removal equipment for icing data precision and general platform for environmental adaptability of air line collaborative operation. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-source data fusion method and system for unmanned aerial vehicle inspection and an unmanned aerial vehicle, to realize safe and stable data transmission and support ground monitoring and operation decision.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows:

[0006] In a first aspect, a multi-source data fusion method for unmanned aerial vehicle inspection is provided, which comprises:

[0007] The original data of the inspection object is collected by the multi-modal sensor mounted on the unmanned aerial vehicle platform, and the multi-modal sensor at least includes a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit.

[0008] The original data is preprocessed, including noise filtering by using sliding window filtering and dynamic threshold detection, and abnormal data is marked, to obtain preprocessed multi-source data.

[0009] The preprocessed multi-source data is time and space aligned and synchronously processed to construct a multi-dimensional fusion data set containing environmental characteristics, inspection object state, ice thickness and flight state information.

[0010] inputting the multi-dimensional fused data set into a deep neural network for feature extraction to obtain extracted features; fusing the extracted features to obtain fused deep features; based on the fused deep features, obtaining, through a classification and regression network, an identification result, a state evaluation result and an icing monitoring result of the inspection object;

[0011] transmitting the identification and evaluation results to a ground station through an anti-interference communication link for real-time monitoring, decision support and deicing decision.

[0012] Further, the original data of the inspection object is collected by the multi-modal sensor mounted on the unmanned aerial vehicle platform, and the multi-modal sensor at least includes a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit, including:

[0013] In the process of flight inspection of the unmanned aerial vehicle, the visual sensor is started to synchronously collect high-resolution visible light images and infrared thermal imaging sequences of the inspection object to obtain preliminary visual data streams;

[0014] Based on the range of the inspection area covered by the visual data stream, the laser radar is started to synchronously scan the inspection object in the corresponding area to obtain surface point cloud data and reflection intensity information to obtain a point cloud data stream;

[0015] Based on the surface geometric features of the inspection object reflected by the point cloud data stream, the dual-polarization radar is started to synchronously transmit and receive electromagnetic wave signals to collect dual-polarization differential reflectivity and differential phase shift data related to the physical characteristics of the icing to obtain a radar polarization data stream;

[0016] In the process of collecting data by the visual sensor, the laser radar and the dual-polarization radar, the inertial measurement unit is started to synchronously collect three-axis acceleration, three-axis angular velocity and three-axis magnetic field data of the unmanned aerial vehicle platform to obtain an inertial measurement data stream;

[0017] The visual data stream, the point cloud data stream, the radar polarization data stream and the inertial measurement data stream are subjected to unified time processing to obtain original data streams with time sequence markers.

[0018] Further, the original data is preprocessed, including noise filtering by using a sliding window filtering and dynamic threshold detection, and abnormal data is marked to obtain preprocessed multi-source data, including:

[0019] The original data stream with time sequence markers is received, and the three-axis acceleration, three-axis angular velocity and three-axis magnetic field data collected by the inertial measurement unit are subjected to smoothing processing by using a sliding window filtering algorithm to obtain denoised inertial measurement data;

[0020] Based on the denoised inertial measurement data, the high-resolution visible light images and infrared thermal imaging sequences collected by the visual sensor are subjected to motion compensation and image enhancement processing to obtain stable visual data.

[0021] Based on the denoised inertial measurement data, the high-precision point cloud data collected by the laser radar is subjected to motion distortion correction to obtain corrected point cloud data.

[0022] The dual-polarization difference reflectivity and differential phase shift data collected by the dual-polarization radar are subjected to noise suppression and outlier detection using dynamic threshold detection to obtain processed radar polarization data.

[0023] The stable visual data, corrected point cloud data and processed radar polarization data are marked for abnormal data, and the timestamp information is integrated to obtain preprocessed multi-source data.

[0024] Further, the preprocessed multi-source data is subjected to spatio-temporal alignment and synchronization processing to construct a multi-dimensional fusion data set containing environmental features, inspection object state, ice layer thickness and flight state information, including:

[0025] Based on the timestamp information contained in the preprocessed multi-source data, a unified time reference is established, and the time synchronization processing is performed on each modality data to obtain time-synchronized multi-source data.

[0026] Based on the time-synchronized multi-source data, the motion trajectory of the unmanned aerial vehicle platform is established based on the inertial measurement data, and the spatial position registration is performed on the collected visual images and point cloud data to obtain spatially registered multi-source data.

[0027] Based on the spatially registered multi-source data, the mapping relationship between the visual image coordinate system and the laser radar point cloud coordinate system is established to realize the spatial alignment of visual features and point cloud features, and to obtain spatially aligned visual and point cloud fusion data.

[0028] Based on the spatially aligned visual and point cloud fusion data, the spatial correlation with the radar polarization data is established, and the differential reflectivity and differential phase shift data collected by the dual-polarization radar are mapped to the corresponding spatial position to obtain multi-modality spatially correlated data.

[0029] Based on the multi-modality spatially correlated data, environmental feature information, inspection object three-dimensional morphological features, ice physical property parameters and unmanned aerial vehicle flight state parameters are extracted to obtain a multi-dimensional fusion data set with a unified spatio-temporal reference.

[0030] Further, the multi-dimensional fusion data set is input into a deep neural network for feature extraction to obtain extracted features; the extracted features are fused to obtain fused deep features, including:

[0031] Based on the multi-dimensional fusion data set with unified space-time reference, the visual data in the multi-dimensional fusion data set is extracted and input into a convolutional neural network for visual feature extraction, obtaining deep visual features;

[0032] Based on the deep visual features and the point cloud data in the multi-dimensional fusion data set, three-dimensional feature extraction is performed through a point cloud processing network, obtaining deep point cloud features associated with the visual features;

[0033] Based on the deep visual features and the deep point cloud features, combined with the radar polarization data in the multi-dimensional fusion data set, polarization feature extraction is performed through a radar feature extraction network, obtaining deep radar features;

[0034] The deep visual features, deep point cloud features and deep radar features are input into a feature fusion network, and weighted fusion is performed through an attention mechanism, obtaining fused deep features.

[0035] Further, based on the fused deep features, through a classification and regression network, the recognition result, state evaluation result and icing monitoring of the inspection object are obtained, including:

[0036] The fused deep features are normalized to obtain standardized deep feature vectors; based on the standardized deep feature vectors, the type of the inspection object is identified through a classification network, obtaining a preliminary recognition result;

[0037] Based on the preliminary recognition result and the deep feature vectors, the structure state parameters of the inspection object are estimated to obtain a preliminary result of state evaluation;

[0038] Based on the preliminary result of state evaluation and the deep feature vectors, the icing thickness and icing type are calculated to obtain icing monitoring data;

[0039] The preliminary recognition result, the preliminary result of state evaluation and the icing monitoring data are input into a result optimization network for comprehensive optimization processing, obtaining the final recognition result, state evaluation result and icing monitoring result.

[0040] Further, the recognition and evaluation results are transmitted to the ground station through an anti-interference communication link for real-time monitoring, decision support and deicing decision, including:

[0041] The recognition result, state evaluation result and icing monitoring data are structured and organized according to a predetermined format, obtaining standardized transmission data packets;

[0042] Based on the standardized transmission data packets, time stamps and check information are added, obtaining data frames with complete transmission information;

[0043] Based on the data frame, the anti-interference coding technology is used for data encryption and channel coding processing to obtain an anti-interference coded data transmission stream.

[0044] Based on the anti-interference coded data transmission stream, data transmission is performed through a communication device of the unmanned aerial vehicle platform according to a preset anti-interference communication protocol, and is sent to the ground station in real time.

[0045] The received data is parsed and decoded at the ground station end, and the parsed identification and evaluation results are pushed to a real-time display unit, a decision analysis unit and a deicing scheduling unit, for realizing real-time state display, operation and maintenance decision making and deicing operation scheduling.

[0046] In a second aspect, a multi-source data fusion system for unmanned aerial vehicle inspection includes:

[0047] The acquisition module is configured to collect original data of the inspection object through a multi-modal sensor mounted on the unmanned aerial vehicle platform, the multi-modal sensor including at least a vision sensor, a laser radar, a dual-polarization radar and an inertial measurement unit; the original data is preprocessed, including noise filtering by using a sliding window filter and dynamic threshold detection, and abnormal data is marked, to obtain preprocessed multi-source data; the preprocessed multi-source data is subjected to spatio-temporal alignment and synchronization processing, to construct a multi-dimensional fusion data set including environmental characteristics, inspection object state, ice layer thickness and flight state information;

[0048] The computing module is configured to input the multi-dimensional fusion data set into a deep neural network for feature extraction, to obtain extracted features; the extracted features are fused to obtain fused deep features; based on the fused deep features, through a classification and regression network, an identification result, a state evaluation result and an icing monitoring of the inspection object are obtained.

[0049] The processing module is configured to transmit the identification and evaluation results to the ground station through an anti-interference communication link, for real-time monitoring, decision support and deicing decision making.

[0050] In a third aspect, an unmanned aerial vehicle includes:

[0051] One or more processors;

[0052] A storage device is configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0053] In a fourth aspect, a computer readable storage medium stores a program, when the program is executed by a processor, the method is implemented.

[0054] The above scheme of the present application at least has the following beneficial effects:

[0055] The technical means of multi-modal sensor cooperation, visual sensor, laser radar, dual-polarization radar, inertial measurement unit, sliding window filtering and dynamic threshold detection preprocessing, spatiotemporal alignment to construct a multi-dimensional fusion dataset, deep neural network and attention mechanism feature fusion and anti-interference communication link transmission overcome the technical problems that in traditional unmanned aerial vehicle inspection, a single sensor is easy to be disturbed by weather, lacks flight state data correction basis, causes large ice layer thickness calculation deviation, deicing decision lag, and is difficult to meet the requirements of unmanned aerial vehicle auxiliary control intelligent deicing equipment icing data precision and air line cooperative operation general platform environmental adaptability, and then multi-source data high-quality fusion processing, data safe and stable transmission are achieved, reliable data support is provided for ground real-time monitoring, operation decision making and deicing operation scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flowchart of a multi-source data fusion method for unmanned aerial vehicle inspection provided by an embodiment of the present application.

[0057] Figure 2 is a multi-source data fusion system schematic diagram for unmanned aerial vehicle inspection provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0059] As Figure 1 shown, an embodiment of the present application proposes a multi-source data fusion method for unmanned aerial vehicle inspection, the method comprising the following steps:

[0060] Step 1, collecting original data of the inspection object by the multi-modal sensor carried on the unmanned aerial vehicle platform, the multi-modal sensor at least including a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit;

[0061] Step 2, preprocessing the original data, including noise filtering by sliding window filtering and dynamic threshold detection, and marking abnormal data, to obtain preprocessed multi-source data;

[0062] Step 3, spatiotemporal alignment and synchronization processing of the preprocessed multi-source data to construct a multi-dimensional fusion dataset containing environmental features, inspection object state, ice layer thickness and flight state information;

[0063] Step 4, input the multi-dimensional fusion data set into a deep neural network for feature extraction to obtain extracted features; fuse the extracted features to obtain fused deep features; based on the fused deep features, through a classification and regression network, obtain the recognition result, state evaluation result and icing monitoring of the inspection object;

[0064] Step 5, transmit the recognition and evaluation results to the ground station through an anti-interference communication link for real-time monitoring, decision support and deicing decision.

[0065] In the embodiment of the application, a multi-modal sensor combination is constructed by using a laser radar, a dual-polarization radar and a visual sensor, an inertial measurement unit, combined with sliding window filtering and dynamic threshold detection preprocessing, spatio-temporal alignment to construct a multi-dimensional fusion data set containing ice layer thickness information, deep neural network and feature fusion technology, and anti-interference communication technology in the deicing equipment scheme, such as wavelet denoising and dual-channel communication technology, which overcomes the technical problems of traditional unmanned aerial vehicle inspection relying on a single sensor being easily disturbed by the environment, being unable to synchronously integrate ice layer thickness, environmental and flight state data, resulting in low measurement accuracy and deicing decision lag, and being difficult to adapt to the high-precision requirements of ice layer thickness error and air line cooperative platform environmental adaptability, thereby achieving efficient collaborative processing of multi-source data, improving ice monitoring precision, and stable and reliable data transmission, providing data support for ground real-time monitoring, deicing decision and air line cooperative operation scheduling.

[0066] In a preferred embodiment of the application, the above step 1 can include:

[0067] Step 1.1, during the flight inspection process of the unmanned aerial vehicle, a visual sensor is started to synchronously collect high-resolution visible light images and infrared thermal imaging sequences of the inspection object to obtain preliminary visual data flow, specifically including: when the unmanned aerial vehicle carries out winter flight inspection on the power line according to the preset inspection path, the visual sensor carried is started, the sensor meets the demand of the intelligent perception module of the general platform scheme for air line cooperative operation for high-resolution image collection, can synchronously capture high-resolution visible light images of the inspection object such as power transmission conductor and tower, and collect infrared thermal imaging sequences, preliminarily identify whether there is ice accumulation, appearance damage and the like on the surface of the inspection object through the visible light images, perceive temperature differences of different regions of the inspection object by means of the infrared thermal imaging sequences, and form preliminary visual data flow in this way.

[0068] Step 1.2, based on the inspection area range covered by the visual data stream, start the laser radar to synchronously scan the inspection object in the corresponding area to obtain the surface point cloud data and the reflection intensity information, and obtain the point cloud data stream, specifically including: after obtaining the visual data stream, analyzing the transmission line inspection area range covered by the data stream, and determining the key parts such as the wire segment and the tower connection that need to be scanned, then starting the laser radar that meets the icing measurement technical parameters in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme, the scanning frequency and resolution of the laser radar meet the high-precision measurement requirements, and the laser radar synchronously scans the transmission line and tower inspection object in the visual data stream covered area, in the scanning process, the laser radar emits laser beams to the surface of the inspection object, receives the reflected laser signals, obtains the high-precision point cloud data of the surface of the inspection object according to the signal feedback, and records the reflection intensity information of the laser signal, so as to generate the point cloud data stream.

[0069] Step 1.3, based on the surface geometric characteristics of the inspection object reflected by the point cloud data stream, start the dual-polarization radar to synchronously transmit and receive electromagnetic wave signals, collect the dual-polarization differential reflectivity and differential phase shift data related to the physical characteristics of the icing, and obtain the radar polarization data stream, specifically including: analyzing the generated point cloud data stream, extracting the information reflecting the surface geometric characteristics of the inspection object such as the transmission line, including whether there is a protrusion on the surface of the wire, the distribution range and the general shape of the protrusion area, so as to determine the area where the icing may exist, then start the dual-polarization radar that also meets the requirements of the icing measurement module of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme, the dual-polarization radar collects icing physical characteristics related data, the dual-polarization radar synchronously transmits electromagnetic wave signals to the determined possible icing area, the electromagnetic wave signals are reflected after contacting the icing surface, the radar receives the reflected electromagnetic wave signals, obtains the dual-polarization differential reflectivity data related to the physical characteristics of the icing by analyzing the reflected signals, the dual-polarization differential reflectivity data can reflect the density and particle size of the icing, and obtains the differential phase shift data, the differential phase shift data can reflect the phase change of the electromagnetic wave when passing through the icing, and then the radar polarization data stream is formed.

[0070] Step 1.4, in the process of collecting data by visual sensor, laser radar and dual polarization radar, the inertial measurement unit is started synchronously to collect the three-axis acceleration, three-axis angular velocity and three-axis magnetic field data of the unmanned aerial vehicle platform, and the inertial measurement data stream is obtained, specifically including: in the whole process of collecting visual data stream by visual sensor, collecting point cloud data stream by laser radar, and collecting radar polarization data stream by dual polarization radar, the inertial measurement unit carried is started synchronously, the inertial measurement unit adapts to the demand of intelligent sensing module for flight state data collection in the general platform scheme for air line collaborative operation, can capture the motion state of the unmanned aerial vehicle platform in the flight process in real time, specifically collect the three-axis acceleration data of the unmanned aerial vehicle in three-dimensional space, so as to judge the change of the flight speed of the unmanned aerial vehicle, collect three-axis angular velocity data, master the change of flight attitude such as inclination angle of the unmanned aerial vehicle, and collect three-axis magnetic field data, assist to judge whether the magnetic field environment where the unmanned aerial vehicle is located interferes with the flight, and integrate the collected data to form the inertial measurement data stream.

[0071] Step 1.5, the visual data stream, point cloud data stream, radar polarization data stream and inertial measurement data stream are uniformly time processed to obtain the original data stream with time sequence mark, specifically including: after obtaining the visual data stream, point cloud data stream, radar polarization data stream and inertial measurement data stream respectively, with the aid of the unified time device carried by the unmanned aerial vehicle, the device meets the demand of the intelligent sensing module for data time sequence synchronization in the general platform scheme for air line collaborative operation, and adapts to the requirement of multi-source data collaborative processing in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme, according to the preset time synchronization standard, the four kinds of data streams are uniformly time processed; add corresponding time mark on each data record, ensure that the high-resolution visible light image and infrared thermal imaging sequence in the visual data stream at a certain moment, the surface point cloud data and reflection intensity information of the inspection object in the point cloud data stream at the same moment, the dual polarization difference reflectivity and difference phase shift data related to the physical characteristics of ice in the radar polarization data stream at the moment, and the three-axis acceleration, three-axis angular velocity and three-axis magnetic field data of the unmanned aerial vehicle platform in the inertial measurement data stream at the moment, all have the same and accurate time sequence mark; finally form the original data stream with time sequence mark.

[0072] In the embodiment of the present application, the visual sensor, the inertial measurement unit, the laser radar for ice monitoring, and the dual-polarization radar are used to collect data in a layered manner according to the logic of visual area definition, three-dimensional laser radar scanning, dual-polarization radar ice monitoring, and inertial measurement unit flight state recording, and all data streams are uniformly time-synchronized, thereby overcoming the technical problems of single sensor type, lack of data collection collaboration, and difficulty in correlating and integrating data due to the lack of a unified time sequence reference in traditional unmanned aerial vehicle inspection, and further meeting the needs of multi-modal data collection for aerial line collaborative operation for visual and flight state monitoring of the inspection area, and meeting the requirements of intelligent deicing equipment for high-precision three-dimensional data and physical property data collection.

[0073] In a preferred embodiment of the present application, step 2 can include:

[0074] Step 2.1, receiving the original data stream with time sequence marking, and using a sliding window filtering algorithm to smooth the three-axis acceleration, three-axis angular velocity, and three-axis magnetic field data collected by the inertial measurement unit, to obtain denoised inertial measurement data, specifically including: after the unmanned aerial vehicle completes the inspection flight of the power line and returns with the original data stream with time sequence marking, the inspection system starts processing the data. First, for the three-axis acceleration, three-axis angular velocity, and three-axis magnetic field data collected by the inertial measurement unit in the original data stream, a sliding window filtering algorithm is selected; according to the requirements for data stability in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme and the requirements for flight state data accuracy in the general platform scheme for aerial line collaborative operation, the size and related parameters of the sliding window are determined. In specific operation, according to the set window size, the window is slid on the inertial measurement data sequence, and statistical processing such as mean calculation is performed on the data in the window, thereby effectively filtering out high-frequency vibration noise, for example, if the window size is set to 10 data points, when the window slides on the acceleration data sequence, the values of the 10 consecutive acceleration data points are added, and then divided by 10 to obtain the average value as the filtering output value at that window position, and then the denoised inertial measurement data is obtained.

[0075] Step 2.2, based on the denoised inertial measurement data, the high-resolution visible light images and infrared thermal imaging sequences collected by the visual sensor are subjected to motion compensation and image enhancement processing to obtain stable visual data, specifically including: after obtaining the denoised inertial measurement data, the high-resolution visible light images and infrared thermal imaging sequences collected by the visual sensor are subjected to expansion processing through the inertial measurement data. In the scene of winter inspection of power lines, the flight attitude of the unmanned aerial vehicle is constantly changing due to factors such as wind gusts, resulting in blurring and displacement deviation of the collected visual images. At this time, according to the information such as three-axis acceleration and angular velocity of the unmanned aerial vehicle recorded in the denoised inertial measurement data, the offset and rotation angle of each frame of image due to the movement of the unmanned aerial vehicle are calculated through a specific algorithm. For the offset and rotation, the high-resolution visible light images and infrared thermal imaging sequences are subjected to motion compensation operation to restore the images to a relatively stable state. At the same time, image enhancement algorithms are used, such as adjusting the contrast and brightness parameters of the image, highlighting the differences between the icing area and the normal area of the inspection object such as the power transmission conductor, making the image details clearer, and further improving the image quality, to ultimately obtain stable visual data.

[0076] Step 2.3, based on the denoised inertial measurement data, the high-precision point cloud data collected by the laser radar is subjected to motion distortion correction to obtain corrected point cloud data, specifically including: based on the denoised inertial measurement data, the point cloud data collected by the laser radar is subjected to motion distortion correction. During the inspection of the unmanned aerial vehicle, due to its own flight movement, the laser radar will cause motion distortion of the collected point cloud data when scanning, for example, translation distortion will cause the point cloud to be stretched, and rotation distortion will cause the point cloud to be distorted, which will seriously affect the accurate restoration of the three-dimensional form of the inspection object. Through the denoised inertial measurement data, the motion state of the laser radar at each moment is analyzed, and through integral operation, the angular velocity and acceleration data of the inertial measurement unit are converted into the pose changes of the laser radar in the scanning process, including displacement and rotation information. For each point in the point cloud data, according to its emission timestamp, the pose change information calculated above is used to convert the point from the local coordinate system in the radar motion to the global coordinate system at the start time of scanning, thereby correcting the motion distortion and obtaining the corrected point cloud data.

[0077] Step 2.4, the dual-polarization differential reflectivity and differential phase shift data collected by the dual-polarization radar are subjected to noise suppression and outlier detection by dynamic threshold detection to obtain processed radar polarization data, specifically including: for the dual-polarization differential reflectivity and differential phase shift data collected by the dual-polarization radar, dynamic threshold detection technology is used. Under the complex electromagnetic environment around the power line, the dual-polarization radar data is easily affected by various interferences, producing noise and outliers, which affects the judgment of the physical properties of ice coating. The dynamic threshold detection technology adjusts the detection threshold in real time according to the characteristics of the power line inspection scene and the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme for radar data processing, combined with the statistical characteristics of the current data. For example, by calculating the mean and standard deviation of the data within a certain period of time, a dynamic threshold range is set. When the dual-polarization differential reflectivity and differential phase shift data exceed this threshold range, it is determined to be an outlier or noise point. For detected noise, suppression measures are taken to reduce its impact on the overall data. For outliers, labeling is performed to obtain processed radar polarization data.

[0078] Step 2.5, the stable visual data, corrected point cloud data and processed radar polarization data are subjected to abnormal data labeling, and the timestamp information is integrated to obtain preprocessed multi-source data, specifically including: after obtaining the stable visual data, corrected point cloud data and processed radar polarization data, the three types of data are subjected to abnormal data labeling. For visual data, if the image features of a certain area are too different from the normal inspection object features, or the image appears obvious damage or loss, part of the data is labeled as abnormal. For corrected point cloud data, if the point cloud distribution appears abnormal aggregation or dispersion that does not conform to the normal geometric shape of the inspection object, the related point cloud data is labeled as abnormal. In the processed radar polarization data, if there are values that deviate greatly from the normal ice coating physical properties data, they are also labeled as abnormal. After completing the abnormal data labeling, the three types of data are integrated, and the corresponding timestamp information in the original data stream is accurately added to the integrated data to ensure that each piece of data has accurate time identification, and finally the preprocessed multi-source data is obtained.

[0079] In the embodiment of the application, the inertial measurement unit data is smoothed and denoised by a sliding window filtering algorithm, the visual sensor data is motion compensated and image enhanced based on the denoised inertial measurement data, the laser radar point cloud data is motion distortion corrected, the dual polarization radar data is detected and suppressed by a dynamic threshold, and the abnormal values are detected. Finally, the data and time stamp are integrated and marked as abnormal. The sliding window filtering and dynamic threshold detection techniques are adapted to the anti-interference requirements of the intelligent perception module in the general platform scheme for air line collaborative operation and the data requirements of the icing measuring device in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme. The problems of high-frequency noise in the inertial measurement data, blurred visual images due to flight shaking, motion distortion in the laser radar point cloud, and abnormal values in the dual polarization radar data in the traditional unmanned aerial vehicle inspection are overcome, so that the quality of the multi-source data is low, and the multi-source data noise and deviation are effectively eliminated, the visual data is more stable, the point cloud data is more accurate, and the radar data is more reliable.

[0080] In a preferred embodiment of the application, step 3 can include:

[0081] Step 3.1, based on the time stamp information contained in the preprocessed multi-source data, a unified time reference is established, and each modal data is time synchronized to obtain time synchronized multi-source data, specifically including: after obtaining the preprocessed multi-source data, first extract the time stamp information carried by the stable visual data, the corrected point cloud data and the processed radar polarization data, refer to the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme for data time consistency, and the time standard of data interaction in the general platform scheme for air line collaborative operation, take the time stamp of one set of data as the reference, such as selecting the time stamp of inertial measurement data as the core reference, and establishing a unified time reference; then, compare the time stamp of the modal data with the difference of the unified reference, adjust the time of each modal data according to the difference value, and unify the time stamp corresponding to the time in different modal data to the same time axis, so that the corresponding visual data, point cloud data, radar polarization data and inertial measurement data can be obtained at the same time node, and finally the time synchronized multi-source data is obtained.

[0082] Step 3.2, based on the time-synchronized multi-source data, the flight trajectory of the UAV platform is established by the inertial measurement data, and the spatial position registration is performed on the collected visual image and point cloud data to obtain the spatially registered multi-source data, which specifically includes: based on the time-synchronized multi-source data, the denoised inertial measurement data is extracted therefrom, and the inertial measurement data contains three-axis acceleration, three-axis angular velocity and three-axis magnetic field information during the flight of the UAV. According to the technical specification for calculating the flight trajectory of the UAV in the general platform scheme for air line collaborative operation, the angular velocity data is integrated to obtain the attitude change of the UAV, and then the acceleration data is integrated to obtain the position change of the UAV in the three-dimensional space, and the complete flight trajectory of the UAV platform is gradually constructed; then, according to the constructed flight trajectory, the spatial position of the UAV at different time nodes is analyzed, and the spatial position registration is performed on the time-synchronized visual image and point cloud data; for example, according to the position information of the UAV at a certain time, the spatial coordinates corresponding to the shooting angle of the visual image and the three-dimensional coordinates of the point cloud data are adjusted, so that the inspection object area reflected by the visual image and the space range of the inspection object presented by the point cloud data are accurately corresponding, and the spatially registered multi-source data is obtained.

[0083] Step 3.3, based on the spatially registered multi-source data, the mapping relationship between the visual image coordinate system and the laser radar point cloud coordinate system is established, the spatial alignment of the visual features and the point cloud features is realized, and the spatially aligned visual and point cloud fusion data is obtained, which specifically includes: for the spatially registered multi-source data, the parameters of the visual image coordinate system and the laser radar point cloud coordinate system are determined respectively; the visual image coordinate system takes the upper left corner of the image as the origin, and the horizontal and vertical directions are x-axis and y-axis respectively; the laser radar point cloud coordinate system establishes a three-dimensional space rectangular coordinate system with the radar center as the origin; according to the precision requirement of the space coordinate conversion of the icing measuring device in the UAV auxiliary control intelligent deicing equipment scheme, the coordinates of the feature points of the same inspection object in the two coordinate systems are collected, such as the end points of the power transmission conductor, the corner points of the tower, etc. The two-dimensional coordinates of the feature points in the visual image coordinate system and the three-dimensional coordinates of the feature points in the laser radar point cloud coordinate system are recorded. Using these corresponding coordinate data, the mapping relationship between the visual image coordinate system and the laser radar point cloud coordinate system is established, and the mapping relationship can realize the mutual conversion of the coordinates in the two coordinate systems. Based on the mapping relationship, the visual features in the visual image, such as the outline of the icing area and the color difference area, and the point cloud features in the point cloud data, such as the three-dimensional convex shape of the icing area, are accurately corresponding in space, the spatial alignment of the visual features and the point cloud features is realized, and the spatially aligned visual and point cloud fusion data is obtained.

[0084] Step 3.4, based on the spatially aligned visual and point cloud fusion data, establish a spatial correlation with the radar polarization data, map the differential reflectivity and differential phase shift data collected by the dual-polarization radar to the corresponding spatial location, and obtain multi-modal spatial correlation data, specifically including: based on the spatially aligned visual and point cloud fusion data, the precise spatial position of each part of the inspection object is clear, combined with the working parameters and data acquisition range of the dual-polarization radar in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme, analyze the corresponding relationship between the differential reflectivity and differential phase shift data collected by the dual-polarization radar and the spatial position of the inspection object; by identifying the signal characteristics related to the characteristics of the inspection object in the radar data, such as the intensity change and phase change rule of the radar reflection signal in the icing area, the radar signal characteristics are matched with the specific spatial position of the inspection object in the visual and point cloud fusion data, the correlation rule between the dual-polarization radar data and the spatial position of the inspection object is established, and according to the correlation rule, each group of differential reflectivity and differential phase shift data collected by the dual-polarization radar is accurately mapped to the corresponding spatial position of the inspection object in the visual and point cloud fusion data, so that the radar data can accurately reflect the physical characteristics of the icing at the corresponding spatial position, and multi-modal spatial correlation data is obtained.

[0085] Step 3.5, based on the multi-modal spatial correlation data, extract environmental feature information, inspection object three-dimensional morphological features, icing physical property parameters and unmanned aerial vehicle flight state parameters, and obtain a multi-dimensional fusion data set with unified space-time reference, specifically including: from the multi-modal spatial correlation data, according to the preset extraction rule, respectively extract various key parameters, when extracting environmental feature information, combined with the weather conditions reflected in the visual image, such as whether it is raining or snowing, visibility, and wind speed information and temperature information indirectly reflected in the inertial measurement data; when extracting the three-dimensional morphological features of the inspection object, based on the point cloud data, the diameter of the power transmission conductor, the height and structure size of the tower, etc. three-dimensional parameters are restored, and the texture and color appearance features of the surface of the inspection object are supplemented combined with the visual image; when extracting the icing physical property parameters, the density of the icing is determined according to the differential reflectivity data of the dual-polarization radar, the thickness of the icing is calculated according to the differential phase shift data, and the weight of the icing is calculated combined with the volume difference between the icing area and the conductor in the point cloud data; when extracting the unmanned aerial vehicle flight state parameters, the three-axis acceleration, three-axis angular velocity, and magnetic field data at each time node are directly obtained from the inertial measurement data, and the flight speed, height, and attitude angle parameters are calculated through the flight trajectory. The extracted parameters are integrated together to ensure that all parameters are based on a unified time reference and space reference, and finally a multi-dimensional fusion data set with unified space-time reference is formed, which completely covers the key information of the environment, the inspection object, the icing, and the unmanned aerial vehicle flight state.

[0086] In the embodiment of the present application, the time synchronization of each modal data is realized based on the establishment of a unified time reference based on pre-processed data timestamps, the spatial registration of vision and point cloud data is completed by building the motion trajectory of the unmanned aerial vehicle combined with inertial measurement data, the spatial alignment is realized by establishing the mapping relationship between the vision and point cloud coordinate systems, the vision and point cloud fusion data and radar polarization data are associated and mapped to the spatial position, and finally the multi-class features are extracted from the multi-modal spatial associated data to construct a multi-dimensional fusion data set with a unified time-space reference. The technical means adapt to the requirements of the icing measurement device in the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme for the time-space consistency of multi-source data, and the requirements of the intelligent sensing module in the general platform scheme for air line collaborative operation for the integration of environmental and inspection object features, overcoming the technical problems in the traditional inspection that the multi-modal data are not synchronized in time and space, and each data modal is isolated and unrelated, which leads to the inability to construct an effective data set containing ice layer thickness, environment and flight state, and the difficulty in supporting icing monitoring and collaborative operation. Thus, the multi-modal data are highly coordinated and unified in time and space dimensions, the multi-dimensional fusion data set formed can completely cover the environmental, inspection object form, icing characteristics and flight state information, meet the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment for icing data analysis, and adapt to the requirements of the general platform for air line collaborative operation for multi-source data integration and application.

[0087] In a preferred embodiment of the present application, step 4 can include:

[0088] Step 4.1, based on the multi-dimensional fusion data set with a unified time-space reference, extracting the vision data in the multi-dimensional fusion data set and inputting it into a convolutional neural network for vision feature extraction to obtain deep vision features, specifically including: after obtaining the multi-dimensional fusion data set with a unified time-space reference, extracting stable vision data from it, the vision data containing high-resolution visible light images and infrared thermal imaging sequences of the inspection object, meeting the requirements of the intelligent sensing of the general platform for air line collaborative operation for using vision data for defect recognition, inputting the vision data into a pre-trained convolutional neural network, the convolutional neural network being trained by a large number of power line inspection image samples and being able to capture key features in the image; in the processing process, the convolutional neural network extracts features from the image through multiple convolutional layers, gradually transitioning from low-level edge and texture features to high-level inspection object structure features such as the outline of the power transmission conductor and the appearance form difference of the icing area; at the same time, combined with the temperature features extracted from the infrared thermal imaging sequence, further distinguishing the temperature difference between the icing area and the normal line area, and finally obtaining deep vision features containing the appearance and temperature information of the inspection object.

[0089] Step 4.2, based on the depth visual features and the point cloud data in the multi-dimensional fusion dataset, three-dimensional feature extraction is performed through a point cloud processing network to obtain depth point cloud features associated with the visual features, specifically including: based on the obtained depth visual features, combining the corrected point cloud data in the multi-dimensional fusion dataset, inputting into the point cloud processing network, using the appearance features of the inspection object in the depth visual features, such as the outline range of the icing area, to preliminarily filter the vast point cloud data, focusing on the point cloud region corresponding to the visual features, reducing the interference of irrelevant point cloud data, then, the point cloud processing network extracts three-dimensional structural features in the point cloud data, such as the thickness distribution of the icing, the three-dimensional morphology of the power transmission conductor, and the spatial structure parameters of the tower, through point cloud segmentation and feature sampling, etc. In the extraction process, the association mapping between the point cloud features and the depth visual features is established to ensure that the three-dimensional features extracted from the point cloud and the appearance features captured by the vision are one-to-one corresponding in space position, for example, the icing protruding area reflected by the point cloud is matched with the icing color block area in the visual image, and finally the depth point cloud features associated with the visual features are obtained.

[0090] Step 4.3, based on the depth visual features and the depth point cloud features, combining the radar polarization data in the multi-dimensional fusion dataset, polarization feature extraction is performed through a radar feature extraction network to obtain depth radar features, specifically including: based on the obtained depth visual features and the obtained depth point cloud features, combining the processed radar polarization data in the multi-dimensional fusion dataset, inputting into the radar feature extraction network, using the depth visual features and the depth point cloud features to determine the spatial position and three-dimensional morphology of the inspection object, such as the attachment position of the icing on the conductor and the spatial range of the icing area, based on which the radar polarization data is spatially positioned, and the radar data segment corresponding to the icing area of the inspection object is selected, excluding environmental clutter interference, the radar feature extraction network analyzes the filtered radar polarization data, extracts polarization features related to the physical properties of the icing, such as judging the density type of the icing based on the dual-polarization differential reflectivity data, calculating the thickness change rate of the icing according to the differential phase shift data, and verifying and correcting the extracted radar polarization features through the icing appearance difference of the depth visual features and the three-dimensional structure of the icing of the depth point cloud features, to ensure that the radar features can accurately reflect the true physical state of the icing, and finally the depth radar features are obtained.

[0091] Step 4.4, the depth visual feature, the depth point cloud feature and the depth radar feature are input into a feature fusion network, and are weighted and fused through an attention mechanism to obtain a fused depth feature, specifically including: the obtained depth visual feature, depth point cloud feature and depth radar feature are input into the feature fusion network, and the attention mechanism is started for weighted fusion, the feature fusion network performs feature dimension unification and standardization processing on the three types of features, to ensure that each feature can be fused and operated in the same dimension space, the attention mechanism dynamically allocates the fusion weights of the three types of features according to the core requirements of the power line winter inspection, the precise ice monitoring and the state evaluation of the inspection object, the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme on the accuracy of the ice data, and the environmental adaptability requirements of the general platform scheme for the aerial line collaborative work, for example, in the ice thickness calculation link, the weights of the depth radar feature and the depth point cloud feature are increased; in the inspection object defect identification link, the weight of the depth visual feature is increased, and the advantages of the three types of features are integrated through weighted operation, such as the visual appearance defect, the three-dimensional structure of the point cloud and the physical characteristics of the radar are fused into a unified feature vector, and finally the fused depth feature is obtained.

[0092] In the embodiment of the application, the visual data is extracted from the multi-dimensional fusion data set of the unified space-time reference, the depth visual feature is extracted by inputting the visual data into a convolutional neural network, the associated depth point cloud feature is extracted by combining the features and the point cloud data through a point cloud processing network, the depth radar feature is extracted based on the first two types of depth features and the radar polarization data through a radar feature extraction network, and finally the three types of features are input into a feature fusion network through the technical means of weighted fusion with the aid of an attention mechanism. The convolutional neural network and the point cloud processing network meet the requirements of the intelligent perception module of the general platform scheme for aerial line collaborative work for precise extraction of visual and three-dimensional features of the inspection object. The radar feature extraction network and the attention mechanism meet the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme for deep analysis of ice polarization features and complementary fusion of multi-modal data. The technical problems of the traditional inspection, such as incomplete single-modal feature extraction and lack of targeted weight distribution in multi-feature fusion, which leads to low precision of ice monitoring and inspection object recognition, and cannot meet the technical requirements of the two schemes for data deep analysis, are solved, so that the depth features of each modality are fully extracted and closely associated, the fused depth features can fully integrate the advantages of vision, three-dimensional structure and ice polarization characteristics, meet the requirements of the general platform for inspection object recognition, and meet the requirements of the deicing equipment for ice physical property analysis.

[0093] In a preferred embodiment of the application, the above step 4 can include:

[0094] Step 4.5, the normalized depth features are obtained by normalizing the fused depth features; based on the normalized depth feature vectors, the type of the inspection object is identified through a classification network to obtain a preliminary identification result, which specifically includes: the normalized depth feature vectors are obtained by normalizing the fused depth features to adjust the feature values to a unified value range, eliminating the interference caused by the value magnitude difference between different feature dimensions; the normalized feature vectors are input into a pre-trained classification network; the classification network refers to the demand of intelligent perception for inspection object identification in the general platform scheme for air line collaborative work, and distinguishes different types of inspection objects such as transmission lines, towers and insulators through a large number of power line inspection samples; the classification network analyzes the feature information in the normalized feature vectors, such as the appearance form in the depth visual feature and the three-dimensional structure in the depth point cloud feature, to judge the type of the inspection object corresponding to the current data, and finally outputs the preliminary identification result.

[0095] Step 4.6, based on the preliminary identification result and the depth feature vector, the type of the inspection object is determined according to the preliminary identification result, which is a transmission line, a tower or an insulator, and the normalized depth feature vector is input into a first regression network. The network matches the corresponding structure state parameter evaluation dimension for different types of objects according to the accuracy requirements and technical specifications of the related scheme for structure state monitoring.

[0096] For the transmission line, the diameter change, surface damage degree and sag parameter are mainly evaluated: the diameter change is calculated by the difference between the cross-sectional size of the depth point cloud feature and the standard diameter; the surface damage degree is determined by comparing the damage area pixel ratio in the depth visual feature with the preset damage level threshold; the sag parameter is obtained by proportionally converting the spatial coordinate difference between the wire suspension point and the lowest point extracted from the point cloud feature, combined with the span parameter.

[0097] For the tower, the tilt angle, connection part fastening state and corrosion area parameter are focused on: the tilt angle is calculated by the angle between the tower central axis and the plumb line in the point cloud feature; the connection part fastening state is judged according to the relative displacement pixel amount of the connection component in the visual feature; the corrosion area is determined by the ratio of the number of pixels in the corrosion area to the total surface area of the tower in the visual feature.

[0098] For the insulator, the surface contamination degree and the umbrella skirt damage condition are mainly evaluated: the surface contamination degree is calculated by comparing the gray value change rate with the clean state reference value in the visual feature; the umbrella skirt damage condition is determined by the ratio of the damage area profile to the complete umbrella skirt area in the visual feature.

[0099] The network extracts relevant features from the deep feature vector, combines the preset evaluation standard to quantitatively calculate various parameters, and finally obtains the preliminary result of state evaluation; wherein, the first regression network is specially designed for the winter operation and maintenance demand of power lines, can determine the object type according to the preliminary identification result, call the corresponding evaluation dimension and calculation logic, and realize the targeted quantification of the structure state parameters of different inspection objects by analyzing the information related to the structure in the deep feature vector.

[0100] Step 4.7, based on the preliminary result of state evaluation and the deep feature vector, the ice thickness and ice type are calculated through the second regression network to obtain the ice monitoring data, specifically including: extracting the baseline structure parameters of the inspection object according to the preliminary result of state evaluation, such as the standard diameter of the power transmission conductor and the standard height of the tower, while calling the standardized deep feature vector, focusing on extracting the deep radar feature and deep point cloud feature to input the second regression network. This network combines related schemes to calculate the requirements for ice monitoring accuracy.

[0101] When calculating the ice thickness, first, the three-dimensional size after icing is obtained through the point cloud feature, and the size difference is obtained by comparing with the baseline parameters; then, the difference is corrected in combination with the radar differential phase shift data, and the correction coefficient is determined according to the corresponding relationship between the differential phase shift value and the known ice thickness; finally, the corrected difference is taken as the actual ice thickness, and for the cylindrical objects such as conductors, the average thickness in the circumference direction is taken; for irregular objects such as towers, the maximum thickness of the key parts is taken.

[0102] When judging the ice type, the radar differential reflectivity data is compared with the preset threshold value: when the differential reflectivity value is within the first threshold value range, it is determined as soft ice; when it is within the second threshold value range, it is determined as hard ice; when it is within the third threshold value range, it is determined as wet snow ice, and the texture features of the icing area in the deep visual feature, such as smoothness and transparency, are combined to assist in verifying the determination result, thereby improving the accuracy of type judgment.

[0103] Integrate the thickness and type information to form the ice monitoring data, wherein, the second regression network is specially designed for the ice monitoring demand of power lines, taking the baseline parameters in the preliminary result of state evaluation and the deep feature vector as input, analyzing the three-dimensional shape of the ice reflected by the point cloud feature and the polarization data contained in the radar feature, combining the physical property differences of different ice types, realizing the measurement of ice thickness and accurate judgment of ice type.

[0104] Step 4.8, the preliminary identification result, the preliminary result of state evaluation and the icing monitoring data are input into a result optimization network for comprehensive optimization processing to obtain the final identification result, the state evaluation result and the icing monitoring result, specifically including: the obtained preliminary identification result, the preliminary state evaluation result and the icing monitoring data are input into the result optimization network, the result optimization network refers to the common requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme and the general platform scheme for air line collaborative operation on data reliability, aiming to eliminate possible contradictions and deviations between different preliminary results; the result optimization network performs consistency checking on the three types of preliminary results, for example, judging whether the ice thickness in the icing monitoring data matches the structural bearing capacity of the inspection object in the preliminary state evaluation result, if the ice thickness exceeds the safe bearing range of the inspection object but the state evaluation does not reflect the abnormality, the related results are modified; then, combined with the historical inspection data and the preset evaluation standard, the fuzzy data or abnormal values in the preliminary result are adjusted, such as correcting the ice thickness data exceeding the reasonable error range to the value conforming to the actual monitoring law. Through such comprehensive optimization processing, the result optimization network finally outputs accurate and consistent final identification result, state evaluation result and icing monitoring result.

[0105] In the embodiment of the application, the fused deep features are normalized to obtain a standardized feature vector, the classification network is used to identify the type of the inspection object, the preliminary identification result and the feature vector are used to estimate the structural state parameters of the inspection object, the preliminary state evaluation result and the feature vector are used to calculate the ice thickness and type, and finally the three types of preliminary results are input into the result optimization network for comprehensive optimization. The classification network and the regression network adapt to the requirements of intelligent perception of the general platform scheme for air line collaborative operation on the identification and state evaluation of the inspection object, and the result optimization network meets the requirements of the unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme on the icing monitoring accuracy and result reliability, which overcomes the problems in the traditional inspection, such as large identification deviation caused by lack of standardized feature processing, rough evaluation result caused by inaccurate calculation of structural state and ice parameters without stages, and data contradiction caused by lack of comprehensive optimization, so that the type identification of the inspection object is more accurate, the estimation of the structural state parameters is more realistic, the calculation error of the ice thickness and type is controllable, and the optimized final result can eliminate data contradiction and improve reliability.

[0106] In a preferred embodiment of the application, the above step 5 can include:

[0107] Step 5.1, the recognition result, state evaluation result and icing monitoring data are structured and organized according to a predetermined format to obtain a standardized transmission data packet, specifically including: arranging the output final recognition result, state evaluation result and icing monitoring data, wherein the recognition result contains the power transmission conductor and the tower inspection object type, the state evaluation result covers the structure parameters of the inspection object, and the icing monitoring data includes the ice thickness and the icing type information; setting a predetermined structured format, which clearly defines the field name, data type and arrangement order of each type of data, for example, taking the inspection object type, structure inclination angle, ice thickness and icing type as a fixed field sequence, and filling the three types of data into the corresponding fields in this format to ensure that the data organization is regular and redundant-free, and finally forming a standardized transmission data packet.

[0108] Step 5.2, based on the standardized transmission data packet, adding a time stamp and verification information to obtain a data frame with complete transmission information, specifically including: after obtaining the standardized transmission data packet, extracting the inspection time information corresponding to the data packet, adding an accurate time stamp, which is consistent with the time reference of the previous data collection and processing link, to ensure that the ground station can accurately trace the inspection time corresponding to the data, at the same time, using a preset verification algorithm to calculate the standardized transmission data packet to generate verification information, which can be used to verify whether the data packet is complete and whether there is data tampering or loss in the transmission process after the ground station receives the data, and the time stamp and verification information are attached to the specified position of the standardized transmission data packet, for example, the time stamp is placed at the head of the data packet and the verification information is placed at the tail of the data packet, which together constitute a data frame with complete transmission information.

[0109] Step 5.3, based on the data frame, the anti-interference coding technology is used for data encryption and channel coding processing to obtain an anti-interference coded data transmission stream, which specifically includes: the obtained complete data frame is processed by using the anti-interference coding technology, the data security in the power line inspection scene is combined to prevent data stealing and tampering risk in the electromagnetic environment around the substation, meet the demand of unmanned aerial vehicle auxiliary control intelligent deicing equipment scheme for icing monitoring data security, and the specification of operation data security transmission of general platform scheme for air line collaborative operation, the selected symmetric encryption algorithm takes the pre-set key as the core, the key is synchronized by offline negotiation or secure key exchange mechanism between the unmanned aerial vehicle platform and the ground station, to ensure that only the authorized ground station can decrypt the data, in the encryption process, first, the complete data frame is divided into several data blocks according to the fixed length, if the length of the last data block is less than the fixed value, it is supplemented to the standard length through the pre-set padding rule, to avoid the encryption deviation caused by the inconsistent length of the data blocks, each data block is sequentially executed round transformation operation, each round transformation includes byte substitution, row shifting and column mixing steps: byte substitution maps each byte in the data block to a new byte through a pre-set substitution table, to realize preliminary confusion of the data; the rows of the data block are circularly shifted according to a fixed rule to disrupt the original arrangement order of the data; column mixing further diffuses the data characteristics through linear transformation to enhance the data security, after completing all rounds of transformation, the processed data blocks are spliced to form encrypted data, the encrypted data is encoded by using channel coding technology, and redundant check code is added to the data, the redundant code can help the ground station to recover the correct data when the data transmission encounters electromagnetic interference leading to signal distortion, to reduce the error code, after the double processing of encryption and channel coding, the anti-interference coded data transmission stream is formed.

[0110] Step 5.4, based on the anti-interference coded data transmission stream, the communication device of the unmanned aerial vehicle platform is used to transmit data according to the pre-set anti-interference communication protocol, and real-time transmission is sent to the ground station, which specifically includes: based on the obtained anti-interference coded data transmission stream, the communication device carried by the unmanned aerial vehicle platform is started, the technical parameters of the data processing and communication module in the communication device support dual-channel communication of 4G or 5G main link and long-distance radio backup network, according to the pre-set anti-interference communication protocol, first try to transmit data through 4G or 5G main link, if the main link causes unstable transmission due to weak signal or strong interference, automatically switch to long-distance radio backup network, to ensure that the data can be transmitted to the ground station in real time and continuously.

[0111] Step 5.5, the received data is parsed and decoded at the ground station end, and the parsed identification and evaluation results are pushed to the real-time display unit, the decision analysis unit and the deicing scheduling unit respectively, for realizing real-time state display, operation and maintenance decision making and deicing operation scheduling, specifically including: after the ground station receives the data transmitted by the unmanned aerial vehicle, the data is decoded according to the preset anti-interference communication protocol, and the data frame before encryption is restored, and then the integrity of the data frame is verified through the check information, if the verification fails, the unmanned aerial vehicle is requested to retransmit, after the verification passes, the timestamp and the standardized transmission data packet in the data frame are parsed, and the identification result, the state evaluation result and the ice cover monitoring data are extracted, then, according to the requirements of the operation scheduling of the air line collaborative operation, the parsed results are pushed to the corresponding units: the information such as the inspection object type and the real-time ice layer thickness is pushed to the real-time display unit, for the operation and maintenance personnel to intuitively view the line state; the data such as the structure state parameter and the ice cover growth trend is pushed to the decision analysis unit, for making operation and maintenance scheme; the accurate ice thickness and ice type data are pushed to the deicing scheduling unit, for providing scheduling basis for the deicing operation of the unmanned aerial vehicle auxiliary control intelligent deicing equipment, and finally realizing the integration and collaboration of real-time state display, operation and maintenance decision making and deicing operation scheduling.

[0112] In the embodiment of the application, the identification, state evaluation and ice cover monitoring results are structured and organized into standardized transmission data packets according to a predetermined format, time stamps and check information are added to form complete data frames, the data frames are encrypted through anti-interference encoding technology and channel encoding, then transmitted to the ground station through the unmanned aerial vehicle communication device according to the preset anti-interference protocol, finally the data is parsed by the ground station and pushed to the corresponding unit, the anti-interference encoding, communication data processing and anti-interference requirements of communication, such as wavelet denoising and double-channel communication, are logically consistent with the general data interaction and operation scheduling requirements of air line collaborative operation in structured organization and data pushing, which overcomes the technical problems of traditional inspection, such as difficulty in parsing due to non-uniform data transmission format, lack of anti-interference processing which is easily affected by the complex electromagnetic environment of the substation, and lack of clear targeting of data pushing which leads to lagging decision-making at the ground end, and thus meets the technical requirements of data transmission safety, stability and application-specificity of the two schemes, and further achieves the standardization of data transmission format, strong anti-interference ability in the transmission process, clear targeting of data application at the ground end, ensures the accuracy of deicing operation scheduling of the deicing equipment, and supports efficient decision-making of the general platform collaborative operation.

[0113] As shown in Figure 2 The embodiment of the application further provides a multi-source data fusion system for unmanned aerial vehicle inspection, comprising:

[0114] An acquisition module is configured to collect original data of the inspection object by a multi-modal sensor mounted on the UAV platform, the multi-modal sensor at least including a vision sensor, a laser radar, a dual-polarization radar and an inertial measurement unit; the original data is preprocessed, including noise filtering by using a sliding window filtering and dynamic threshold detection, and abnormal data is marked to obtain preprocessed multi-source data; the preprocessed multi-source data is subjected to spatio-temporal alignment and synchronization processing to construct a multi-dimensional fusion data set containing environmental characteristics, inspection object state, ice layer thickness and flight state information;

[0115] A calculation module is configured to input the multi-dimensional fusion data set into a deep neural network for feature extraction to obtain extracted features; the extracted features are fused to obtain fused deep features; based on the fused deep features, a classification and regression network is used to obtain the recognition result, state evaluation result and ice monitoring of the inspection object;

[0116] A processing module is configured to transmit the recognition and evaluation results to the ground station through an anti-interference communication link for real-time monitoring, decision support and deicing decision.

[0117] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A multi-source data fusion method for unmanned aerial vehicle inspection, characterized in that, The method comprises: Collecting original data of the inspection object by a multi-modal sensor mounted on the unmanned aerial vehicle platform, the multi-modal sensor comprising at least a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit; Preprocessing the original data, including noise filtering by using a sliding window filter and dynamic threshold detection, and marking abnormal data, to obtain preprocessed multi-source data; Performing spatio-temporal alignment and synchronization processing on the preprocessed multi-source data to construct a multi-dimensional fusion data set containing environmental characteristics, inspection object state, ice layer thickness and flight state information; Based on the multi-dimensional fusion data set with a unified spatio-temporal reference, extracting visual data in the multi-dimensional fusion data set and inputting into a convolutional neural network for visual feature extraction to obtain deep visual features; based on the deep visual features and point cloud data in the multi-dimensional fusion data set, performing three-dimensional feature extraction by a point cloud processing network to obtain deep point cloud features associated with the visual features; based on the deep visual features and the deep point cloud features, combining radar polarization data in the multi-dimensional fusion data set, performing polarization feature extraction by a radar feature extraction network to obtain deep radar features; inputting the deep visual features, the deep point cloud features and the deep radar features into a feature fusion network, performing weighted fusion by an attention mechanism to obtain fused deep features; performing normalization processing on the fused deep features to obtain a standardized deep feature vector; based on the standardized deep feature vector, performing type identification of the inspection object by a classification network to obtain a preliminary identification result; based on the preliminary identification result and the deep feature vector, estimating the structural state parameters of the inspection object by a first regression network to obtain a preliminary result of state evaluation; based on the preliminary result of state evaluation and the deep feature vector, calculating the ice thickness and the ice type by a second regression network to obtain ice monitoring data; inputting the preliminary identification result, the preliminary result of state evaluation and the ice monitoring data into a result optimization network for comprehensive optimization processing to obtain final identification results, state evaluation results and ice monitoring results; Structuring the identification results, the state evaluation results and the ice monitoring data according to a predetermined format to obtain standardized transmission data packets; based on the standardized transmission data packets, adding time stamps and check information to obtain data frames with complete transmission information; based on the data frames, performing data encryption and channel coding processing by an anti-interference coding technology to obtain anti-interference coded data transmission streams; based on the anti-interference coded data transmission streams, performing data transmission according to a predetermined anti-interference communication protocol by a communication device of the unmanned aerial vehicle platform, and sending in real time to a ground station; performing analysis and decoding processing on the received data at the ground station end, and pushing the analyzed identification and evaluation results to a real-time display unit, a decision analysis unit and an ice removal scheduling unit respectively, for realizing real-time state display, operation and maintenance decision making and ice removal operation scheduling. 2.The multi-source data fusion method for unmanned aerial vehicle inspection of claim 1, wherein, The original data of the inspection object is collected by a multi-modal sensor mounted on a UAV platform, the multi-modal sensor at least including a visual sensor, a laser radar, a dual-polarization radar and an inertial measurement unit, comprising: In the process of UAV flight inspection, the visual sensor is started to synchronously collect high-resolution visible light images and infrared thermal imaging sequences of the inspection object, to obtain preliminary visual data stream; Based on the range of the inspection area covered by the visual data stream, the laser radar is started to synchronously scan the inspection object in the corresponding area, to obtain surface point cloud data and reflection intensity information, to obtain point cloud data stream; Based on the surface geometric features of the inspection object reflected by the point cloud data stream, the dual-polarization radar is started to synchronously transmit and receive electromagnetic wave signals, to collect dual-polarization differential reflectivity and differential phase shift data related to the physical characteristics of ice, to obtain radar polarization data stream; In the process of collecting data by the visual sensor, the laser radar and the dual-polarization radar, the inertial measurement unit is started to synchronously collect three-axis acceleration, three-axis angular velocity and three-axis magnetic field data of the UAV platform, to obtain inertial measurement data stream; The visual data stream, the point cloud data stream, the radar polarization data stream and the inertial measurement data stream are subjected to unified time processing, to obtain original data stream with time sequence marking. 3.The multi-source data fusion method for unmanned aerial vehicle inspection of claim 2, wherein, The original data is preprocessed, including noise filtering by sliding window filtering and dynamic threshold detection, and abnormal data is marked, to obtain preprocessed multi-source data, comprising: The original data stream with time sequence marking is received, and the three-axis acceleration, three-axis angular velocity and three-axis magnetic field data collected by the inertial measurement unit are subjected to smoothing processing by sliding window filtering algorithm, to obtain denoised inertial measurement data; Based on the denoised inertial measurement data, the high-resolution visible light images and infrared thermal imaging sequences collected by the visual sensor are subjected to motion compensation and image enhancement processing, to obtain stable visual data; Based on the denoised inertial measurement data, the high-precision point cloud data collected by the laser radar is subjected to motion distortion correction, to obtain corrected point cloud data; The dual-polarization differential reflectivity and differential phase shift data collected by the dual-polarization radar are subjected to noise suppression and abnormal value detection by dynamic threshold detection, to obtain processed radar polarization data; The stable visual data, the corrected point cloud data and the processed radar polarization data are marked with abnormal data, and the time stamp information is integrated to obtain preprocessed multi-source data. 4.The multi-source data fusion method for unmanned aerial vehicle inspection of claim 3, wherein, The preprocessed multi-source data is subjected to space-time alignment and synchronization processing, to construct a multi-dimensional fusion data set containing environmental features, inspection object state, ice thickness and flight state information, comprising: Based on the time stamp information contained in the preprocessed multi-source data, a unified time reference is established, and each modal data is subjected to time synchronization processing, to obtain time-synchronized multi-source data; Based on the time-synchronized multi-source data, the motion trajectory of the UAV platform is established by the inertial measurement data, and the collected visual images and point cloud data are subjected to spatial position registration, to obtain spatially registered multi-source data; Based on the multi-source data after spatial registration, a mapping relationship between the visual image coordinate system and the laser radar point cloud coordinate system is established, the spatial alignment of the visual features and the point cloud features is realized, and the spatially aligned visual and point cloud fusion data are obtained; Based on the spatially aligned visual and point cloud fusion data, a spatial correlation with the radar polarization data is established, the differential reflectivity and differential phase shift data collected by the dual-polarization radar are mapped to the corresponding spatial positions, and multi-modal spatial correlation data are obtained; Based on the multi-modal spatial correlation data, environmental feature information, three-dimensional morphological features of the inspection object, icing physical property parameters, and unmanned aerial vehicle flight state parameters are extracted, and a multi-dimensional fusion data set with a unified space-time reference is obtained.

5. A multi-source data fusion system for unmanned aerial vehicle inspection, the system implementing the method of any one of claims 1 to 4, characterized in that, It comprises: An acquisition module is configured to collect original data of an inspection object through multi-modal sensors mounted on an unmanned aerial vehicle platform, the multi-modal sensors including at least a visual sensor, a laser radar, a dual-polarization radar, and an inertial measurement unit; the original data is preprocessed, including noise filtering by using a sliding window filter and dynamic threshold detection, and abnormal data is marked to obtain preprocessed multi-source data; the preprocessed multi-source data is subjected to space-time alignment and synchronization processing to construct a multi-dimensional fusion data set containing environmental features, inspection object state, ice layer thickness, and flight state information; A computing module is configured to input the multi-dimensional fusion data set into a deep neural network for feature extraction to obtain extracted features; The extracted features are fused to obtain fused deep features; Based on the fused deep features, a classification and regression network is used to obtain the recognition result, state evaluation result, and icing monitoring of the inspection object; A processing module is configured to transmit the recognition and evaluation results to a ground station through an anti-interference communication link for real-time monitoring, decision support, and deicing decision.

6. A drone, characterized in that, It comprises: One or more processors; A storage device is configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program which, when executed by a processor, implements the method of any one of claims 1 to 4.

Images