A multi-dimensional information fusion method and system for UAV based on acoustic, optical and electrical composite detection

Abstract

The present application provides a multi-dimensional information fusion method and system for drones based on acoustic, optical and electrical composite detection. Specifically, the present application generates acoustic, optical and electrical characteristic streams by collecting acoustic frequency band signals, visible light image sequences and electromagnetic field intensity change data. Based on the time deviation between the low-frequency vibration mode and the transient electromagnetic pulse, the image distortion area in the optical characteristic stream is calibrated in multiple channels dynamically, and the spatiotemporal coordinate parameters of the acoustic, optical and electrical characteristic streams are corrected to generate a calibration characteristic stream. The calibration characteristic stream is subjected to anti-interference fusion processing, and multi-dimensional coupling features are extracted. The multi-dimensional coupling features are input into the drone flight path planning model to generate a three-dimensional spatial situation model. The present application realizes the physical spatiotemporal consistency alignment and noise suppression of multimodal data, significantly improving the obstacle positioning accuracy and the real-time performance of dynamic trajectory prediction.

Description

Technical Field

[0001] The present application relates to the technical field of multi-dimensional information fusion of unmanned aerial vehicles (UAVs), and in particular to a method and system for multi-dimensional information fusion of UAVs based on acoustic, optical and electrical composite detection. Background Art

[0002] When drones perform complex tasks like inspecting high-voltage transmission lines and conducting search and rescue operations at disaster sites, they must detect both dynamic and static obstacles in real time while overcoming environmental interference, such as strong electromagnetic interference and sudden changes in illumination. These scenarios require multi-source sensor data to possess high-precision spatiotemporal synchronization and anti-interference fusion capabilities to ensure accurate obstacle location and real-time path planning in harsh weather or complex electromagnetic environments.

[0003] To meet these needs, existing technologies primarily utilize a vision and LiDAR fusion solution. This solution uses binocular cameras to acquire environmental depth information, combines this with LiDAR point cloud data, and uses deep learning models to extract obstacle geometry. This is then combined with attitude data from an inertial measurement unit to generate a 3D environmental map. This solution uses LiDAR to compensate for the limitations of vision in low-light environments, attempting to achieve complementary fusion of multimodal data.

[0004] However, this solution suffers from significant technical flaws. In low-visibility environments such as rain, fog, and dust, binocular vision fails to model the environment due to missing feature points, while lidar's ability to detect non-reflective obstacles (such as transparent plastic or dark objects) is severely insufficient, resulting in frequent missed detections. Furthermore, in areas of strong electromagnetic interference, such as high-voltage transmission lines, lidar point cloud data is susceptible to magnetic field interference and distortion, significantly reducing obstacle location accuracy. More critically, the difference in sampling rates between vision and lidar leads to cumulative timeline errors in the prediction of dynamic obstacle trajectories, resulting in significant delays in path planning responses and making it difficult to meet the real-time requirements of emergency obstacle avoidance. Summary of the Invention

[0005] The present application provides a multi-dimensional information fusion method and system for unmanned aerial vehicles based on acoustic, optical and electrical composite detection, which is used to solve the problems of poor environmental adaptability, insufficient spatiotemporal synchronization accuracy and weak anti-interference ability in the existing technology.

[0006] In the first aspect, the present application provides a method for multi-dimensional information fusion of UAVs based on acoustic, optical and electrical composite detection, comprising:

[0007] Synchronously collect acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data as the drone moves in the target area;

[0008] Synchronously correlating the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow;

[0009] Based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, a multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibrated characteristic flow;

[0010] According to the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data in the target area, the calibration feature flow is subjected to anti-interference fusion processing, and the multi-dimensional coupling features in the fusion result are extracted;

[0011] The multi-dimensional coupling features are input into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area.

[0012] Optionally, based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibrated characteristic flow, including:

[0013] Extracting a low-frequency vibration peak time series synchronized with the UAV rotor vibration from the acoustic characteristic flow, and separating a transient pulse front time series triggered by electromagnetic reflections from obstacles from the electrical characteristic flow, calculating the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series, and generating an acoustic-electrical synergy time deviation value;

[0014] constructing a dynamic calibration matrix based on the acoustic-electrical synergy time deviation value, and performing a frame-by-frame convolution operation on the dynamic calibration matrix and the pixel displacement of the image distortion area in the optical feature stream to generate optical distortion compensation parameters;

[0015] According to the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter, a time-domain stretching process is performed on the low-frequency vibration peak time series in the acoustic characteristic flow to generate time-axis aligned acoustic calibration parameters, and a time-shift compensation process is performed on the transient pulse front time series in the electrical characteristic flow to generate time-axis aligned electrical calibration parameters;

[0016] Inputting the acoustic calibration parameters, electrical calibration parameters, and optical distortion compensation parameters into a multi-channel coupler, and calculating the spatial projection offset of the acoustic characteristic flow, the optical axis pointing compensation of the optical characteristic flow, and the vertical gradient compensation of the electrical characteristic flow based on the real-time attitude angular velocity and geomagnetic azimuth deviation of the UAV;

[0017] Superimposing the spatial projection offset onto the spatiotemporal coordinates of the acoustic feature flow to generate a calibrated acoustic feature flow, superimposing the optical axis pointing compensation onto the image distortion region of the optical feature flow to generate a calibrated optical feature flow, and superimposing the vertical gradient compensation onto the spatiotemporal coordinates of the electrical feature flow to generate a calibrated electrical feature flow;

[0018] The time and space coordinate parameters of the calibrated acoustic characteristic flow, optical characteristic flow and electrical characteristic flow are normalized and fused to generate a calibrated characteristic flow.

[0019] Optionally, based on the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter, time-domain stretching processing is performed on the low-frequency vibration peak time series in the acoustic characteristic flow to generate time-axis aligned acoustic calibration parameters, including:

[0020] Calculating a time domain stretch factor of a low-frequency vibration peak time series in the acoustic characteristic flow based on a product value of the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter;

[0021] performing non-uniform interpolation processing on the low-frequency vibration peak time series, dynamically scaling the interval time of the vibration peaks according to the time domain stretching factor, and generating an interpolated vibration peak time series;

[0022] Extracting phase differences between adjacent vibration peaks from the interpolated vibration peak time series, and generating phase correction parameters for the vibration peaks based on a multiplication relationship between the phase differences and the optical distortion compensation parameters;

[0023] Performing a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change of the pitch angle of the UAV, and generate a phase-aligned vibration peak time series;

[0024] Based on the phase-aligned vibration peak time series and the original spectrum distribution of the acoustic characteristic flow, time-axis aligned acoustic calibration parameters are calculated, wherein the acoustic calibration parameters include the time domain remapping coefficient and the phase compensation factor of the vibration peak.

[0025] Optionally, performing time shift compensation processing on the transient pulse front time series in the electrical characteristic flow to generate time axis aligned electrical calibration parameters includes:

[0026] Performing a sliding window detection on the transient pulse front time series, extracting the interval time difference between adjacent pulse fronts, and performing a proportional operation on the interval time difference and the acoustic-electrical coordination time deviation value to generate a dynamic time shift factor;

[0027] constructing a time shift compensation function based on the dynamic time shift factor, performing non-uniform interpolation processing on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series;

[0028] Extracting a pulse front steepness parameter from the interpolated pulse front time series, and performing a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate pulse waveform distortion caused by multipath reflection of metal obstacles, thereby generating a pulse time series with a corrected waveform;

[0029] Calculating time-axis aligned electrical calibration parameters based on the waveform-corrected pulse time sequence and the original intensity distribution of the electrical characteristic flow, wherein the electrical calibration parameters include a time-shift remapping coefficient of the pulse leading edge and a waveform compensation factor;

[0030] Optionally, the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data are synchronously associated with the current position and attitude parameters of the UAV to generate an acoustic feature flow, an optical feature flow, and an electrical feature flow, including:

[0031] Performing time-frequency analysis on the acoustic frequency band signal to extract a low-frequency component that matches the vibration frequency of the drone rotor, and performing spatial projection conversion on the low-frequency component and the real-time three-dimensional coordinates of the drone to generate vibration characteristic parameters with position markers in the acoustic data stream;

[0032] performing inter-frame attitude calculation on the visible light image sequence, calculating the optical axis pointing vector of each frame in the visible light image sequence based on the pitch and yaw angle parameters of the drone gimbal, mapping the optical axis pointing vector to the flight altitude of the drone in a spherical coordinate system, and generating a perspective compensation parameter carrying an attitude tag in the optical data stream;

[0033] Performing altitude attenuation compensation on the electromagnetic field intensity change data, dynamically adjusting the electromagnetic field gradient threshold according to the relative distance between the UAV and ground obstacles, and generating a field intensity correction factor carrying an altitude marker in the electrical data stream;

[0034] Inputting the vibration characteristic parameters, the viewing angle compensation parameters, and the field strength correction factor into a pre-trained spatiotemporal coupling model, and calculating the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream based on the motion acceleration and angular velocity parameters of the UAV;

[0035] Performing Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate an acoustic intermediate data stream aligned with the time axis; performing perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate an optical intermediate data stream aligned with the spatial axis;

[0036] Based on the correlation between the field strength correction factor and the altitude change rate of the UAV, vertical field strength gradient compensation is performed on the electrical data stream to generate an electrical intermediate data stream that is temporally and spatially matched with the acoustic intermediate data stream and the optical intermediate data stream;

[0037] The acoustic intermediate data stream is bound to the vibration characteristic parameters to generate an acoustic characteristic stream, and the optical intermediate data stream is bound to the viewing angle compensation parameters to generate an optical characteristic stream. At the same time, the electrical intermediate data stream is bound to the field strength correction factor to generate an electrical characteristic stream.

[0038] Optionally, according to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, anti-interference fusion processing is performed on the calibration feature stream, and multi-dimensional coupling features in the fusion result are extracted, including:

[0039] Constructing a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and performing channel-by-channel convolution operations on the acoustic calibration feature flow, the optical calibration feature flow, and the electrical calibration feature flow in the calibration feature flow with the reflection correlation weight matrix to generate acoustic reflection correlation features, optical reflection correlation features, and electromagnetic reflection correlation features;

[0040] Calculating a dynamic attenuation factor for the electromagnetic field intensity change data, and weightedly superimposing the dynamic attenuation factor with the electromagnetic reflection correlation feature to generate an electromagnetic anti-interference feature flow, wherein the dynamic attenuation factor is generated based on the product relationship between the real-time altitude of the UAV and the reflection correlation weight matrix;

[0041] Extracting low-frequency vibration energy distribution parameters from the acoustic reflection correlation feature and performing a cross-modal product operation with image profile gradient parameters from the optical reflection correlation feature to generate a vibration profile coupling coefficient;

[0042] The vibration profile coupling coefficient is matched with the electromagnetic anti-interference characteristic flow in time and space, and the multimodal fusion weight is calculated based on the curvature parameter of the UAV flight trajectory and the geomagnetic azimuth deviation to generate a fusion feature matrix including acoustic vibration energy distribution, optical profile gradient and electromagnetic anti-interference strength;

[0043] The fusion feature matrix is ​​decomposed at multiple scales to extract the obstacle position distribution features in the low-frequency component and the moving target trajectory features in the high-frequency component. The position distribution features and trajectory features are correlated and superimposed in the time domain to generate the multi-dimensional coupling features, wherein the multi-dimensional coupling features include the obstacle spatial topological relationship and the moving target motion vector parameters.

[0044] Optionally, the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data of the UAV as it moves in the target area are collected simultaneously, including:

[0045] Based on the obstacle spatial topological relationship parameters in the multi-dimensional coupling feature, the target area is divided into three-dimensional grids, and a dynamic obstacle distribution model of the terrain surface is generated according to the mapping relationship between the UAV flight altitude and the obstacle reflection coefficient;

[0046] Extracting the moving target motion vector parameters from the multi-dimensional coupling features, combining them with the real-time gradient distribution of the electromagnetic field intensity change data, and performing dynamic probability prediction on the moving target's motion trajectory to generate trajectory prediction parameters including velocity direction and collision probability;

[0047] Performing spatial interpolation processing on the dynamic obstacle distribution model, generating an obstacle height field strength map covering the target area based on the UAV flight speed and gimbal viewing angle parameters, and simultaneously superimposing the trajectory prediction parameters with the obstacle height field strength map in the time domain to generate a dynamic threat field strength map;

[0048] Based on the field intensity gradient distribution in the dynamic threat field intensity map, the potential field navigation algorithm in the UAV flight path planning model is used to calculate the path node sequence between the current position of the UAV and the target point, and generate an initial obstacle avoidance path;

[0049] Performing electromagnetic interference robustness verification on the initial obstacle avoidance path, performing a weighted assessment of the safety of path nodes based on the transient pulse peak intensity in the electromagnetic field intensity change data and the collision probability in the trajectory prediction parameters, and generating an optimal path node sequence including threat avoidance weights;

[0050] The optimal path node sequence is fused with the dynamic threat field strength map in three-dimensional space to generate a three-dimensional spatial situation model covering the target area, wherein the three-dimensional spatial situation model includes obstacle elevation distribution, moving target trajectory prediction and electromagnetic interference hot zone marking.

[0051] In a second aspect, the present application provides a multi-dimensional information fusion system for drones based on acoustic, optical and electrical composite detection, comprising:

[0052] The acquisition module is used to synchronously collect the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data of the drone during its movement in the target area;

[0053] an association module, configured to synchronously associate the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV, respectively, to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow;

[0054] a correction module, configured to perform multi-channel dynamic calibration on the image distortion region in the optical characteristic stream based on the time deviation between the low-frequency vibration mode in the acoustic characteristic stream and the transient electromagnetic pulse in the electrical characteristic stream, and synchronously correct the spatiotemporal coordinate parameters of the acoustic characteristic stream, the optical characteristic stream, and the electrical characteristic stream based on the calibration result to generate a calibrated characteristic stream;

[0055] A fusion module is used to perform anti-interference fusion processing on the calibration feature flow based on the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data in the target area, and extract multi-dimensional coupling features from the fusion result;

[0056] A generation module is used to input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area.

[0057] In a third aspect, an embodiment of the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a multi-dimensional information fusion method for drones based on acoustic, optical and electrical composite detection as described in the first aspect above.

[0058] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements a multi-dimensional information fusion method of a drone based on acoustic, optical and electrical composite detection as described in the first aspect.

[0059] The embodiments of the present application form acoustic, optical and electrical characteristic flows with physical consistency through the collection and space-time correlation of multi-modal data of sound, light and electricity; dynamically calibrate the optical distortion area based on the time deviation of acoustic low-frequency vibration and electromagnetic transient pulse, and significantly improve the space-time alignment accuracy of multi-source data in complex electromagnetic interference environment by synchronously correcting the space-time coordinate parameters of the three channels; combine the reflection coefficient of surface obstacles with the mapping relationship of electromagnetic field intensity for anti-interference fusion processing, and extract the multi-dimensional coupling features to effectively distinguish real obstacles from noise interference. The dynamic three-dimensional spatial situation model finally generated realizes centimeter-level obstacle positioning and real-time obstacle avoidance in scenarios such as high-voltage transmission line inspection and disaster search and rescue, solving the core problems of poor environmental adaptability, insufficient space-time synchronization accuracy and weak anti-interference ability in the existing technology.

[0060] Furthermore, a dynamic calibration matrix is ​​constructed by extracting the acoustic-electrical collaborative time deviation value to achieve frame-by-frame pixel-level compensation for the optical distortion area. Combined with acoustic time domain stretching and electrical time shift compensation technology, the time-space axis offset of multi-sensor data caused by sudden changes in the drone's attitude is eliminated; the multi-channel spatial compensation amount calculated based on attitude angular velocity and geomagnetic azimuth deviation ensures the precise alignment of the spatial projection of the acoustic characteristic flow, the optical axis pointing of the optical characteristic flow, and the vertical gradient of the electrical characteristic flow in physical space; the calibration feature flow generated by the normalized fusion of time-space coordinates provides a highly consistent data foundation for subsequent anti-interference fusion, especially in complex terrain and strong electromagnetic interference scenarios. The collaborative calibration accuracy of multimodal data and the real-time prediction of dynamic obstacle trajectories are significantly better than traditional single-modal calibration methods, effectively improving the detection reliability in complex environments.

[0061] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 A flowchart of a multi-dimensional information fusion method for UAV based on acoustic, optical and electrical composite detection provided by the present application is shown;

[0064] Figure 2 The present invention provides a schematic diagram of a multi-dimensional information fusion system for UAVs based on acoustic, optical and electrical composite detection.

[0065] Figure 3 A schematic structural diagram of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0067] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0068] Current drone technology for complex scenarios like high-voltage power line inspections and disaster rescue relies primarily on a fusion of vision and lidar. However, these technologies present significant technical bottlenecks. In low-visibility environments like rain, fog, and dust, binocular vision fails to accurately model the environment due to missing feature points, while lidar often misses non-reflective obstacles like transparent plastic and dark objects. In environments with strong electromagnetic interference, lidar point cloud data is susceptible to magnetic field distortion, significantly reducing obstacle location accuracy. More critically, the difference in sampling rates between vision and lidar leads to a mismatch in the spatiotemporal references of multi-source data, resulting in cumulative errors in dynamic obstacle trajectory predictions and significant delays in path planning responses, making it difficult to meet real-time obstacle avoidance requirements. These issues severely limit the reliability and adaptability of existing technologies in complex electromagnetic interference and dynamic environments.

[0069] In response to the above-mentioned defects, this application proposes a multi-dimensional information fusion method for UAVs based on acoustic, optical and electrical composite detection, which breaks through the bottleneck of existing technologies through the collaborative calibration and anti-interference fusion of acoustic, optical and electromagnetic multimodal data. Specifically, by extracting the time deviation between the acoustic low-frequency vibration mode and the electromagnetic transient pulse, a dynamic calibration matrix is ​​constructed to compensate the optical distortion area frame by frame; the spatiotemporal coordinates of the three-channel data are corrected in real time based on the UAV attitude parameters to eliminate the multi-source data offset caused by sudden changes in flight attitude; the anti-interference coupling characteristics are extracted by combining the surface obstacle reflection coefficient and the electromagnetic field gradient correlation mapping. In scenarios such as strong electromagnetic interference and severe weather, this method realizes the physical spatiotemporal consistency alignment and noise suppression of multimodal data, significantly improving the obstacle positioning accuracy and the real-time performance of dynamic trajectory prediction, solving the three core problems of poor environmental adaptability, weak anti-interference ability and insufficient spatiotemporal synchronization in the existing technology, and providing technical support for reliable detection and autonomous obstacle avoidance of UAVs in complex scenarios.

[0070] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0071] Figure 1 The present invention provides a flowchart of a multi-dimensional information fusion method for UAV based on acoustic, optical and electrical composite detection, as shown in FIG. Figure 1 As shown, the method includes:

[0072] Step 101, synchronously collecting acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data of the UAV during its movement in the target area;

[0073] In this step, the acoustic wave frequency band signal refers to the acoustic wave signal of the target area collected by the acoustic sensor onboard the drone, including 200-500Hz low-frequency mechanical vibration noise (such as rotor operation soundprint) and 5-20kHz high-frequency obstacle reflection sound waves (such as metal collision sound); the visible light image sequence refers to the RGB image frames of the target area continuously taken by the drone's gimbal camera, which is used to identify the obstacle outline and surface texture features. It is prone to image distortion (such as perspective deformation and motion blur) due to the influence of flight posture; the electromagnetic field intensity change data refers to the electromagnetic field intensity gradient information collected by the electromagnetic sensor onboard the drone, including transient pulse waveforms caused by metal obstacles or high-voltage transmission lines (such as electromagnetic interference signals with steep fronts).

[0074] In this embodiment, acoustic sensors are first used to collect acoustic signals from the target area at a sampling rate of 20kHz. Adaptive bandpass filters are then used to separate low-frequency mechanical vibration noise from high-frequency obstacle reflections, generating an acoustic frequency band signal containing rotor vibration characteristics and acoustic markers of obstacles. Second, a global shutter camera onboard the drone's gimbal captures a visible light image sequence at a frame rate of 30fps. An optical flow estimation algorithm is used to perform motion compensation on adjacent frames, and a deblurred, stabilized image sequence is generated based on the drone's real-time attitude parameters (pitch and yaw angles). Simultaneously, a three-axis electromagnetic sensor collects electromagnetic field intensity data at a sampling rate of 1kHz. A sliding time window is used to detect transient pulse waveforms, and pulse peak intensity and timestamp information are recorded to generate intensity variation data containing electromagnetic reflection characteristics of obstacles. Finally, the time synchronization module of the drone's flight control system unifies the timestamps of the acoustic frequency band signal, visible light image sequence, and electromagnetic field intensity variation data to the same reference clock, ensuring the spatiotemporal synchronization of the three-channel data.

[0075] For example, during a high-rise building fire, a drone flew to the north side of the fire (34.05°N, 118.25°E, at an altitude of 20 meters). Acoustic sensors collected the 220Hz low-frequency vibrations caused by the burning flames and the 6-8kHz high-frequency sound waves from glass explosions, separating these features using an adaptive bandpass filter. A thermal imaging camera captured the flame outline and smoke diffusion at 30fps, using optical flow to compensate for pixel shifts caused by thermal airflow disturbances. An electromagnetic sensor detected 1.5GHz transient pulses caused by cable shorts, recording the peak intensity and timestamp. The multi-sensor data was aligned by the flight control system's time synchronization module, generating a raw acoustic, optical, and electromagnetic dataset with consistent temporal and spatial references.

[0076] This step generates high-integrity acoustic, optical, and electromagnetic raw data sets through the synchronous acquisition and preprocessing of multimodal sensors. Acoustic frequency band signals provide mechanical vibration and acoustic characteristics of obstacles, visible light image sequences retain target contours and texture information, and electromagnetic field intensity variation data captures the reflective characteristics of metal obstacles. A time synchronization module eliminates temporal deviations in multi-sensor data, providing a physical consistency foundation for subsequent multimodal calibration and fusion. This method significantly improves the signal-to-noise ratio and spatiotemporal alignment accuracy of raw data in scenarios with complex electromagnetic interference, dynamic lighting changes, and sudden changes in drone attitude, ensuring the reliability of subsequent processing steps.

[0077] Step 102: Synchronously correlating the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow;

[0078] In this step, the current position and attitude parameters include the drone's real-time latitude and longitude (GPS / RTK positioning), altitude (barometer or laser ranging), and pitch, roll, and yaw angles (IMU output). The acoustic / optical / electrical feature stream refers to the structured data stream that fuses the sensor raw data with spatiotemporal parameters, including timestamps, geographic coordinates, sensor raw data, and extracted physical features (such as acoustic wave spectrum, image texture, and electromagnetic pulse waveform).

[0079] In this embodiment, first, the acquisition time of acoustic wave, image, and electromagnetic data is synchronized with the reference clock of the UAV flight control system through timestamp alignment. For example, an interpolation algorithm is used to compensate for the sampling interval difference between the acoustic sensor (10kHz sampling rate) and the electromagnetic sensor (1kHz sampling rate) to ensure that the time deviation of all data is less than 1ms. Secondly, the sensor data is bound to the real-time position of the UAV through spatial coordinate mapping: for acoustic wave signals, the spherical projection model is used to calculate the azimuth of the sound source in combination with the altitude and pitch angle of the UAV; for visible light images, the image pixels are mapped to the three-dimensional geographic coordinate system through the geographic coordinate system conversion algorithm (UTM projection) according to the latitude and longitude and yaw angle of the UAV. Space; for electromagnetic field data, spatial interpolation is used to generate an electromagnetic intensity distribution heat map based on the drone's position and roll angle; then, a structured feature stream is generated through feature fusion: Mel-frequency cepstral coefficients (MFCC) are applied to the acoustic frequency band signal to extract spectral features, and merged with the sound source azimuth angle to form an acoustic feature stream; a convolutional neural network (ResNet-18) is used to extract deep semantic features from the image sequence, and the geographic coordinates are superimposed to generate an optical feature stream; the transient pulse energy features of the electromagnetic data are extracted through wavelet packet decomposition, and the electrical feature stream is generated by combining the heat map coordinates; finally, the three types of feature streams are integrated into a multimodal dataset according to a unified spatiotemporal benchmark through data encapsulation for subsequent algorithm calls.

[0080] Continuing with the previous example, based on the raw data from step 101, the acoustic feature stream uses a short-time Fourier transform (STFT) to extract the 220Hz combustion vibration spectrum. Combined with the drone's 3.5° pitch angle and altitude data, the fire origin coordinates are calculated as 34.0503°N, 118.2502°E. The optical feature stream uses a ResNet-50 model to extract the flame outline from the thermal image. This is then overlaid with the drone's 2.8° yaw angle and GPS coordinates to generate a geotagged image feature stream. The electrical feature stream uses a wavelet transform to extract the 1.5GHz pulse leading edge features. After binding these features to the drone's position, the coordinates of the short-circuit point (34.0501°N, 118.2505°E) are output. After the three feature streams are aligned in time and space, the fire is identified as spreading southeastward along the ventilation duct.

[0081] Step 103: Based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, a multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibration characteristic flow;

[0082] First, time deviation analysis is used to extract the timing differences between the low-frequency vibration modes of flame combustion in the acoustic characteristic flow and the transient electromagnetic pulses of cable short circuits in the electrical characteristic flow. For example, the generalized cross-correlation (GCC-PHAT) algorithm is used to calculate the time offset between the two, identifying the causal relationship between structural vibration caused by thermal expansion and electromagnetic interference. Second, multi-channel dynamic calibration is used to correct the image distortion of the optical characteristic flow. Based on the time deviation, a non-rigid image registration algorithm (such as a B-spline-based deformation model) is used to align the flame outline captured by the thermal imaging camera with the smoke-occluded area in the visible light image. At the same time, the optical flow method is used to compensate for the pixel displacement caused by the thermal airflow. Subsequently, the multimodal characteristic flow parameters are updated through spatiotemporal coordinate correction. Based on the calibrated flame position and smoke diffusion model, the drone's altitude and pitch angle are inversely optimized, and the combustion intensity estimate of the acoustic characteristic flow and the coordinates of the short-circuit point of the electrical characteristic flow are synchronously adjusted to ensure the spatial consistency of the multimodal data. Finally, an adaptive fusion engine is used to encapsulate the corrected acoustic, optical, and electrical characteristic flows into a calibrated characteristic flow according to a unified spatiotemporal reference.

[0083] Continuing with the previous example, the acoustic feature stream detected the 180Hz low-frequency vibration of the steel beam caused by thermal deformation, while the electrical feature stream captured the 1.5GHz pulse of the secondary short circuit. Generalized cross-correlation (GCC-PHAT) analysis revealed that the electromagnetic pulse preceded the vibration peak by 80ms, indicating that the electrical fault preceded the structural deformation. Based on this deviation, B-spline non-rigid registration was performed on the exterior wall image distorted by high-temperature deformation in the optical feature stream to restore the true building outline. The steel beam coordinates in the acoustic feature stream were simultaneously corrected to 34.0506°N, 118.2498°E, and the short circuit point coordinates in the electrical feature stream were corrected to 34.0502°N, 118.2495°E. The drone's pitch angle was also optimized to 4.1°. The calibrated feature streams were input into the positioning model, accurately locating the two trapped individuals to the west safety passage on the fifth floor (34.051°N, 118.249°E), and issuing a warning of the risk of ceiling collapse.

[0084] Step 104: performing anti-interference fusion processing on the calibration feature stream according to the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data in the target area, and extracting multi-dimensional coupling features from the fusion result;

[0085] In this step, the surface obstacle reflection coefficient refers to the pre-calibrated reflection ability of obstacles of different materials to electromagnetic waves; anti-interference fusion processing improves the consistency of multimodal data by suppressing the cross-interference of electromagnetic noise (such as cable short-circuit pulses) and optical thermal distortion; multi-dimensional coupling characteristics refer to the cross-modal joint features after the fusion of acoustic combustion intensity, optical flame profile, and electromagnetic material reflection characteristics, such as the flame-metal structure thermal coupling effect.

[0086] In this embodiment, reflection coefficient mapping is first used to establish a correlation between electromagnetic field intensity and surface obstacle material. For example, a random forest model is used to match the peak intensity of an electromagnetic pulse (1.5 GHz) with preset metal / concrete reflection coefficients (0.8 / 0.3), generating a probability distribution map of obstacle material and electromagnetic reflection. Second, cross-interference in multimodal data is suppressed through anti-interference weighted fusion. For the electromagnetic data in the calibration feature stream, a wavelet threshold denoising algorithm is used to filter out random pulses caused by cable shorts. For the optical feature stream, an attention mechanism (such as SENet) is used based on the reflection coefficient mapping results to enhance the flame outline weights in metal obstacle areas, thereby suppressing misjudgment of smoke-obstructed areas. Subsequently, coupled feature extraction is used to integrate cross-modal information: material reflection features are extracted from the denoised electromagnetic data, flame spatial distribution features are extracted from the weighted optical data, and combustion intensity temporal features are extracted from the acoustic data. A graph convolutional network (GCN) is used to model the correlation between these three features, outputting multidimensional coupled features (such as the "metal pipe-flame contact thermal conductivity coefficient").

[0087] Continuing with the previous example, the reflection coefficients of the metal pipe and the concrete wall were preset to 0.8 and 0.3, respectively. The electromagnetic feature flow detected a 92% probability of mapping the 1.5 GHz pulse intensity to metal. The random forest model determined the presence of a metal structure on the west side of the fifth floor. After filtering out interference pulses caused by cable short circuits through wavelet threshold denoising, the optical feature flow used SENet to enhance the flame contour in the metal pipe area and identify the flame spreading along the pipe to the sixth floor. The acoustic feature flow extracted the temporal correlation between the peak value of the combustion intensity and the vibration frequency of the metal structure at 180 Hz. The GCN model integrated these features and output a "metal-flame thermal coupling coefficient" of 0.75, indicating a high risk of thermal deformation of the pipe. Finally, feature encoding was used to locate the trapped personnel to a safe area behind the metal pipe (34.051°N, 118.249°E), and an early warning was issued regarding the time window for the pipe to burst.

[0088] Step 105: input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area;

[0089] In this step, the UAV flight path planning model refers to a dynamic route generation framework based on reinforcement learning (such as the PPO algorithm) and three-dimensional obstacle avoidance constraints (such as thermal radiation threshold and structural stability index); the three-dimensional spatial situation model refers to a visual three-dimensional grid map that integrates fire spread prediction, obstacle distribution (metal pipes, collapsed walls) and risk level (high temperature area, structural deformation area), including risk heat map, safe channel layer and real-time update mechanism.

[0090] In this example, a path planning model is first initialized by loading multi-dimensional coupling features, including the metal-flame thermal coupling coefficient (e.g., 0.75), structural deformation frequency (e.g., 180Hz), and electromagnetic reflection characteristics (e.g., 92% metal probability). This constructs the state space of the fire scene environment and defines the action space as the drone's six degrees of freedom (pitch, yaw, and lift). Next, an initial path is generated through reinforcement learning decision-making: a proximal policy optimization (PPO) algorithm is employed, with the objective functions of minimizing thermal radiation exposure and maximizing search and rescue coverage. In combination with real-time coupling features (e.g., pipeline burst risk window), a reward value for each waypoint is calculated, generating a three-dimensional obstacle avoidance path from the current coordinates (34.051°N, 118.249°E) to the trapped personnel's location. Subsequently, the environment is dynamically updated through three-dimensional situation modeling: based on the path planning results, the Gaussian mixture model (GMM) is used to divide the fire risk levels (high-risk red zone, medium-risk yellow zone, and safe green zone), and the flame contours extracted by the optical characteristic flow and the structural vibration hotspots identified by the acoustic characteristic flow are superimposed to generate a three-dimensional grid map containing risk thermal maps and safe channels.

[0091] Continuing with the previous example, the multi-dimensional coupling features include a 0.75 thermal coupling coefficient for metal pipes, a 180Hz vibration frequency for the fifth-floor west structure, and a 92% probability of a metal obstacle. After loading these features into the path planning model, the PPO algorithm generates a three-dimensional path from the drone's current location (34.051°N, 118.249°E) to the trapped personnel. The path planning model vertically avoids the fifth-floor high-risk red zone (excessive thermal radiation), horizontally weaves through the safe green zone (stable concrete structure), and dynamically avoids windows where metal pipes burst. The three-dimensional situation model uses GMM to divide the high-risk area (the sixth-floor ventilation duct fire), the medium-risk area (metal deformation on the fifth-floor west side), and the safe passage (the southeast fire door). Thermal imaged flame outlines are overlaid with acoustic vibration hotspots to output a real-time grid map. When the electromagnetic sensor detects a new short-circuit pulse, the LSTM predicts the fire's spread to the seventh floor. The model dynamically adjusts the path to a vertical landing followed by a lateral evacuation, and updates the situation model's risk layer.

[0092] This step enables autonomous obstacle avoidance and search and rescue decision-making in complex fire environments through multi-dimensional, feature-driven path planning and 3D situation modeling. Dynamic path planning considers both thermal radiation safety thresholds and structural deformation risks. The 3D situation model visually presents fire evolution and safe passages. A real-time feedback mechanism ensures the timeliness of route and risk predictions, significantly improving the safety and efficiency of fire search and rescue missions.

[0093] In some embodiments, according to step 103, based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, a multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibrated characteristic flow, including:

[0094] Step 201: extracting a low-frequency vibration peak time series synchronized with the UAV rotor vibration from the acoustic characteristic flow, and separating a transient pulse front time series triggered by electromagnetic reflections from obstacles from the electrical characteristic flow. Calculating the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series to generate an acoustic-electrical synergy time deviation value.

[0095] In this step, the low-frequency vibration peak time series refers to the set of periodic vibration waveform peak time points extracted from the acoustic characteristic flow and synchronized with the UAV rotor speed (such as 200Hz), which represents the vibration interference of the body itself; the transient pulse front time series refers to the set of rising edge trigger time points of the electromagnetic reflection event of the obstacle (such as the reflection of a 1.5GHz pulse by a metal pipe) separated from the electrical characteristic flow; the average phase difference quantifies the timing deviation of the two types of signals by statistically averaging the time offset between the acoustic vibration peak and the electromagnetic pulse front (such as 15ms); the acoustic-electrical synergy time deviation value is used to calibrate the benchmark parameters of the multimodal data time axis (such as 15ms).

[0096] In this embodiment, first, a peak detection algorithm (such as local maximum screening) is used to extract the low-frequency peak time series of the rotor vibration from the acoustic characteristic flow. For example, the peak time point appearing every 5 ms in the vibration waveform is detected with a period of 200 Hz. Secondly, a threshold triggering method is used to separate the transient pulse front time series from the electrical characteristic flow. For example, 30% of the 1.5 GHz pulse amplitude is set as the threshold, and the rising edge time point exceeding the threshold is recorded. Subsequently, the phase difference between the two types of time series is calculated by the cross-correlation function. For example, the time offset average of the acoustic peak and the electromagnetic pulse front is statistically calculated within a 10-second time window to generate an acoustic-electrical coordination time deviation value (such as 15 ms).

[0097] Step 202: constructing a dynamic calibration matrix based on the acoustic-electrical synergy time deviation value, performing a frame-by-frame convolution operation on the dynamic calibration matrix and the pixel displacement of the image distortion area in the optical feature stream to generate optical distortion compensation parameters;

[0098] The dynamic calibration matrix refers to a two-dimensional weight matrix (such as a 5×5 convolution kernel) constructed based on the acoustic-electrical synergy time deviation value (such as 15ms), which is used to characterize the compensation weight distribution of the optical distortion area (such as image blur caused by smoke refraction) in the time-space dimension; the pixel displacement refers to the pixel position offset (such as a horizontal offset of 3 pixels and a vertical offset of 2 pixels) between adjacent frames in the optical feature flow due to drone posture jitter or environmental interference (such as thermal airflow disturbance); the optical distortion compensation parameter is a correction coefficient generated by convolution operation (such as a horizontal compensation coefficient of 0.8 and a vertical compensation coefficient of 0.6), which is used to restore the geometric deformation of the image.

[0099] In this embodiment, first, the acoustic-electrical synergy time deviation value (15ms) is converted into the convolution kernel weight of the dynamic calibration matrix through a matrix mapping algorithm. For example, a 5×5 matrix is ​​constructed with a center weight of 0.8 (reflecting the compensation strength of the time deviation on the center pixel), and the edge weight decays according to a Gaussian distribution. Secondly, the optical flow method (such as the Lucas-Kanade algorithm) is used to calculate the pixel displacement of the image distortion area in the optical feature flow. For example, the horizontal displacement of adjacent frame pixels in the smoke-occluded area is 3 pixels and the vertical displacement is 2 pixels. Finally, the dynamic calibration matrix and the pixel displacement are weightedly fused through a frame-by-frame convolution operation to generate optical distortion compensation parameters (such as a horizontal compensation coefficient of 0.8) for subsequent image geometric correction.

[0100] Step 203: Based on the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter, a time-domain stretching process is performed on the low-frequency vibration peak time series in the acoustic characteristic flow to generate time-axis aligned acoustic calibration parameters. At the same time, a time-shift compensation process is performed on the transient pulse front time series in the electrical characteristic flow to generate time-axis aligned electrical calibration parameters.

[0101] In this step, the acoustic-electrical synergistic time deviation value refers to the average time deviation between the acoustic vibration and the electromagnetic reflection event calculated in step 201, which is used to quantify the timing difference between the two types of signals. The optical distortion compensation parameter refers to the compensation coefficient generated in step 202 for correcting the geometric deformation of the image. Time domain stretching processing refers to linear interpolation adjustment of the time axis of the acoustic feature flow based on the product relationship between the deviation value and the compensation parameter to eliminate timing jitter. Time shift compensation processing refers to the overall translation of the time series of the electrical feature flow to align the time reference after optical distortion compensation. The acoustic calibration parameters and electrical calibration parameters are the acoustic and electromagnetic signals with aligned time axes after time domain adjustment.

[0102] In this embodiment, the acoustic-electrical synergy time deviation value is first multiplied by the optical distortion compensation parameter to obtain a time domain adjustment value for subsequent calibration. A linear interpolation algorithm is used to perform time domain stretching on the low-frequency vibration peak time series in the acoustic characteristic flow. For example, the length of the signal time axis is expanded or compressed according to the adjustment amount to synchronize the acoustic signal with the time reference of the optical data. At the same time, the transient pulse front time series of the electrical characteristic flow is overall time-shifted, for example, each pulse trigger time point is shifted by the adjustment amount to match the calibrated time axis. Finally, the acoustic calibration parameters and electrical calibration parameters with time axis alignment are output.

[0103] Step 204: Input the acoustic calibration parameters, electrical calibration parameters, and optical distortion compensation parameters into a multi-channel coupler, and calculate the spatial projection offset of the acoustic characteristic flow, the optical axis pointing compensation of the optical characteristic flow, and the vertical gradient compensation of the electrical characteristic flow based on the real-time attitude angular velocity and geomagnetic azimuth deviation of the UAV;

[0104] In this step, the spatial projection offset is the projection error of the sound source coordinates in the acoustic feature flow in three dimensions due to the drone's real-time angular velocity, and needs to be corrected using the attitude parameters. The optical axis pointing compensation is a correction parameter for the center offset of the optical feature flow image caused by the drone's geomagnetic orientation deviation, used to restore the target's true position. The vertical gradient compensation is a correction factor for the vertical distribution error of the electromagnetic field intensity caused by the drone's altitude change, used to improve the detection reliability of the electromagnetic reflection signature.

[0105] In this embodiment, acoustic calibration parameters, electrical calibration parameters, and optical distortion compensation parameters are first loaded via a multi-channel coupler. Next, the spatial projection offset of the acoustic characteristic flow is calculated using a spatial coordinate transformation model based on the drone's real-time angular velocity. Simultaneously, the optical axis pointing compensation of the optical characteristic flow is calculated using a geometric projection model based on the geomagnetic azimuth deviation. Finally, the vertical gradient compensation of the electrical characteristic flow is calculated using an electromagnetic field distribution model based on the drone's altitude change rate.

[0106] Step 205: superimpose the spatial projection offset onto the spatiotemporal coordinates of the acoustic feature flow to generate a calibrated acoustic feature flow, superimpose the optical axis pointing compensation onto the image distortion region of the optical feature flow to generate a calibrated optical feature flow, and superimpose the vertical gradient compensation onto the spatiotemporal coordinates of the electrical feature flow to generate a calibrated electrical feature flow.

[0107] In this step, the calibrated acoustic feature stream refers to the corrected data stream generated by superimposing the spatial projection offset onto the spatiotemporal coordinates of the original acoustic feature stream. This is used to eliminate sound source localization errors caused by changes in the drone's attitude. The calibrated optical feature stream refers to the geometrically corrected image stream generated by superimposing the optical axis pointing compensation onto the image distortion area. This is used to restore target deformation caused by drone orientation deviation or thermal airflow disturbances. The calibrated electrical feature stream refers to the corrected data stream generated by superimposing the vertical gradient compensation onto the spatiotemporal coordinates of the electromagnetic field. This is used to compensate for deviations in electromagnetic field intensity distribution caused by altitude changes.

[0108] In this embodiment, first, based on the spatial projection offset, the three-dimensional geographic coordinates of the sound source in the acoustic characteristic flow are adjusted through the coordinate transformation model. For example, the original sound source position is translated according to the offset caused by the pitch angular velocity, so that the positioning result is consistent with the real spatial position; secondly, based on the optical axis pointing compensation amount, a bilinear interpolation algorithm is used to perform pixel-level geometric correction on the distorted area of ​​the optical characteristic flow. For example, the image center offset area caused by the yaw angle deviation is realigned to the actual target position; finally, based on the vertical gradient compensation amount, the spatiotemporal coordinates of the electromagnetic pulse in the electrical characteristic flow are corrected through the field strength distribution model. For example, the electromagnetic field strength is gradient compensated according to the lifting and lowering speed, so as to improve the detection accuracy of the reflection signal of the metal obstacle.

[0109] Step 206, normalizing and fusing the time and space coordinate parameters of the calibrated acoustic characteristic flow, optical characteristic flow, and electrical characteristic flow to generate a calibrated characteristic flow;

[0110] In this step, normalized fusion refers to the multimodal data integration after converting the spatiotemporal coordinate parameters of the calibrated acoustic, optical and electrical feature streams into a unified dimension and eliminating the magnitude differences; the calibrated feature stream is a standardized data stream generated by fusion with consistent spatiotemporal references and associated physical features, which contains collaborative information such as sound source positioning coordinates, flame contour images and electromagnetic obstacle mapping.

[0111] In this embodiment, first, the sound source coordinates of the acoustic feature flow, the image geographic tags of the optical feature flow, and the electromagnetic target coordinates of the electrical feature flow are unified into the UTM (Universal Transverse Mercator) coordinate system through geographic coordinate system conversion, thereby eliminating the geographic reference differences of data from different sensors. Secondly, the acoustic vibration intensity, optical pixel brightness, and electromagnetic field strength are normalized using the Z-score normalization method, so that the numerical range of the multimodal parameters is unified into the [-1,1] interval. Finally, the normalized acoustic, optical, and electrical features are superimposed according to preset weights through a weighted fusion algorithm to generate a calibration feature flow that is synchronized in time and space and has comparable physical quantities.

[0112] In some embodiments, according to step 203, time-domain stretching is performed on the low-frequency vibration peak time series in the acoustic characteristic flow based on the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter to generate time-axis aligned acoustic calibration parameters, including:

[0113] Step 301: Calculate the time domain stretch factor of the low-frequency vibration peak time series in the acoustic characteristic flow based on the product value of the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter;

[0114] In this embodiment, the acoustic-electrical synergy time deviation value is first multiplied by the optical distortion compensation parameter to obtain a time domain stretching factor; secondly, the low-frequency vibration peak time series in the acoustic feature flow is subjected to time domain stretching processing, for example, a linear interpolation algorithm is used to expand or compress the time axis length according to the stretching factor, so that the time reference of the acoustic signal is synchronized with the image time axis after optical distortion compensation; finally, the adjusted acoustic feature flow time series is output as a calibration parameter for subsequent multimodal data fusion.

[0115] Step 302: performing non-uniform interpolation processing on the low-frequency vibration peak time series, dynamically scaling the interval time of the vibration peaks according to the time domain stretching factor, and generating an interpolated vibration peak time series;

[0116] In this step, non-uniform interpolation processing refers to dynamically adjusting the time interval of the low-frequency vibration peak time series in the acoustic characteristic flow according to the time domain stretching factor, rather than fixed-step interpolation; the time domain stretching factor is the product of the acoustic-electrical synergy time deviation value calculated in step 301 and the optical distortion compensation parameter, which is used to determine the scaling ratio of different time periods; dynamic scaling refers to nonlinear adjustment of the vibration peak interval time according to the stretching factor, such as expanding or compressing the interval of a specific time period; the interpolated vibration peak time series refers to a new time series generated by non-uniform interpolation processing, whose time axis is synchronized with the optical and electromagnetic data.

[0117] In this embodiment, a time domain stretch factor is first obtained, which reflects the time proportion of the acoustic signal that needs to be stretched or compressed. Secondly, the original low-frequency vibration peak time series is divided into multiple time periods according to the stretch factor, and the corresponding scaling ratio is calculated for each time period. Subsequently, a linear interpolation algorithm is used to perform non-uniform adjustment on the peak interval time within each time period, for example, expanding the time interval corresponding to a high stretch factor and compressing the time interval corresponding to a low stretch factor. Finally, an interpolated vibration peak time series is generated, and its time axis is aligned with the image sequence after optical distortion compensation and the electromagnetic pulse front time series.

[0118] Step 303: extracting the phase difference between adjacent vibration peaks from the interpolated vibration peak time series, and generating a phase correction parameter for the vibration peak based on the product relationship between the phase difference and the optical distortion compensation parameter;

[0119] In this step, the phase difference between adjacent vibration peaks refers to the difference in time intervals between two consecutive peaks in the interpolated vibration peak time series, which is used to characterize the local timing fluctuations of the acoustic signal; the optical distortion compensation parameter refers to the image deformation correction coefficient generated in step 202, which is used to quantify the time axis compensation requirements of the optical data; the phase correction parameter of the vibration peak is a local time adjustment coefficient generated by multiplying the phase difference with the optical distortion compensation parameter, which is used to eliminate the phase distortion of the acoustic signal caused by the drone movement.

[0120] In this embodiment, the phase difference between adjacent peaks is first extracted from the interpolated vibration peak time series, for example, the time interval difference between each pair of peaks is calculated through a differential operation; secondly, each phase difference is multiplied by the optical distortion compensation parameter to obtain the phase correction parameter of the corresponding time period; finally, a set of phase correction parameters that match the length of the vibration peak time series is generated for subsequent convolution operations.

[0121] Step 304: performing a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change in the pitch angle of the drone, and generate a phase-aligned vibration peak time series;

[0122] In this step, the convolution operation refers to using the phase correction parameter as the convolution kernel weight and performing a sliding window multiplication and accumulation operation on the interpolated vibration peak time series; phase distortion refers to the nonlinear offset of the acoustic signal time axis caused by the sudden change of the UAV pitch angle; the phase-aligned vibration peak time series refers to the acoustic characteristic stream generated after the distortion is eliminated by convolution, and the time axis is strictly synchronized with the optical and electromagnetic data.

[0123] In this embodiment, a convolution kernel with a phase correction parameter as a weight is first constructed, for example, a Gaussian kernel function is used to smooth the correction parameter; secondly, a convolution operation is performed on the convolution kernel and the interpolated vibration peak time series, for example, high-frequency jitter on the time axis is eliminated by sliding window multiplication and accumulation; finally, a phase-aligned vibration peak time series is output, and its time interval is completely synchronized with the image sequence after optical distortion compensation and the electromagnetic pulse event.

[0124] Step 305: Calculate time-axis aligned acoustic calibration parameters based on the phase-aligned vibration peak time series and the original spectrum distribution of the acoustic feature flow, wherein the acoustic calibration parameters include a time-domain remapping coefficient and a phase compensation factor of the vibration peak.

[0125] In this step, the original spectrum distribution refers to the frequency-energy distribution generated by Fourier transform of the low-frequency vibration signal in the acoustic characteristic flow, which reflects the frequency component characteristics of the vibration signal; the time domain remapping coefficient refers to the time axis scaling ratio calculated based on the correlation between the vibration peak time series after time axis alignment and the original spectrum, which is used to correct the time reference of the acoustic signal; the phase compensation factor refers to the correction parameter generated by comparing the phase difference between the aligned time series and the original spectrum, which is used to eliminate the phase offset of the frequency component.

[0126] In this embodiment, a short-time Fourier transform (STFT) is first performed on the phase-aligned vibration peak time series to extract its spectral distribution; secondly, a correlation analysis is performed between the spectral distribution of the original acoustic feature flow and the aligned spectrum, and the frequency energy matching degree between the two is calculated; then, a time domain remapping coefficient is generated based on the matching degree (for example, by fitting the time axis scaling ratio through the least squares method), and the phase difference between the original spectrum and the aligned spectrum is compared to generate a phase compensation factor; finally, the time domain remapping coefficient and the phase compensation factor are combined into the acoustic calibration parameter.

[0127] In some embodiments, according to step 203, performing time shift compensation processing on the transient pulse front time series in the electrical characteristic flow to generate time axis aligned electrical calibration parameters includes:

[0128] Step 401: Perform sliding window detection on the transient pulse front time series, extract the interval time difference between adjacent pulse fronts, and perform proportional operation on the interval time difference and the acoustic-electrical coordination time deviation value to generate a dynamic time shift factor;

[0129] In this step, sliding window detection refers to dividing the transient pulse front time series into segments of fixed time lengths and analyzing the time intervals between adjacent pulses segment by segment; the interval time difference refers to the time difference between adjacent pulse fronts in the same window, which is used to characterize the local timing fluctuations of electromagnetic reflection events; the dynamic time shift factor is an adjustment coefficient obtained by performing a proportional operation (such as division or multiplication) on the interval time difference and the acoustic-electrical coordination time deviation value generated in step 201, which is used to quantify the pulse timing compensation requirements in different time periods.

[0130] In this embodiment, the transient pulse front time series is first divided into multiple sliding windows (for example, each window contains three pulses), and the interval time difference between adjacent pulses in each window is calculated; secondly, each interval time difference is proportionally calculated with the acoustic-electrical coordination time deviation value, for example, the dynamic time shift factor is calculated by division, which represents the ratio by which the pulse interval in the current time period needs to be scaled; finally, a set of dynamic time shift factors corresponding to the time window is output for subsequent interpolation processing.

[0131] Step 402: constructing a time shift compensation function based on the dynamic time shift factor, performing non-uniform interpolation processing on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series;

[0132] In this step, the time-shift compensation function refers to a time-axis adjustment function (such as a linear or nonlinear mapping relationship) constructed based on a dynamic time-shift factor, which is used to map the original pulse front time series to the calibrated time axis; non-uniform interpolation processing refers to the dynamic scaling and interpolation of the pulse time points according to the time-shift compensation function, such as expanding or compressing the intervals of different time periods; the interpolated pulse front time series refers to the electromagnetic characteristic flow generated by non-uniform interpolation, with the time axis synchronized with the acoustic and optical data.

[0133] In this embodiment, a piecewise linear time-shift compensation function is first constructed based on the dynamic time-shift factor. For example, a high-order interpolation function is used for the time period corresponding to the high dynamic time-shift factor. Secondly, the time-shift compensation function and the transient pulse front time series are subjected to non-uniform interpolation processing. For example, a cubic spline interpolation algorithm is used to dynamically adjust the pulse time point according to the compensation function. Finally, the interpolated pulse front time series is generated.

[0134] Step 403: extracting a pulse front steepness parameter from the interpolated pulse front time series, and performing a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate pulse waveform distortion caused by multipath reflection of metal obstacles, thereby generating a pulse time series with a corrected waveform.

[0135] In this step, the pulse front steepness parameter refers to the slope of the pulse rising edge in the interpolated pulse front time series, which is used to characterize the degree of distortion of the electromagnetic reflection waveform; the dynamic time shift factor refers to the pulse interval scaling coefficient generated in step 401, which is used to quantify the timing compensation requirements in different time periods; multipath reflection refers to the waveform superposition distortion caused by multiple reflections of electromagnetic waves by metal obstacles; the waveform-corrected pulse time series refers to the electromagnetic characteristic flow with clear waveform characteristics generated by eliminating multipath reflection distortion.

[0136] In this embodiment, the steepness parameter of each pulse rising edge in the interpolated pulse front time series is first calculated through derivative operation. Secondly, the steepness parameter and the dynamic time shift factor are used to construct the convolution kernel weight. Subsequently, the pulse waveform is filtered through convolution operation to suppress high-frequency distortion components with abnormal steepness. Finally, a waveform-corrected pulse time series is output, whose waveform front steepness is synchronized with the time axis of the acoustic and optical data.

[0137] Step 404: Calculate time-axis aligned electrical calibration parameters based on the waveform-corrected pulse time sequence and the original intensity distribution of the electrical characteristic flow, wherein the electrical calibration parameters include a time-shift remapping coefficient of the pulse leading edge and a waveform compensation factor;

[0138] In this step, the original intensity distribution refers to the amplitude-time distribution of the electromagnetic pulse in the electrical characteristic flow, which reflects the electromagnetic energy characteristics reflected by the obstacle; the time-shift remapping coefficient refers to the timing scaling ratio calculated based on the time axis offset between the pulse time series after waveform correction and the original intensity distribution; the waveform compensation factor refers to the amplitude correction parameter generated by comparing the spectral difference between the corrected waveform and the original waveform, which is used to eliminate the energy attenuation caused by multipath reflection.

[0139] In this embodiment, first, a short-time energy integration is performed on the waveform-corrected pulse time series to extract its intensity distribution; second, a correlation analysis is performed between the intensity distribution of the original electrical characteristic flow and the corrected distribution, the time axis offset is calculated, and a time-shift remapping coefficient is generated; then, a waveform compensation factor is generated through spectrum comparison analysis; finally, the time-shift remapping coefficient and the waveform compensation factor are combined into electrical calibration parameters.

[0140] In some embodiments, according to step 102, the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data are synchronously associated with the current position and attitude parameters of the drone to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream, including:

[0141] Step 501: Perform time-frequency analysis on the acoustic frequency band signal to extract a low-frequency component that matches the vibration frequency of the drone rotor, and perform spatial projection conversion on the low-frequency component and the real-time three-dimensional coordinates of the drone to generate vibration characteristic parameters with position markers in the acoustic data stream.

[0142] In this step, time-frequency analysis refers to extracting the frequency-time distribution characteristics of the acoustic frequency band signal through short-time Fourier transform or wavelet transform; the low-frequency component refers to the acoustic signal component that matches the vibration frequency range of the drone rotor, such as the mechanical vibration characteristics in the frequency range of one hundred Hz to five hundred Hz; spatial projection conversion refers to combining the physical characteristics of the acoustic signal (such as vibration energy) with the real-time three-dimensional coordinates (latitude, longitude, and altitude) of the drone to generate vibration characteristic parameters with geographic tags in the acoustic data stream, such as mapping the vibration energy peak to within ten meters of the drone's current position.

[0143] In this embodiment, a short-time Fourier transform is first performed on the acoustic frequency band signal to extract the low-frequency energy distribution from one hundred Hz to five hundred Hz. Secondly, the real-time three-dimensional coordinates of the UAV are obtained through the UAV flight control system, and the acoustic vibration energy peak is mapped to the geographic space using a spherical projection model to generate vibration characteristic parameters with location markers for use in multimodal fusion analysis.

[0144] Step 502: performing inter-frame attitude calculation on the visible light image sequence, calculating the optical axis pointing vector of each frame in the visible light image sequence based on the pitch and yaw angle parameters of the drone gimbal, mapping the optical axis pointing vector to the flight altitude of the drone in a spherical coordinate system, and generating a perspective compensation parameter carrying an attitude tag in the optical data stream;

[0145] In this step, inter-frame attitude solution refers to calculating the relative motion parameters of the drone by matching feature points of adjacent image frames (such as SIFT or optical flow method); the optical axis pointing vector refers to the direction vector of the image center optical axis in three-dimensional space generated based on the pitch angle and yaw angle of the gimbal; spherical coordinate system mapping refers to combining the optical axis pointing vector with the drone's flight altitude and converting it into spherical coordinates (azimuth, pitch angle, radius) to generate parameters in the optical data stream used to compensate for perspective distortion, such as the image edge stretching correction coefficient.

[0146] In this embodiment, the optical flow method is first used to calculate the attitude change between adjacent image frames, and the optical axis pointing vector of each frame of the image is generated by combining the real-time pitch angle of two degrees and yaw angle of five degrees of the gimbal. Secondly, the optical axis vector and the drone altitude of twenty meters are input into the spherical coordinate system model to calculate the geographic azimuth and pitch angle corresponding to the image center. Finally, the viewing angle compensation parameters are generated, such as radial stretching of the image edge pixels to correct the perspective distortion caused by the pitch angle.

[0147] Step 503: Performing altitude attenuation compensation on the electromagnetic field intensity change data, dynamically adjusting the electromagnetic field gradient threshold according to the relative distance between the UAV and the ground obstacle, and generating a field intensity correction factor carrying an altitude marker in the electrical data stream;

[0148] In this step, altitude attenuation compensation refers to the energy correction of the electromagnetic field intensity data collected by the UAV according to the electromagnetic wave propagation characteristics, so as to eliminate the signal attenuation error caused by the change of flight altitude; relative distance refers to the real-time vertical distance between the UAV and the ground obstacle, which is obtained by the altitude sensor or laser ranging equipment; the electromagnetic field gradient threshold refers to the critical value of the field intensity change determined by the dynamic adjustment according to the relative distance, which is used to distinguish the effective reflected signal from the environmental noise; the field strength correction factor refers to the electromagnetic field intensity correction coefficient generated by compensating for altitude attenuation and filtering noise, which is used to improve the spatial consistency of the electrical data stream.

[0149] In this embodiment, based on the physical law of electromagnetic wave attenuation and combined with the real-time altitude data of the UAV, the electromagnetic field strength is reversely compensated to restore the true field strength of the obstacle reflection signal. Secondly, the gradient threshold is dynamically adjusted according to the relative distance between the UAV and the obstacle to suppress the random interference caused by metal debris or cables during low-altitude flight. Finally, a field strength correction factor carrying an altitude marker is generated for subsequent multimodal data fusion.

[0150] Step 504: Input the vibration characteristic parameters, the viewing angle compensation parameters, and the field strength correction factor into a pre-trained spatiotemporal coupling model, and calculate the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream based on the motion acceleration and angular velocity parameters of the UAV.

[0151] In this step, the spatiotemporal coupling model refers to a dynamic calibration model established through a multimodal data association algorithm, which is used to analyze the spatiotemporal correlation between acoustic, optical, and electrical data streams; the vibration characteristic parameter refers to the acoustic vibration energy distribution of the position marker generated in step 501; the viewing angle compensation parameter refers to the spherical coordinate mapping coefficient generated in step 502 for correcting optical image distortion; the field strength correction factor refers to the electromagnetic field strength correction coefficient of the height marker generated in step 503; the motion acceleration and angular velocity parameters refer to the three-axis acceleration and angular velocity data output in real time by the drone's inertial measurement unit, which is used to quantify the impact of dynamic changes in flight attitude on multimodal data; the time synchronization deviation refers to the timing offset of the acoustic and optical data streams due to differences in sensor sampling rates or environmental interference; the spatial distortion coefficient refers to the spatial coordinate misalignment of the optical and electrical data streams due to the rotation or displacement of the drone.

[0152] In this embodiment, vibration characteristic parameters, viewing angle compensation parameters and field strength correction factors are first input into the spatiotemporal coupling model, where the vibration characteristic parameters include the spatial distribution information of acoustic vibration energy, the viewing angle compensation parameters include the spherical coordinate mapping relationship of the optical axis pointing vector, and the field strength correction factor includes the electromagnetic field intensity gradient after height compensation; secondly, based on the motion acceleration and angular velocity parameters of the UAV, the spatiotemporal influence weights of attitude changes on multimodal data are calculated through kinematic equations; then, a correlation analysis algorithm is used to calculate the time synchronization deviation of the acoustic and optical data streams, for example, the time offset of the vibration energy peak and the flame profile change is detected by the cross-correlation function; at the same time, the spatial distortion coefficients of the optical and electrical data streams caused by the sudden change of the pitch angle are calculated through the spatial projection transformation model; finally, the time synchronization deviation and spatial distortion coefficient are output as the calibration basis for multimodal data fusion.

[0153] Step 505: Perform Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate a time-axis aligned acoustic intermediate data stream; perform perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate a spatial-axis aligned optical intermediate data stream;

[0154] In this step, Doppler frequency shift compensation refers to the dynamic correction of the acoustic signal frequency offset caused by the UAV movement based on the time synchronization deviation between the acoustic data stream and the optical data stream; the time-axis aligned acoustic intermediate data stream refers to the characteristic stream in which the frequency and timing of the acoustic signal after compensation are strictly synchronized with the optical data; perspective projection correction refers to the correction of the geometric deformation of the optical image caused by the sudden change of the UAV's posture based on the spatial distortion coefficient; the spatial axis aligned optical intermediate data stream refers to the characteristic stream in which the spatial coordinates of the optical image and the electrical data are consistent after the perspective distortion is eliminated.

[0155] In this embodiment, first, based on the time synchronization deviation, the acoustic data stream is compensated for Doppler frequency shift through a dynamic frequency adjustment algorithm. For example, a sliding window Fourier transform is used to track the frequency offset in real time and make inverse corrections. Secondly, a perspective projection matrix is ​​constructed according to the spatial distortion coefficient, and the optical image is reprojected through a geometric transformation model. For example, an affine transformation is used to correct an image with an oblique perspective to an orthographic projection. Finally, an acoustic intermediate data stream aligned with the time axis and an optical intermediate data stream aligned with the spatial axis are output for subsequent multimodal fusion engine calls.

[0156] Step 506: Based on the correlation between the field intensity correction factor and the altitude change rate of the UAV, vertical field intensity gradient compensation is performed on the electrical data stream to generate an electrical intermediate data stream that is temporally and spatially matched with the acoustic intermediate data stream and the optical intermediate data stream.

[0157] In this step, the field strength correction factor refers to the electromagnetic field strength correction coefficient generated in step 503, which is used to compensate for the field strength loss caused by altitude attenuation; the drone altitude change rate refers to the vertical movement speed of the drone obtained by a barometer or laser rangefinder, which characterizes the dynamic changes of the aircraft in the vertical direction; the vertical direction field strength gradient compensation refers to the dynamic adjustment of the field strength distribution of the electrical data stream based on the altitude change rate to eliminate the vertical gradient error caused by altitude fluctuations; the electrical intermediate data stream refers to the electromagnetic characteristic stream that is time-space matched with the acoustic intermediate data stream and the optical intermediate data stream after vertical gradient compensation, and contains highly consistent field strength spatial distribution and timing characteristics.

[0158] In this embodiment, a correlation model between the field strength correction factor and the drone's altitude change rate is first established, for example, by analyzing the dynamic relationship between the two through linear regression; secondly, the vertical field strength gradient compensation amount is calculated based on the real-time altitude change rate, for example, the low-altitude field strength data is proportionally enhanced when the drone rises; then, the compensation amount is superimposed on the field strength distribution of the electrical data stream, for example, a vertical gradient continuous electromagnetic field distribution is generated through a spatial interpolation algorithm; finally, the compensated electrical data stream is time-space matched with the acoustic intermediate data stream and the optical intermediate data stream generated in step 505 to ensure that the timestamps of the three are aligned and the spatial coordinates are consistent, thereby generating an electrical intermediate data stream.

[0159] Step 507: Bind the acoustic intermediate data stream to the vibration characteristic parameters to generate an acoustic characteristic stream, bind the optical intermediate data stream to the viewing angle compensation parameters to generate an optical characteristic stream, and bind the electrical intermediate data stream to the field strength correction factor to generate an electrical characteristic stream.

[0160] In this step, the acoustic feature stream refers to a structured data stream that binds the acoustic intermediate data stream aligned with the time axis to the vibration characteristic parameters (such as vibration energy peak, frequency distribution), which contains the physical characteristics of the acoustic signal and its spatiotemporal calibration information; the optical feature stream refers to a structured data stream that binds the optical intermediate data stream aligned with the spatial axis to the viewing angle compensation parameters (such as the optical axis pointing vector, perspective correction coefficient), which contains the geometric characteristics of the optical image and its dynamic posture calibration information; the electrical feature stream refers to a structured data stream that binds the spatiotemporal matching electrical intermediate data stream to the field strength correction factor (such as the height compensation coefficient, vertical gradient parameter), which contains the electromagnetic field intensity distribution and its dynamic height calibration information.

[0161] In this embodiment, the time-domain aligned vibration signal in the acoustic intermediate data stream is firstly data-associated with the vibration characteristic parameters to generate an acoustic feature stream carrying a complete spatiotemporal calibration label; secondly, the corrected image in the optical intermediate data stream is bound to the perspective compensation parameter to generate an optical feature stream carrying a posture compensation label; at the same time, the vertical gradient compensation field strength data in the electrical intermediate data stream is bound to the field strength correction factor to generate an electrical feature stream carrying a height calibration label.

[0162] In some embodiments, according to step 104, based on the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, anti-interference fusion processing is performed on the calibration feature stream, and multi-dimensional coupling features are extracted from the fusion result, including:

[0163] Step 601: construct a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and perform channel-by-channel convolution operations on the acoustic calibration feature stream, the optical calibration feature stream, and the electrical calibration feature stream in the calibration feature stream with the reflection correlation weight matrix to generate acoustic reflection correlation features, optical reflection correlation features, and electromagnetic reflection correlation features.

[0164] In this step, the reflection association weight matrix refers to the weight distribution model constructed based on the reflection characteristics of surface obstacles, which is used to quantify the correlation between the reflection intensity of different materials to sound, light, and electromagnetic waves; the channel-by-channel convolution operation refers to the independent convolution processing of the acoustic, optical, and electrical characteristic streams respectively to extract the correlation characteristics between each modal data and the material reflection characteristics; the acoustic / optical / electromagnetic reflection association feature refers to the multimodal data feature generated by convolution and carrying the reflection characteristic label, such as the acoustic vibration energy concentration feature corresponding to the high reflection area of ​​the metal obstacle.

[0165] In this embodiment, a reflection correlation weight matrix is ​​first constructed based on a preset reflection characteristic table (such as high reflection intensity of metal materials and low reflection intensity of concrete materials); secondly, the acoustic, optical, and electrical feature streams in the calibration feature stream are respectively convolved with the weight matrix channel by channel. For example, the acoustic feature stream extracts the correlation between vibration energy and metal reflection through time convolution, the optical feature stream enhances the texture contrast of the metal contour through spatial convolution, and the electrical feature stream highlights the field intensity gradient change in the metal area through space-time convolution; finally, the acoustic reflection correlation features (such as metal vibration energy concentration), optical reflection correlation features (such as sharpened image of the metal pipe edge) and electromagnetic reflection correlation features (such as sudden increase in field intensity gradient in the metal area) are output.

[0166] Step 602: Calculate a dynamic attenuation factor for the electromagnetic field intensity change data, and perform weighted superposition of the dynamic attenuation factor and the electromagnetic reflection correlation feature to generate an electromagnetic anti-interference feature flow, wherein the dynamic attenuation factor is generated based on the product relationship between the real-time altitude of the UAV and the reflection correlation weight matrix;

[0167] In this step, the dynamic attenuation factor refers to the field strength attenuation compensation coefficient generated according to the product relationship between the UAV flight altitude and the reflection correlation weight matrix, which is used to suppress the interference of low-altitude environmental noise on electromagnetic data; weighted superposition refers to the fusion of the dynamic attenuation factor and the electromagnetic reflection correlation characteristics according to the weight to eliminate the influence of multipath reflection or random interference; the electromagnetic anti-interference feature stream refers to the noise-resistant and highly robust electromagnetic field data stream generated after superposition.

[0168] In this embodiment, the dynamic attenuation factor is first calculated based on the real-time altitude of the UAV (such as high-altitude or low-altitude flight status) and the weight relationship of the metal material in the reflection correlation weight matrix; secondly, the attenuation factor is weightedly superimposed with the electromagnetic reflection correlation characteristics, for example, the field strength gradient of the metal area is enhanced while the noise interference of the non-metal area is suppressed; finally, an electromagnetic anti-interference characteristic flow is generated, in which the metal reflection field strength gradient is preserved intact, while the interference signal of the non-metal area is effectively filtered.

[0169] Step 603: extracting the low-frequency vibration energy distribution parameter from the acoustic reflection correlation feature, and performing a cross-modal product operation on the low-frequency vibration energy distribution parameter from the optical reflection correlation feature to generate a vibration profile coupling coefficient.

[0170] In this step, the low-frequency vibration energy distribution parameter refers to the temporal distribution characteristics of vibration energy in a specific frequency band extracted from the acoustic reflection correlation features, which is used to characterize the mechanical vibration intensity caused by metal structure deformation or combustion; the image contour gradient parameter refers to the image edge gradient amplitude extracted from the optical reflection correlation features, which is used to quantify the sharpness of the metal obstacle contour; the vibration contour coupling coefficient refers to the correlation parameter between the acoustic vibration energy and the optical contour gradient generated by the cross-modal product operation, which reflects the synergistic characteristics of mechanical vibration and the geometric deformation of the object.

[0171] In this embodiment, first, a short-time Fourier transform is performed on the acoustic reflection correlation features to extract the low-frequency vibration energy distribution parameters (such as the proportion of the main frequency energy of the metal pipe vibration); secondly, the Sobel operator is used to extract the image contour gradient parameters (such as the gradient amplitude of the metal pipe edge) of the optical reflection correlation features; then, the two types of parameters are normalized and point-by-point multiplication operation is performed according to the time window to generate the vibration contour coupling coefficient (such as the metal area energy-contour synergy coefficient).

[0172] Step 604: performing spatiotemporal consistency matching on the vibration profile coupling coefficient and the electromagnetic anti-interference characteristic flow, calculating a multimodal fusion weight based on the curvature parameter of the UAV flight trajectory and the geomagnetic azimuth deviation, and generating a fusion feature matrix including acoustic vibration energy distribution, optical profile gradient, and electromagnetic anti-interference strength;

[0173] In this step, spatiotemporal consistency matching refers to aligning the vibration profile coupling coefficient with the spatiotemporal coordinates of the electromagnetic anti-interference characteristic flow to ensure that the timestamps of the acoustic, optical, and electromagnetic data are consistent with the spatial positions; the curvature parameter of the UAV flight trajectory refers to the degree of curvature of the flight path calculated by the inertial navigation system, which is used to quantify the impact of dynamic motion on multimodal data; the geomagnetic azimuth deviation refers to the angular deviation between the actual heading of the UAV and the geomagnetic North Pole, which is used to correct the spatial pointing error of the electromagnetic data; the multimodal fusion weight refers to the fusion ratio of acoustic, optical, and electromagnetic data dynamically allocated according to the curvature and azimuth deviation; the fusion feature matrix refers to a multidimensional data set that integrates the acoustic vibration energy distribution, optical profile gradient, and electromagnetic anti-interference strength, which is used for dynamic modeling of the fire scene.

[0174] In this embodiment, the vibration profile coupling coefficient is first aligned with the electromagnetic anti-interference feature flow according to the timestamp and UTM coordinates; secondly, based on the curvature of the UAV flight trajectory (such as high curvature during sharp turns) and the geomagnetic azimuth deviation (such as a yaw angle deviation of five degrees), the fusion weight is dynamically allocated through a fuzzy logic model (such as reducing the optical weight and increasing the electromagnetic weight when the curvature is high); finally, the fusion feature matrix is ​​generated by weighted superposition according to the weights.

[0175] Step 605: Perform multi-scale decomposition on the fused feature matrix to extract obstacle position distribution features in the low-frequency component and moving target trajectory features in the high-frequency component. The position distribution features and trajectory features are correlated and superimposed in the time domain to generate the multi-dimensional coupling features, wherein the multi-dimensional coupling features include the obstacle spatial topological relationship and the moving target motion vector parameters.

[0176] In this step, multi-scale decomposition refers to decomposing the fused feature matrix into signal layers of different frequency components through wavelet transform or similar algorithms to separate the low-frequency global structure from the high-frequency transient changes; the obstacle position distribution characteristics in the low-frequency component refer to the spatial distribution information of static obstacles (such as metal pipes and concrete walls) extracted through low-frequency signals; the moving target trajectory characteristics in the high-frequency component refer to the motion path information of dynamic targets (such as rescue personnel and flying burning objects) extracted through high-frequency signals; time domain correlation superposition refers to the alignment and fusion of static obstacle positions and dynamic target trajectories along the time axis; multi-dimensional coupling characteristics refer to the composite data stream that integrates the spatial topological relationship of obstacles (such as the relative position of metal pipes and walls) and the motion vector parameters of moving targets (such as speed and direction).

[0177] In this embodiment, the fused feature matrix is ​​first subjected to wavelet multi-scale decomposition to extract the obstacle location distribution in the low-frequency layer signal (such as the metal pipe in the northeast corner of the fifth floor) and the moving target trajectory in the high-frequency layer signal (such as the rescuer moving in the northwest direction). Secondly, the low-frequency obstacle location and the high-frequency moving trajectory are superimposed through timestamp alignment. For example, the metal pipe location is bound to the personnel movement velocity vector in each frame of data. Finally, a multi-dimensional coupling feature is generated, which includes the spatial topological network of static obstacles and the real-time motion parameters of dynamic targets.

[0178] In some embodiments, according to step 101, synchronously collecting acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data of the drone during its movement in the target area includes:

[0179] Step 701: Based on the obstacle spatial topology relationship parameters in the multi-dimensional coupling feature, a three-dimensional grid is divided into the target area, and a dynamic obstacle distribution model of the terrain surface is generated according to the mapping relationship between the UAV flight altitude and the obstacle reflection coefficient;

[0180] In this step, three-dimensional meshing refers to decomposing the target area into a three-dimensional grid composed of cubic units according to the spatial resolution, which is used to quantify the distribution density and geometric properties of obstacles; the obstacle space topological relationship parameters refer to the relative position and connection relationship between static obstacles (such as metal pipes and walls) described in the multi-dimensional coupling characteristics; the dynamic obstacle distribution model refers to the real-time updated three-dimensional obstacle space model generated based on the mapping relationship between the UAV flight altitude and the obstacle reflection coefficient, reflecting the material reflection characteristics and distribution dynamics of obstacles in different altitude layers.

[0181] In this embodiment, the target area is first divided into a three-dimensional grid based on the obstacle spatial topological relationship parameters in the multi-dimensional coupling characteristics. For example, the fifth floor of the fire scene is divided into a cubic grid with a side length of one meter. Secondly, based on the mapping relationship between the real-time flight altitude of the drone and the obstacle reflection coefficient (such as high reflection coefficient for metal and low reflection coefficient for concrete), the obstacle distribution weight within the grid unit is dynamically adjusted. For example, the grid weight of metal obstacles is increased during low-altitude flight. Finally, a dynamic obstacle distribution model is generated, and the three-dimensional position and reflection intensity of obstacles such as metal pipes and burning debris are annotated in real time.

[0182] Step 702: extracting the moving target motion vector parameters from the multi-dimensional coupling features, combining them with the real-time gradient distribution of the electromagnetic field intensity change data, and performing dynamic probability prediction on the moving target's motion trajectory to generate trajectory prediction parameters including velocity direction and collision probability;

[0183] In this step, the motion vector parameters of the moving target refer to the speed and direction data of the moving target (such as rescue personnel, scattered burning materials) extracted from the multi-dimensional coupling characteristics; the real-time gradient distribution of the electromagnetic field intensity change data refers to the spatial gradient information of the electromagnetic field intensity in the anti-interference electromagnetic characteristic flow; dynamic probability prediction refers to predicting the future trajectory of the moving target and its collision risk by fusing the motion vector and electromagnetic gradient data; the trajectory prediction parameters refer to the composite parameters in the output results that include the target speed direction, path probability and collision probability.

[0184] In this embodiment, the motion vector of the moving target is first extracted from the multi-dimensional coupling features. Secondly, the probability distribution of the target's future trajectory is predicted through a hidden Markov model or Monte Carlo sampling, combined with the electromagnetic field intensity gradient distribution (such as a sudden increase in the electromagnetic field intensity gradient in the northwest direction). Finally, the trajectory prediction parameters are generated.

[0185] Step 703: Perform spatial interpolation processing on the dynamic obstacle distribution model, generate an obstacle height field strength map covering the target area based on the UAV flight speed and gimbal viewing angle parameters, and simultaneously perform time domain superposition of the trajectory prediction parameters and the obstacle height field strength map to generate a dynamic threat field strength map.

[0186] In this step, spatial interpolation processing refers to converting discrete obstacle distribution data into continuous spatial field strength distribution through Kriging interpolation or inverse distance weighted algorithm; the obstacle height field strength map refers to a two-dimensional grid map reflecting the distribution of obstacle reflection field strength at different altitudes in the target area; the dynamic threat field strength map refers to a three-dimensional spatial model containing real-time threat levels generated by superimposing trajectory prediction parameters and obstacle height field strength map along the time axis, which is used to characterize the comprehensive threat posed by dynamic obstacles and moving targets to drones.

[0187] In this embodiment, spatial interpolation processing is first performed on the dynamic obstacle distribution model to fill in the data missing due to sensor blind spots and generate an obstacle height field strength map covering the target area. Secondly, the resolution of the field strength map is adjusted based on the drone's flight speed and gimbal viewing angle parameters. For example, the resolution is reduced during high-speed flight to improve processing efficiency. Finally, the trajectory prediction parameters (such as the collision probability of the moving target) are superimposed with the obstacle height field strength map according to the timestamp to generate a dynamic threat field strength map, in which high-threat areas are marked with superimposed features of sudden increases in field strength gradients and increased collision probabilities.

[0188] Step 704: Based on the field intensity gradient distribution in the dynamic threat field intensity map, a potential field navigation algorithm in the UAV flight path planning model is used to calculate a path node sequence between the UAV's current position and the target point, thereby generating an initial obstacle avoidance path.

[0189] In this step, the potential field navigation algorithm refers to a path planning method that calculates the direction of movement of the UAV by simulating gravity (attraction of the target point) and repulsion (repulsion of the threat field strength); the path node sequence refers to a set of discrete path points from the current position to the target point generated by the algorithm, which is used to guide the UAV's segmented flight.

[0190] In this embodiment, first, based on the field strength gradient distribution in the dynamic threat field strength map, the direction of the gravitational-repulsive force between the current position of the drone and the target point is calculated through the potential field algorithm; secondly, a path node sequence is generated along the direction of the resultant force, for example, dense nodes are set in low-threat areas to improve path smoothness, and sparse nodes are set in high-threat areas for rapid traversal; finally, the initial obstacle avoidance path is output to ensure that the path nodes avoid high-threat field strength areas.

[0191] Step 705: Perform electromagnetic interference robustness verification on the initial obstacle avoidance path, perform a weighted assessment of the safety of the path nodes based on the transient pulse peak intensity in the electromagnetic field intensity change data and the collision probability in the trajectory prediction parameters, and generate an optimal path node sequence including threat avoidance weights;

[0192] In this step, electromagnetic interference robustness verification refers to evaluating the stability of path nodes in a strong electromagnetic interference environment by analyzing the transient pulse peak intensity in the electromagnetic field intensity change data; the threat avoidance weight refers to the safety level of the path node calculated based on the collision probability and electromagnetic interference intensity. The higher the weight, the safer the node.

[0193] In this embodiment, the electromagnetic interference intensity of each node in the initial obstacle avoidance path is first detected, for example, to detect whether the transient pulse peak exceeds the safety threshold; secondly, the node threat avoidance weight is calculated through a weighted evaluation model based on the collision probability in the trajectory prediction parameters; finally, the nodes with weights higher than the safety threshold are screened to generate the optimal path node sequence.

[0194] Step 706: Perform three-dimensional spatial fusion of the optimal path node sequence and the dynamic threat field strength map to generate a three-dimensional spatial situation model covering the target area, wherein the three-dimensional spatial situation model includes obstacle elevation distribution, moving target trajectory prediction, and electromagnetic interference hotspot markers;

[0195] In this step, the three-dimensional spatial situation model refers to a three-dimensional environmental perception model generated by fusing the optimal path node sequence and the dynamic threat field strength map, which includes obstacle elevation distribution, moving target trajectory prediction and electromagnetic interference hot zone marking; obstacle elevation distribution refers to the height stratification information of obstacles in the vertical direction; moving target trajectory prediction refers to the dynamically updated movement path of rescue personnel or scattered burning materials; electromagnetic interference hot zone marking refers to the three-dimensional spatial annotation of abnormal electromagnetic field strength areas.

[0196] In this embodiment, the optimal path node sequence is first projected onto the dynamic threat field strength map through a three-dimensional spatial mapping algorithm to generate a spatial correlation between the path nodes and the threat field strength. Secondly, the obstacle elevation distribution and mobile target trajectory prediction data are integrated through point cloud fusion technology. For example, the height information of metal pipes is superimposed on the movement path of rescue personnel. Finally, the electromagnetic interference hot zone is marked as a three-dimensional thermal layer and fused with the obstacle and mobile target data to generate a three-dimensional spatial situation model covering the target area.

[0197] Figure 2 The present invention provides a schematic diagram of a multi-dimensional information fusion system for UAVs based on acoustic, optical and electrical composite detection. Figure 2 As shown, the system includes:

[0198] The acquisition module 21 is used to synchronously collect the acoustic frequency band signals, visible light image sequences and electromagnetic field intensity change data of the UAV during its movement in the target area;

[0199] an association module 22 for synchronously associating the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV, to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow;

[0200] a correction module 23 for performing multi-channel dynamic calibration on the image distortion region in the optical characteristic stream based on the time deviation between the low-frequency vibration mode in the acoustic characteristic stream and the transient electromagnetic pulse in the electrical characteristic stream, and synchronously correcting the spatiotemporal coordinate parameters of the acoustic characteristic stream, the optical characteristic stream, and the electrical characteristic stream based on the calibration result to generate a calibrated characteristic stream;

[0201] A fusion module 24 is configured to perform anti-interference fusion processing on the calibration feature stream based on a correlation mapping relationship between a preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, and extract multi-dimensional coupling features from the fusion result;

[0202] The generation module 25 is used to input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area. Figure 2 The multi-dimensional information fusion system of UAV based on acoustic, optical and electrical composite detection can perform Figure 1 The implementation principles and technical effects of the multi-dimensional information fusion method for drones based on acoustic, optical, and optical combined detection described in the illustrated embodiment are not further elaborated. The specific manner in which each module and unit performs operations in the multi-dimensional information fusion system for drones based on acoustic, optical, and optical combined detection in the aforementioned embodiment has been described in detail in the relevant embodiments of the method and will not be further elaborated here.

[0203] In one possible design, Figure 2 The multi-dimensional information fusion system of a UAV based on acoustic, optical and electrical composite detection of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0204] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0205] The processing component 32 is used for the above Figure 1 The embodiment provides a multi-dimensional information fusion method for UAV based on acoustic, optical and electrical composite detection.

[0206] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0207] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0208] Of course, a computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0209] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0210] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0211] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0212] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The embodiment shown is a method for multi-dimensional information fusion of a UAV based on acoustic, optical and electrical composite detection.

[0213] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0214] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0215] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-dimensional information fusion method for UAV based on acoustic, optical and electrical composite detection, characterized in that: include: Synchronously collect acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data as the drone moves in the target area; Synchronously correlating the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow; Based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, a multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibrated characteristic flow; According to the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data in the target area, the calibration feature flow is subjected to anti-interference fusion processing, and the multi-dimensional coupling features in the fusion result are extracted; The multi-dimensional coupling features are input into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area.

2. The method according to claim 1, characterized in that Based on the time deviation between the low-frequency vibration mode in the acoustic characteristic flow and the transient electromagnetic pulse in the electrical characteristic flow, a multi-channel dynamic calibration is performed on the image distortion area in the optical characteristic flow, and based on the calibration result, the spatiotemporal coordinate parameters of the acoustic characteristic flow, the optical characteristic flow, and the electrical characteristic flow are synchronously corrected to generate a calibrated characteristic flow, including: Extracting a low-frequency vibration peak time series synchronized with the UAV rotor vibration from the acoustic characteristic flow, and separating a transient pulse front time series triggered by electromagnetic reflections from obstacles from the electrical characteristic flow, calculating the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series, and generating an acoustic-electrical synergy time deviation value; constructing a dynamic calibration matrix based on the acoustic-electrical synergy time deviation value, and performing a frame-by-frame convolution operation on the dynamic calibration matrix and the pixel displacement of the image distortion area in the optical feature stream to generate optical distortion compensation parameters; According to the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter, a time-domain stretching process is performed on the low-frequency vibration peak time series in the acoustic characteristic flow to generate time-axis aligned acoustic calibration parameters, and a time-shift compensation process is performed on the transient pulse front time series in the electrical characteristic flow to generate time-axis aligned electrical calibration parameters; Inputting the acoustic calibration parameters, electrical calibration parameters, and optical distortion compensation parameters into a multi-channel coupler, and calculating the spatial projection offset of the acoustic characteristic flow, the optical axis pointing compensation of the optical characteristic flow, and the vertical gradient compensation of the electrical characteristic flow based on the real-time attitude angular velocity and geomagnetic azimuth deviation of the UAV; Superimposing the spatial projection offset onto the spatiotemporal coordinates of the acoustic feature flow to generate a calibrated acoustic feature flow, superimposing the optical axis pointing compensation onto the image distortion region of the optical feature flow to generate a calibrated optical feature flow, and superimposing the vertical gradient compensation onto the spatiotemporal coordinates of the electrical feature flow to generate a calibrated electrical feature flow; The time and space coordinate parameters of the calibrated acoustic characteristic flow, optical characteristic flow and electrical characteristic flow are normalized and fused to generate a calibrated characteristic flow.

3. The method according to claim 2, characterized in that According to the product relationship between the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter, a time domain stretching process is performed on the low-frequency vibration peak time series in the acoustic characteristic flow to generate time axis aligned acoustic calibration parameters, including: Calculating a time domain stretch factor of a low-frequency vibration peak time series in the acoustic characteristic flow based on a product value of the acoustic-electrical synergy time deviation value and the optical distortion compensation parameter; performing non-uniform interpolation processing on the low-frequency vibration peak time series, dynamically scaling the interval time of the vibration peaks according to the time domain stretching factor, and generating an interpolated vibration peak time series; Extracting phase differences between adjacent vibration peaks from the interpolated vibration peak time series, and generating phase correction parameters for the vibration peaks based on a multiplication relationship between the phase differences and the optical distortion compensation parameters; Performing a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change of the pitch angle of the UAV, and generate a phase-aligned vibration peak time series; Based on the phase-aligned vibration peak time series and the original spectrum distribution of the acoustic characteristic flow, time-axis aligned acoustic calibration parameters are calculated, wherein the acoustic calibration parameters include the time domain remapping coefficient and the phase compensation factor of the vibration peak.

4. The method according to claim 3, characterized in that Performing time shift compensation processing on the transient pulse front time series in the electrical characteristic flow to generate time axis aligned electrical calibration parameters, including: Performing a sliding window detection on the transient pulse front time series, extracting the interval time difference between adjacent pulse fronts, and performing a proportional operation on the interval time difference and the acoustic-electrical coordination time deviation value to generate a dynamic time shift factor; constructing a time shift compensation function based on the dynamic time shift factor, performing non-uniform interpolation processing on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series; Extracting a pulse front steepness parameter from the interpolated pulse front time series, and performing a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate pulse waveform distortion caused by multipath reflection of metal obstacles, thereby generating a pulse time series with a corrected waveform; Based on the waveform-corrected pulse time sequence and the original intensity distribution of the electrical characteristic flow, electrical calibration parameters aligned with the time axis are calculated, wherein the electrical calibration parameters include a time-shift remapping coefficient of the pulse leading edge and a waveform compensation factor.

5. The method according to claim 1, characterized in that Synchronously correlating the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV to generate an acoustic feature flow, an optical feature flow, and an electrical feature flow, including: Performing time-frequency analysis on the acoustic frequency band signal to extract a low-frequency component that matches the vibration frequency of the drone rotor, and performing spatial projection conversion on the low-frequency component and the real-time three-dimensional coordinates of the drone to generate vibration characteristic parameters with position markers in the acoustic data stream; performing inter-frame attitude calculation on the visible light image sequence, calculating the optical axis pointing vector of each frame in the visible light image sequence based on the pitch and yaw angle parameters of the drone gimbal, mapping the optical axis pointing vector to the flight altitude of the drone in a spherical coordinate system, and generating a perspective compensation parameter carrying an attitude tag in the optical data stream; Performing altitude attenuation compensation on the electromagnetic field intensity change data, dynamically adjusting the electromagnetic field gradient threshold according to the relative distance between the UAV and ground obstacles, and generating a field intensity correction factor carrying an altitude marker in the electrical data stream; Inputting the vibration characteristic parameters, the viewing angle compensation parameters, and the field strength correction factor into a pre-trained spatiotemporal coupling model, and calculating the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream based on the motion acceleration and angular velocity parameters of the UAV; Performing Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate an acoustic intermediate data stream aligned with the time axis; performing perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate an optical intermediate data stream aligned with the spatial axis; Based on the correlation between the field strength correction factor and the altitude change rate of the UAV, vertical field strength gradient compensation is performed on the electrical data stream to generate an electrical intermediate data stream that is temporally and spatially matched with the acoustic intermediate data stream and the optical intermediate data stream; The acoustic intermediate data stream is bound to the vibration characteristic parameters to generate an acoustic characteristic stream, and the optical intermediate data stream is bound to the viewing angle compensation parameters to generate an optical characteristic stream. At the same time, the electrical intermediate data stream is bound to the field strength correction factor to generate an electrical characteristic stream.

6. The method according to claim 1, characterized in that According to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, the calibration feature flow is subjected to anti-interference fusion processing, and the multi-dimensional coupling features in the fusion result are extracted, including: Constructing a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and performing channel-by-channel convolution operations on the acoustic calibration feature flow, the optical calibration feature flow, and the electrical calibration feature flow in the calibration feature flow with the reflection correlation weight matrix to generate acoustic reflection correlation features, optical reflection correlation features, and electromagnetic reflection correlation features; Calculating a dynamic attenuation factor for the electromagnetic field intensity change data, and weightedly superimposing the dynamic attenuation factor with the electromagnetic reflection correlation feature to generate an electromagnetic anti-interference feature flow, wherein the dynamic attenuation factor is generated based on the product relationship between the real-time altitude of the UAV and the reflection correlation weight matrix; Extracting low-frequency vibration energy distribution parameters from the acoustic reflection correlation feature and performing a cross-modal product operation with image profile gradient parameters from the optical reflection correlation feature to generate a vibration profile coupling coefficient; The vibration profile coupling coefficient is matched with the electromagnetic anti-interference characteristic flow in time and space, and the multimodal fusion weight is calculated based on the curvature parameter of the UAV flight trajectory and the geomagnetic azimuth deviation to generate a fusion feature matrix including acoustic vibration energy distribution, optical profile gradient and electromagnetic anti-interference strength; The fusion feature matrix is ​​decomposed at multiple scales to extract the obstacle position distribution features in the low-frequency component and the moving target trajectory features in the high-frequency component. The position distribution features and trajectory features are correlated and superimposed in the time domain to generate the multi-dimensional coupling features, wherein the multi-dimensional coupling features include the obstacle spatial topological relationship and the moving target motion vector parameters.

7. The method according to claim 1, characterized in that Synchronously collect acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data as the drone moves in the target area, including: Based on the obstacle spatial topological relationship parameters in the multi-dimensional coupling feature, the target area is divided into three-dimensional grids, and a dynamic obstacle distribution model of the terrain surface is generated according to the mapping relationship between the UAV flight altitude and the obstacle reflection coefficient; Extracting the moving target motion vector parameters from the multi-dimensional coupling features, combining them with the real-time gradient distribution of the electromagnetic field intensity change data, and performing dynamic probability prediction on the moving target's motion trajectory to generate trajectory prediction parameters including velocity direction and collision probability; Performing spatial interpolation processing on the dynamic obstacle distribution model, generating an obstacle height field strength map covering the target area based on the UAV flight speed and gimbal viewing angle parameters, and simultaneously superimposing the trajectory prediction parameters with the obstacle height field strength map in the time domain to generate a dynamic threat field strength map; Based on the field intensity gradient distribution in the dynamic threat field intensity map, the potential field navigation algorithm in the UAV flight path planning model is used to calculate the path node sequence between the current position of the UAV and the target point, and generate an initial obstacle avoidance path; Performing electromagnetic interference robustness verification on the initial obstacle avoidance path, performing a weighted assessment of the safety of path nodes based on the transient pulse peak intensity in the electromagnetic field intensity change data and the collision probability in the trajectory prediction parameters, and generating an optimal path node sequence including threat avoidance weights; The optimal path node sequence is fused with the dynamic threat field strength map in three-dimensional space to generate a three-dimensional spatial situation model covering the target area, wherein the three-dimensional spatial situation model includes obstacle elevation distribution, moving target trajectory prediction and electromagnetic interference hot zone marking.

8. A multi-dimensional information fusion system for UAVs based on acoustic, optical and electrical composite detection, characterized in that: include: The acquisition module is used to synchronously collect the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data of the drone during its movement in the target area; an association module, configured to synchronously associate the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV, respectively, to generate an acoustic characteristic flow, an optical characteristic flow, and an electrical characteristic flow; a correction module, configured to perform multi-channel dynamic calibration on the image distortion region in the optical characteristic stream based on the time deviation between the low-frequency vibration mode in the acoustic characteristic stream and the transient electromagnetic pulse in the electrical characteristic stream, and synchronously correct the spatiotemporal coordinate parameters of the acoustic characteristic stream, the optical characteristic stream, and the electrical characteristic stream based on the calibration result to generate a calibrated characteristic stream; A fusion module is used to perform anti-interference fusion processing on the calibration feature flow based on the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data in the target area, and extract multi-dimensional coupling features from the fusion result; A generation module is used to input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering the target area.

9. A computing device, characterized in that It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a multi-dimensional information fusion method for unmanned aerial vehicle based on acoustic, optical and electrical composite detection as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that A computer program is stored, and when the computer program is executed by a computer, the method for multi-dimensional information fusion of a UAV based on acoustic, optical and electrical composite detection as described in any one of claims 1 to 7 is implemented.

Images