Unmanned aerial vehicle multi-dimensional information fusion method and system based on acousto-optic-electric composite detection

Abstract

The invention provides an unmanned aerial vehicle multi-dimensional information fusion method and system based on acousto-optic-electric composite detection. According to the method, acoustic frequency band signals, visible light image sequences and electromagnetic field intensity change data are collected, acoustic characteristic flow, optical characteristic flow and electrical characteristic flow are generated, and based on the time deviation between a low-frequency vibration mode and transient electromagnetic pulses, multi-channel dynamic calibration is carried out on an image distortion area in the optical characteristic flow. The method comprises the following steps: acquiring an acoustic feature flow, an optical feature flow and an electrical feature flow, correcting space-time coordinate parameters of the acoustic feature flow, the optical feature flow and the electrical feature flow, generating a calibration feature flow, carrying out anti-interference fusion processing on the calibration feature flow, extracting multi-dimensional coupling features, inputting the multi-dimensional coupling features into an unmanned aerial vehicle flight path planning model, and generating a three-dimensional space situation model; according to the invention, the physical space-time consistency alignment and noise suppression of the multi-modal data are realized, and the obstacle positioning precision and the dynamic trajectory prediction real-time performance are significantly improved.

Description

Technical Field

[0001] This application relates to the technical field of multi-dimensional information fusion of unmanned aerial vehicles (UAVs), and particularly to a method and system for multi-dimensional information fusion of UAVs based on acoustic-optic-electric composite detection. Background Art

[0002] When a UAV performs complex tasks such as high-voltage transmission line inspection and disaster site search and rescue, it is necessary to detect dynamic and static obstacles in real time, and at the same time overcome environmental interferences such as strong electromagnetic interference and sudden changes in light. Such scenarios require multi-source sensor data to have high-precision spatio-temporal synchronization capabilities and anti-interference fusion characteristics to ensure that the UAV can still achieve accurate obstacle positioning and real-time path planning in bad weather or complex electromagnetic environments.

[0003] In response to the above requirements, the existing technology mainly adopts a vision and lidar fusion scheme: obtaining environmental depth information through a binocular camera, combining lidar point cloud data, using a deep learning model to extract the geometric features of obstacles, and fusing the attitude data of the inertial measurement unit to generate a three-dimensional environmental map. This scheme attempts to achieve complementary fusion of multi-modal data by using lidar to make up for the limitations of vision in low-light environments.

[0004] However, this scheme has significant technical defects. In low visibility environments such as rain, fog, and sandstorms, binocular vision fails in environmental modeling due to the lack of feature points, while the detection ability of lidar for obstacles made of non-reflective materials (such as transparent plastics or dark objects) is seriously insufficient, and missed detections occur frequently. In addition, in strong electromagnetic interference areas such as high-voltage transmission lines, lidar point cloud data is easily distorted by magnetic fields, resulting in a significant decrease in obstacle positioning accuracy. More critically, the sampling rate difference between vision and lidar leads to cumulative errors in the time axis of dynamic obstacle trajectory prediction, and the path planning response delay is obvious, making it difficult to meet the real-time requirements of emergency obstacle avoidance. Summary of the Invention

[0005] This application provides a method and system for multi-dimensional information fusion of UAVs based on acoustic-optic-electric composite detection to solve the problems of poor environmental adaptability, insufficient spatio-temporal synchronization accuracy, and weak anti-interference ability in the existing technology.

[0006] In a first aspect, this application provides a method for multi-dimensional information fusion of UAVs based on acoustic-optic-electric composite detection, including: Synchronously collecting acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the UAV in the target area; Synchronously associating the acoustic frequency band signals, the visible light image sequences, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; Based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, perform multi-channel dynamic calibration on the image distortion area in the optical feature stream, and synchronously correct the spatio-temporal coordinate parameters of the acoustic feature stream, optical feature stream, and electrical feature stream based on the calibration result to generate a calibrated feature stream; According to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, perform anti-interference fusion processing on the calibrated feature stream, and extract multi-dimensional coupling features in the fusion result; Input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional space situation model covering the target area.

[0007] Optionally, based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, perform multi-channel dynamic calibration on the image distortion area in the optical feature stream, and synchronously correct the spatio-temporal coordinate parameters of the acoustic feature stream, optical feature stream, and electrical feature stream based on the calibration result to generate a calibrated feature stream, including: Extract the low-frequency vibration peak time series synchronized with the UAV rotor vibration from the acoustic feature stream, and at the same time separate the transient pulse front time series triggered by the electromagnetic reflection of obstacles from the electrical feature stream, calculate the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series, and generate an acoustic-electric collaborative time deviation value; Construct a dynamic calibration matrix based on the acoustic-electric collaborative time deviation value, perform frame-by-frame convolution operation on the dynamic calibration matrix and the pixel displacement of the image distortion area in the optical feature stream to generate optical distortion compensation parameters; According to the product relationship between the acoustic-electric collaborative time deviation value and the optical distortion compensation parameters, perform time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream to generate acoustical calibration parameters with aligned time axes, and at the same time perform time shift compensation processing on the transient pulse front time series in the electrical feature stream to generate electrical calibration parameters with aligned time axes; Input the acoustical calibration parameters, electrical calibration parameters, and optical distortion compensation parameters into a multi-channel coupler, and calculate the spatial projection offset of the acoustic feature stream, the optical axis pointing compensation of the optical feature stream, and the vertical gradient compensation of the electrical feature stream based on the real-time attitude angular velocity and geomagnetic azimuth deviation of the UAV; Superimpose the spatial projection offset onto the spatio-temporal coordinates of the acoustic feature stream to generate a calibrated acoustic feature stream, superimpose the optical axis pointing compensation onto the image distortion area of the optical feature stream to generate a calibrated optical feature stream, and superimpose the vertical gradient compensation onto the spatio-temporal coordinates of the electrical feature stream to generate a calibrated electrical feature stream; Normalize and fuse the spatio-temporal coordinate parameters of the calibrated acoustic feature stream, optical feature stream, and electrical feature stream to generate a calibrated feature stream.

[0008] Optionally, perform time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream according to the product relationship between the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter to generate acoustical calibration parameters with aligned time axes, including: Calculate the time-domain stretching factor of the low-frequency vibration peak time series in the acoustic feature stream based on the product value of the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter; Perform non-uniform interpolation processing on the low-frequency vibration peak time series, dynamically scale the interval time between vibration peaks according to the time-domain stretching factor, and generate an interpolated vibration peak time series; Extract the phase difference between adjacent vibration peaks from the interpolated vibration peak time series, and generate a phase correction parameter for the vibration peak according to the product relationship between the phase difference and the optical distortion compensation parameter; Perform a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change of the UAV pitch angle, and generate a vibration peak time series with aligned phases; Calculate the acoustical calibration parameters with aligned time axes based on the vibration peak time series with aligned phases and the original spectral distribution of the acoustic feature stream, where the acoustical calibration parameters include the time-domain remapping coefficient and phase compensation factor of the vibration peak.

[0009] Optionally, perform time shift compensation processing on the transient pulse front time series in the electrical feature stream to generate electrical calibration parameters with aligned time axes, including: Perform a sliding window detection on the transient pulse front time series, extract the interval time difference between adjacent pulse fronts, and perform a proportional operation on the interval time difference and the acoustic-electric collaborative time deviation value to generate a dynamic time shift factor; Construct a time shift compensation function based on the dynamic time shift factor, and perform non-uniform interpolation processing on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series; Extract the pulse front steepness parameter from the interpolated pulse front time series, and perform a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate the pulse waveform distortion caused by the multi-path reflection of the metal obstacle, and generate a pulse time series with corrected waveform; Calculate the electrocalibration parameters aligned with the time axis based on the pulse time series after waveform correction and the original intensity distribution of the electrical characteristic stream, where the electrocalibration parameters include the time shift remapping coefficient and the waveform compensation factor of the pulse front edge; Optionally, synchronously associate the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the drone to generate an acoustic characteristic stream, an optical characteristic stream, and an electrical characteristic stream, including: Perform time-frequency analysis on the acoustic frequency band signal, extract the low-frequency components matching the vibration frequency of the drone rotor, and perform spatial projection conversion on the low-frequency components and the real-time three-dimensional coordinates of the drone to generate vibration characteristic parameters with position marks in the acoustic data stream; Perform inter-frame attitude calculation on the visible light image sequence, calculate the optical axis pointing vector of each frame of the visible light image sequence based on the pitch angle and yaw angle parameters of the drone gimbal, and perform spherical coordinate mapping on the optical axis pointing vector and the flight altitude of the drone to generate perspective compensation parameters with attitude marks in the optical data stream; Perform height attenuation compensation on the electromagnetic field intensity change data, and dynamically adjust the electromagnetic field gradient threshold according to the relative distance between the drone and the ground obstacle to generate a field strength correction factor with height marks in the electrical data stream; Input the vibration characteristic parameters, the perspective compensation parameters, and the field strength correction factor into a pre-trained spatio-temporal coupling model, and calculate the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream based on the motion acceleration and angular velocity parameters of the drone; Perform Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate an acoustic intermediate data stream aligned with the time axis; perform perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate an optical intermediate data stream aligned with the spatial axis; Perform vertical direction field strength gradient compensation on the electrical data stream based on the correlation between the field strength correction factor and the drone height change rate to generate an electrical intermediate data stream that is spatio-temporally matched with the acoustic intermediate data stream and the optical intermediate data stream; Bind the acoustic intermediate data stream with the vibration characteristic parameters to generate an acoustic characteristic stream, bind the optical intermediate data stream with the perspective compensation parameters to generate an optical characteristic stream, and at the same time bind the electrical intermediate data stream with the field strength correction factor to generate an electrical characteristic stream.

[0010] Optionally, according to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field strength change data, perform anti-interference fusion processing on the calibration feature stream, and extract multi-dimensional coupling features in the fusion result, including: Construct a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and perform channel-by-channel convolution operations on the acoustic calibration feature stream, optical calibration feature stream, and electrical calibration feature stream in the calibration feature stream with the reflection correlation weight matrix respectively to generate acoustic reflection correlation features, optical reflection correlation features, and electromagnetic reflection correlation features; Calculate the dynamic attenuation factor for the electromagnetic field strength change data, and perform weighted superposition of the dynamic attenuation factor and the electromagnetic reflection correlation features to generate an electromagnetic anti-interference feature stream, where the dynamic attenuation factor is generated according to the product relationship between the real-time height of the drone and the reflection correlation weight matrix; Extract the low-frequency vibration energy distribution parameters in the acoustic reflection correlation features, and perform cross-modal product operations with the image contour gradient parameters in the optical reflection correlation features to generate a vibration contour coupling coefficient; Perform spatio-temporal consistency matching on the vibration contour coupling coefficient and the electromagnetic anti-interference feature stream, calculate the multi-modal fusion weight based on the curvature parameter of the drone flight trajectory and the geomagnetic azimuth deviation, and generate a fusion feature matrix including acoustic vibration energy distribution, optical contour gradient, and electromagnetic anti-interference intensity; Perform multi-scale decomposition on the fusion feature matrix, extract the obstacle position distribution features in the low-frequency component and the moving target trajectory features in the high-frequency component, perform time-domain correlation superposition on the position distribution features and the trajectory features, and generate the multi-dimensional coupling features, where the multi-dimensional coupling features include the obstacle spatial topological relationship and the moving target motion vector parameters.

[0011] Optionally, synchronously collect the acoustic frequency band signal, visible light image sequence, and electromagnetic field strength change data during the movement of the drone in the target area, including: Based on the obstacle spatial topological relationship parameters in the multi-dimensional coupling features, perform three-dimensional grid division on the target area, and generate a dynamic obstacle distribution model of the terrain surface according to the mapping relationship between the drone flight height and the obstacle reflection coefficient; Extract the moving target motion vector parameters in the multi-dimensional coupling features, and combine the real-time gradient distribution of the electromagnetic field strength change data to perform dynamic probability prediction on the motion trajectory of the moving target, and generate trajectory prediction parameters including the speed direction and the collision probability; Perform spatial interpolation processing on the dynamic obstacle distribution model, generate an obstacle height field intensity map covering the target area according to the UAV flight speed and gimbal view angle parameters, and at the same time perform time-domain superposition of the trajectory prediction parameters and the obstacle height field intensity map to generate a dynamic threat field intensity map; Based on the field intensity gradient distribution in the dynamic threat field intensity map, calculate the path node sequence between the current position of the UAV and the target point through the potential field navigation algorithm in the UAV flight path planning model to generate an initial obstacle avoidance path; Perform electromagnetic interference robustness verification on the initial obstacle avoidance path, and perform weighted evaluation on the safety of path nodes according to the transient pulse peak intensity in the electromagnetic field intensity change data and the collision probability in the trajectory prediction parameters to generate an optimal path node sequence including threat avoidance weights; Fuse the optimal path node sequence with the dynamic threat field intensity map in three-dimensional space to generate a three-dimensional space situation model covering the target area, where the three-dimensional space situation model includes obstacle elevation distribution, moving target trajectory prediction, and electromagnetic interference hot zone marking.

[0012] In a second aspect, the present application provides a UAV multi-dimensional information fusion system based on acoustic-optic-electric composite detection, including: An acquisition module for synchronously acquiring acoustic wave band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the UAV in the target area; An association module for synchronously associating the acoustic wave band signals, the visible light image sequences, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; A correction module for dynamically calibrating the image distortion area in the optical feature stream in multiple channels based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, and synchronously correcting the spatio-temporal coordinate parameters of the acoustic feature stream, the optical feature stream, and the electrical feature stream based on the calibration result to generate a calibrated feature stream; A fusion module for performing anti-interference fusion processing on the calibrated feature stream according to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, and extracting multi-dimensional coupling features in the fusion result; A generation module for inputting the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional space situation model covering the target area.

[0013] In a third aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a method for multi-dimensional information fusion of an unmanned aerial vehicle based on acoustic, optical and electrical composite detection as described in the first aspect above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, which when executed by a computer, implements a method for multi-dimensional information fusion of an unmanned aerial vehicle based on acoustic, optical and electrical composite detection as described in the first aspect.

[0015] In the embodiment of the present application, through acoustic, optical and electrical multi-modal data acquisition and spatio-temporal correlation, acoustic, optical and electrical characteristic streams with physical consistency are formed; based on the time deviation between acoustic low-frequency vibration and electromagnetic transient pulse, the optical distortion area is dynamically calibrated, and by synchronously correcting the spatio-temporal coordinate parameters of the three channels, the spatio-temporal alignment accuracy of multi-source data in a complex electromagnetic interference environment is significantly improved; combined with the mapping relationship between the reflection coefficient of surface obstacles and the electromagnetic field intensity for anti-interference fusion processing, the extracted multi-dimensional coupling characteristics can effectively distinguish real obstacles from noise interference, and finally the generated dynamic three-dimensional space situation model realizes centimeter-level obstacle positioning and real-time obstacle avoidance in scenarios such as high-voltage transmission line inspection and disaster search and rescue, solving the core problems of poor environmental adaptability, insufficient spatio-temporal synchronization accuracy and weak anti-interference ability in the prior art.

[0016] Further, by extracting the acoustic-electric collaborative time deviation value to construct a dynamic calibration matrix, pixel-level compensation for each frame of the optical distortion area is realized. Combining the acoustic time-domain stretching and electrical time-shift compensation technologies, the spatio-temporal axis offset of multi-sensor data caused by sudden changes in the attitude of the unmanned aerial vehicle is eliminated; based on the multi-channel space compensation amount calculated from the attitude angular velocity and geomagnetic azimuth deviation, the spatial projection of the acoustic characteristic stream, the optical axis direction of the optical characteristic stream, and the vertical gradient of the electrical characteristic stream are ensured to be accurately aligned in physical space; the calibrated characteristic stream generated by spatio-temporal coordinate normalization fusion provides a high-consistency data basis for subsequent anti-interference fusion. Especially in complex terrain and strong electromagnetic interference scenarios, the collaborative calibration accuracy of multi-modal data and the real-time performance of dynamic obstacle trajectory prediction are significantly better than traditional single-modal calibration methods, effectively improving the detection reliability in complex environments.

[0017] These aspects or other aspects of the present application will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0019] Figure 1 It shows a flowchart of a multi-dimensional information fusion method for unmanned aerial vehicles based on acoustic-optic-electric composite detection provided by the present application; Figure 2 It shows a schematic structural diagram of a multi-dimensional information fusion system for unmanned aerial vehicles based on acoustic-optic-electric composite detection provided by the present application; Figure 3 It shows a schematic structural diagram of a computing device provided by the present application. Detailed implementation manners

[0020] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application.

[0021] In some processes described in the specification and claims of the present application and the above-mentioned accompanying drawings, there are multiple operations that appear in a specific order. However, it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., do not represent the sequence, and do not limit that "first" and "second" are of different types.

[0022] Currently, in complex scenarios such as high-voltage transmission line inspection and disaster search and rescue by unmanned aerial vehicles, the prior art mainly relies on the vision and lidar fusion scheme, but there are significant technical bottlenecks: in low visibility environments such as rain, fog, and sand and dust, binocular vision fails in environmental modeling due to the lack of feature points, while the lidar has prominent problems of missing detection of non-reflective material obstacles (such as transparent plastics and dark objects); in a strong electromagnetic interference environment, the lidar point cloud data is easily distorted by the magnetic field, resulting in a significant decrease in the obstacle positioning accuracy; more critically, the sampling rate difference between vision and lidar causes the spatio-temporal reference mismatch of multi-source data, and there are cumulative errors in the dynamic obstacle trajectory prediction, and the path planning response delay is significant, making it difficult to meet the real-time obstacle avoidance requirements. The above problems seriously restrict the reliability and adaptability of the prior art in complex electromagnetic interference and dynamic environments.

[0023] In view of the above defects, the present application proposes a multi-dimensional information fusion method for unmanned aerial vehicles (UAVs) based on acoustic-optic-electric composite detection. By means of the collaborative calibration and anti-interference fusion of multi-modal data of acoustics, optics and electromagnetics, the bottleneck of the existing technology is broken through. Specifically, by extracting the time deviation between the acoustic low-frequency vibration mode and the electromagnetic transient pulse, a dynamic calibration matrix is constructed to perform frame-by-frame compensation on the optical distortion area; based on the attitude parameters of the UAV, the spatio-temporal coordinates of the three-channel data are corrected in real time to eliminate the multi-source data offset caused by sudden changes in the flight attitude; combined with the correlation mapping between the surface obstacle reflection coefficient and the electromagnetic field gradient, anti-interference coupling features are extracted. This method realizes the physical spatio-temporal consistency alignment and noise suppression of multi-modal data in scenarios such as strong electromagnetic interference and bad weather, significantly improves the obstacle positioning accuracy and the real-time performance of dynamic trajectory prediction, solves the three core problems of poor environmental adaptability, weak anti-interference ability and insufficient spatio-temporal synchronization in the existing technology, and provides technical support for the reliable detection and autonomous obstacle avoidance of UAVs in complex scenarios.

[0024] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0025] Figure 1 The flowchart of a multi-dimensional information fusion method for unmanned aerial vehicles (UAVs) based on acoustic-optic-electric composite detection provided by the embodiments of the present application is as Figure 1 shown, and the method includes: Step 101, synchronously collect the acoustic frequency band signal, visible light image sequence and electromagnetic field strength change data during the movement of the UAV in the target area; In this step, the acoustic frequency band signal refers to the acoustic signal of the target area collected by the acoustic sensor carried by the UAV, including 200 - 500 Hz low-frequency mechanical vibration noise (such as the rotor operation sound pattern) and 5 - 20 kHz high-frequency obstacle reflection sound waves (such as metal collision sounds); the visible light image sequence refers to the RGB image frames of the target area continuously captured by the UAV gimbal camera, which is used to identify the obstacle contour and surface texture features and is prone to image distortion (such as perspective deformation, motion blur) affected by the flight attitude; the electromagnetic field strength change data refers to the electromagnetic field strength gradient information collected by the electromagnetic sensor carried by the UAV, including transient pulse waveforms (such as electromagnetic interference signals with steep fronts) caused by metal obstacles or high-voltage transmission lines.

[0026] In this embodiment, first, an acoustic sensor is used to collect acoustic wave signals in the target area at a sampling rate of 20 kHz. An adaptive band-pass filter is utilized to separate low-frequency mechanical vibration noise and high-frequency obstacle-reflected acoustic waves, generating an acoustic wave frequency band signal containing rotor vibration characteristics and obstacle acoustic signatures. Secondly, a global shutter camera mounted on the UAV gimbal is used to collect visible light image sequences at a frame rate of 30 fps. An optical flow estimation algorithm is employed to perform motion compensation on adjacent frames, and combined with the real-time attitude parameters (pitch angle, yaw angle) of the UAV to generate a de-blurred and stable image sequence. At the same time, a three-axis electromagnetic sensor is used to collect electromagnetic field intensity data at a sampling rate of 1 kHz. A sliding time window is adopted to detect transient pulse waveforms, record the pulse peak intensity and timestamp information, generating intensity change data containing obstacle electromagnetic reflection characteristics. Finally, through the time synchronization module of the UAV flight control system, the timestamps of the acoustic wave frequency band signal, visible light image sequence, and electromagnetic field intensity change data are unified to the same reference clock to ensure the spatio-temporal synchronization of the three-channel data.

[0027] For example, in a high-rise building fire, the UAV flies to the north side of the fire scene (latitude 34.05°, longitude 118.25°, altitude 20 meters). The acoustic sensor collects the 220 Hz low-frequency vibration caused by flame combustion and the 6 - 8 kHz high-frequency acoustic waves of glass bursting, and separates the characteristics through an adaptive band-pass filter. The thermal imaging camera captures the flame contour and smoke diffusion images at 30 fps, and compensates for the pixel displacement caused by the disturbance of the hot air flow through the optical flow method. The electromagnetic sensor detects the 1.5 GHz transient pulse caused by cable short circuit and records the peak intensity and timestamp. The multi-sensor data are aligned by the time synchronization module of the flight control system to generate an original dataset of acoustic, optical, and electromagnetic with consistent spatio-temporal reference.

[0028] In this step, through the synchronous acquisition and preprocessing of multi-modal sensors, an original dataset of high-integrity acoustic, optical, and electromagnetic is generated. The acoustic wave frequency band signal provides mechanical vibration and obstacle acoustic characteristics, the visible light image sequence retains the target contour and texture information, and the electromagnetic field intensity change data captures the reflection characteristics of metal obstacles. The time-axis deviation of multi-sensor data is eliminated through the time synchronization module, providing a physical consistency basis for subsequent multi-modal calibration and fusion. In scenarios of complex electromagnetic interference, dynamic light changes, and sudden UAV attitude changes, this method significantly improves the signal-to-noise ratio and spatio-temporal alignment accuracy of the original data, ensuring the reliability of the subsequent processing flow.

[0029] Step 102: Synchronously associate the acoustic wave frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; In this step, the current position and attitude parameters include the real-time longitude and latitude of the UAV (GPS / RTK positioning), altitude (barometer or laser rangefinder), and pitch angle, roll angle, and yaw angle (IMU output); the acoustic / optical / electrical feature stream refers to the structured data stream obtained by fusing the original sensor data with spatio-temporal parameters, including timestamp, geographical coordinates, original sensor data, and extracted physical features (such as acoustic wave spectrum, image texture, electromagnetic pulse waveform).

[0030] In this embodiment, first, the acquisition times of acoustic, image, and electromagnetic data are synchronized with the reference clock of the UAV flight control system through timestamp alignment. For example, an interpolation algorithm is used to compensate for the sampling interval difference between the acoustic sensor (10 kHz sampling rate) and the electromagnetic sensor (1 kHz sampling rate) to ensure that the time deviation of all data is less than 1 ms. Second, the sensor data is bound to the real-time position of the UAV through spatial coordinate mapping: for acoustic signals, the azimuth angle of the sound source is calculated using a spherical projection model in combination with the UAV altitude and pitch angle; for visible light images, the image pixels are mapped to a three-dimensional geographical space through a geographical coordinate transformation algorithm (UTM projection) based on the UAV longitude and latitude and yaw angle; for electromagnetic field data, a spatial interpolation is used to generate a heat map of the electromagnetic intensity distribution based on the UAV position and roll angle. Subsequently, a structured feature stream is generated through feature fusion: for acoustic frequency band signals, Mel-frequency cepstral coefficients (MFCC) are applied to extract spectral features, which are combined with the azimuth angle of the sound source to form an acoustic feature stream; for image sequences, a convolutional neural network (ResNet-18) is used to extract deep semantic features, and the geographical coordinates are superimposed to generate an optical feature stream; for electromagnetic data, transient pulse energy features are extracted through wavelet packet decomposition, and the electrical feature stream is generated in combination with the heat map coordinates. Finally, the three types of feature streams are integrated into a multi-modal data set according to a unified spatio-temporal reference through data encapsulation for subsequent algorithm calls.

[0031] Continuing the embodiment of the previous step, based on the original data of step 101, the acoustic feature stream extracts the 220 Hz combustion vibration spectrum through short-time Fourier transform (STFT), and combines the UAV pitch angle of 3.5° and altitude data to calculate the coordinates of the ignition point as 34.0503°N, 118.2502°E; the optical feature stream uses the ResNet-50 model to extract the flame contour in the thermal image, and superimposes the UAV yaw angle of 2.8° and GPS coordinates to generate an image feature stream with geographical tags; the electrical feature stream extracts the 1.5 GHz pulse front edge feature through wavelet transform, and outputs the short-circuit point coordinates (34.0501°N, 118.2505°E) after binding to the UAV position. After spatio-temporal alignment of the three types of feature streams, it is identified that the fire spreads southeast along the ventilation duct.

[0032] Step 103: Based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, perform multi-channel dynamic calibration on the image distortion area in the optical feature stream, and synchronously correct the spatio-temporal coordinate parameters of the acoustic feature stream, optical feature stream, and electrical feature stream based on the calibration result to generate a calibrated feature stream; First, extract the timing difference between the low-frequency vibration mode of flame combustion in the acoustic feature stream and the transient electromagnetic pulse of cable short circuit in the electrical feature stream through time deviation analysis. For example, use the Generalized Cross-Correlation (GCC-PHAT) algorithm to calculate the time offset between the two, and identify the causal relationship between the structural vibration caused by thermal expansion and electromagnetic interference. Second, correct the image distortion of the optical feature stream through multi-channel dynamic calibration: Based on the time deviation, use a non-rigid image registration algorithm (such as a deformation model based on B-spline) to align the flame contour captured by the thermal imaging camera with the smoke occlusion area in the visible light image, and at the same time compensate for the pixel displacement caused by the hot air flow through the optical flow method. Subsequently, update the multi-modal feature stream parameters through spatio-temporal coordinate correction: According to the calibrated flame position and smoke diffusion model, reverse optimize the drone altitude and pitch angle, and synchronously adjust the combustion intensity estimation of the acoustic feature stream and the short-circuit point coordinates of the electrical feature stream to ensure the spatial consistency of multi-modal data. Finally, package the corrected acoustic, optical, and electrical feature streams into a calibrated feature stream by the adaptive fusion engine according to a unified spatio-temporal reference.

[0033] Continuing the embodiment of the previous step, the acoustic feature stream detects a 180Hz low-frequency vibration of the steel beam due to heat deformation, and the electrical feature stream captures a 1.5GHz pulse of a secondary short circuit. Through Generalized Cross-Correlation (GCC-PHAT) analysis, it is found that the electromagnetic pulse leads the vibration peak by 80ms, indicating that the electrical fault precedes the structural deformation. Based on this deviation, perform B-spline non-rigid registration on the image distortion area of the exterior wall caused by high-temperature deformation in the optical feature stream to restore the true building contour; synchronously correct the steel beam coordinates of the acoustic feature stream to 34.0506°N, 118.2498°E, the short-circuit point coordinates of the electrical feature stream to 34.0502°N, 118.2495°E, and optimize the drone pitch angle to 4.1°. The calibrated feature stream is input into the positioning model to accurately locate two trapped persons to the west safety passage on the fifth floor (34.051°N, 118.249°E), and give an early warning of the risk of ceiling collapse.

[0034] Step 104: According to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field strength change data, perform anti-interference fusion processing on the calibrated feature stream, and extract the multi-dimensional coupling features in the fusion result; In this step, the surface obstacle reflection coefficient refers to the reflection ability of obstacles of different materials pre-calibrated for electromagnetic waves; anti-interference fusion processing improves the consistency of multi-modal data by suppressing the cross-interference between electromagnetic noise (such as cable short-circuit pulses) and optical thermal distortion; multi-dimensional coupling features refer to the cross-modal joint features after fusing acoustic combustion intensity, optical flame contour, and electromagnetic material reflection characteristics, such as the thermal coupling effect of flame-metal structure.

[0035] In this embodiment, first, an association relationship between the electromagnetic field intensity and the surface obstacle material is established through reflection coefficient mapping modeling. For example, a random forest model is used to match the peak intensity of electromagnetic pulses (1.5 GHz) with the preset metal / concrete reflection coefficients (0.8 / 0.3) to generate an obstacle material-electromagnetic reflection probability distribution map. Secondly, cross-interference in multi-modal data is suppressed through anti-interference weighted fusion: for the electromagnetic data in the calibrated feature stream, the wavelet threshold denoising algorithm is used to filter out random pulses of cable short circuits; for the optical feature stream, based on the reflection coefficient mapping results, an attention mechanism (such as SENet) is used to enhance the flame contour weight in the metal obstacle area and suppress misjudgments in the smoke occlusion area. Subsequently, cross-modal information is integrated through coupled feature extraction: the material reflection feature is extracted from the denoised electromagnetic data, the flame spatial distribution feature is extracted from the weighted optical data, and the combustion intensity time series feature is extracted from the acoustic data. The correlation between the three is modeled through a graph convolutional network (GCN) to output multi-dimensional coupling features (such as "metal pipe-flame contact heat conduction coefficient").

[0036] Continuing the embodiment of the previous step, with a preset metal pipe reflection coefficient of 0.8 and a concrete wall of 0.3, the probability that the 1.5 GHz pulse intensity detected by the electromagnetic feature stream is mapped to the metal material is 92%. The random forest model determines that there is a metal structure on the west side of the fifth floor; after filtering out interference pulses caused by cable short circuits through wavelet threshold denoising, the optical feature stream uses SENet to enhance the flame contour in the metal pipe area and identifies the flame spreading along the pipe to the sixth floor; the acoustic feature stream extracts the time series correlation between the peak combustion intensity and the vibration frequency of 180 Hz of the metal structure; the GCN model fuses the above features and outputs a "metal-flame thermal coupling coefficient" of 0.75, indicating a very high risk of thermal deformation of the pipe; finally, through feature coding, the trapped person is located in the safe area behind the metal pipe (34.051°N, 118.249°E), and the time window for pipe burst is warned.

[0037] Step 105, input the multi-dimensional coupling features into a preset unmanned aerial vehicle flight path planning model to generate a three-dimensional space situation model covering the target area; In this step, the UAV flight path planning model refers to a dynamic route generation framework based on reinforcement learning (such as the PPO algorithm) and three-dimensional obstacle avoidance constraints (such as the thermal radiation threshold and the structural stability index); the three-dimensional space situation model refers to a visual three-dimensional grid map that integrates fire spread prediction, obstacle distribution (metal pipes, collapsed walls), and risk levels (high-temperature areas, structural deformation areas), including a risk heat map, a safe passage layer, and a real-time update mechanism.

[0038] In this embodiment, first, a multi-dimensional coupling feature is initialized and loaded through the path planning model, including the metal-flame thermal coupling coefficient (such as 0.75), the structural deformation frequency (such as 180 Hz), and the electromagnetic reflection feature (such as the metal probability of 92%), to construct the state space of the fire scene environment, and the action space is defined as the six-degree-of-freedom movement of the UAV (pitch, yaw, lift, etc.). Secondly, an initial path is generated through reinforcement learning decision-making: the proximal policy optimization (PPO) algorithm is used, with minimizing the thermal radiation exposure and maximizing the search and rescue coverage rate as the objective function, and the reward value of each waypoint is calculated in combination with real-time coupling features (such as the pipeline burst risk window) to generate a three-dimensional obstacle avoidance path from the current coordinate (34.051°N, 118.249°E) to the location of the trapped person. Subsequently, the environment is dynamically updated through three-dimensional situation modeling: based on the path planning result, the Gaussian mixture model (GMM) is used to divide the fire scene risk levels (high-risk red area, medium-risk yellow area, safe green area), and the flame contour extracted by the optical feature flow and the structural vibration hot spot identified by the acoustic feature flow are superimposed to generate a three-dimensional grid map including the risk heat map and the safe passage.

[0039] Continuing the embodiment of the previous step, the multi-dimensional coupling features include the metal pipe thermal coupling coefficient of 0.75, the structural vibration frequency of 180 Hz on the west side of the fifth floor, and the metal obstacle probability of 92%. After the path planning model loads the above features, the PPO algorithm generates a three-dimensional path from the current position of the UAV (34.051°N, 118.249°E) to the trapped person: vertically climb to avoid the high-risk red area on the fifth floor (excessive thermal radiation), horizontally detour through the safe green area (stable concrete structure), and dynamically avoid the metal pipe burst window; the three-dimensional situation model divides the high-risk area (fire in the ventilation pipe on the sixth floor), the medium-risk area (metal deformation on the west side of the fifth floor), and the safe passage (fire door on the southeast side) through the GMM, superimposes the thermal imaging flame contour and the acoustic vibration hot spot, and outputs a real-time updated grid map; when the electromagnetic sensor detects a new short-circuit pulse, the LSTM predicts the fire spread to the seventh floor, and the model dynamically adjusts the path to vertically descend and then laterally evacuate, and updates the risk layer of the situation model.

[0040] This step realizes the autonomous obstacle avoidance and search and rescue decision-making of drones in complex fire environments through multi-dimensional coupled feature-driven path planning and three-dimensional situation modeling. The dynamic path planning takes into account the thermal radiation safety threshold and the risk of structural deformation. The three-dimensional situation model intuitively presents the fire evolution and safe passages, and the real-time feedback mechanism ensures the timeliness of route and risk prediction, significantly improving the safety and search and rescue efficiency of fire search and rescue tasks.

[0041] In some embodiments, according to step 103, based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, multi-channel dynamic calibration is performed on the image distortion region in the optical feature stream, and based on the calibration result, the spatio-temporal coordinate parameters of the acoustic feature stream, optical feature stream, and electrical feature stream are synchronously corrected to generate a calibrated feature stream, including: Step 201: Extract the low-frequency vibration peak time series synchronized with the drone rotor vibration from the acoustic feature stream, and at the same time separate the transient pulse front time series triggered by the electromagnetic reflection of obstacles from the electrical feature stream, and calculate the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series to generate an acoustic-electric collaborative time deviation value; In this step, the low-frequency vibration peak time series refers to the set of peak time points of the periodic vibration waveform synchronized with the drone rotor speed (such as 200 Hz) extracted from the acoustic feature stream, representing the vibration interference of the aircraft itself; the transient pulse front time series refers to the set of rising edge trigger time points of the obstacle electromagnetic reflection event (such as the reflection of a 1.5 GHz pulse by a metal pipe) separated from the electrical feature stream; the average phase difference quantifies the timing deviation of the two types of signals by statistically averaging the time offset between the acoustic vibration peak and the electromagnetic pulse front (such as 15 ms); the acoustic-electric collaborative time deviation value is used to calibrate the reference parameter of the multi-modal data time axis (such as 15 ms).

[0042] In this embodiment, first, the rotor vibration low-frequency peak time series is extracted from the acoustic feature stream through a peak detection algorithm (such as local maximum screening). For example, the peak time points that appear every 5 ms in the vibration waveform are detected at a period of 200 Hz. Secondly, the transient pulse front time series is separated from the electrical feature stream by using the threshold triggering method. For example, 30% of the amplitude of the 1.5 GHz pulse is set as the threshold, and the rising edge time points exceeding the threshold are recorded. Subsequently, the phase difference between the two types of time series is calculated through the cross-correlation function. For example, the average value of the time offset between the acoustic peak and the electromagnetic pulse front is statistically calculated within a 10-second time window to generate an acoustic-electric collaborative time deviation value (such as 15 ms).

[0043] Step 202: Construct a dynamic calibration matrix based on the acoustic-electric collaborative time deviation value, and perform a frame-by-frame convolution operation on the dynamic calibration matrix and the pixel displacement amount in the image distortion area of the optical feature flow to generate an optical distortion compensation parameter; The dynamic calibration matrix refers to a two-dimensional weight matrix (such as a 5×5 convolution kernel) constructed based on the acoustic-electric collaborative time deviation value (such as 15 ms), which is used to characterize the compensation weight distribution of the optical distortion area (such as image blurring caused by smoke refraction) in the time-space dimension; the pixel displacement amount refers to the pixel position offset amount (such as 3 pixels horizontally and 2 pixels vertically) generated between adjacent frames in the optical feature flow due to the attitude jitter of the drone or environmental interference (such as thermal air flow disturbance); the optical distortion compensation parameter is a correction coefficient (such as a horizontal compensation coefficient of 0.8 and a vertical compensation coefficient of 0.6) generated through convolution operation, which is used to restore the geometric deformation of the image.

[0044] In this embodiment, first, the acoustic-electric collaborative time deviation value (15 ms) is converted into the convolution kernel weight of the dynamic calibration matrix through a matrix mapping algorithm. For example, a 5×5 matrix is constructed, the central weight is 0.8 (reflecting the compensation intensity of the time deviation for the central pixel), and the edge weights decay according to the Gaussian distribution; second, the pixel displacement amount in the image distortion area of the optical feature flow is calculated using an optical flow method (such as the Lucas-Kanade algorithm). For example, the horizontal displacement of adjacent frame pixels in the smoke occlusion area is detected as 3 pixels and the vertical displacement is 2 pixels; finally, the dynamic calibration matrix and the pixel displacement amount are weighted and fused through a frame-by-frame convolution operation to generate an optical distortion compensation parameter (such as a horizontal compensation coefficient of 0.8) for subsequent image geometric correction.

[0045] Step 203: According to the product relationship between the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter, perform a time-domain stretching process on the low-frequency vibration peak time series in the acoustic feature flow to generate an acoustically calibrated parameter with aligned time axes, and at the same time perform a time shift compensation process on the transient pulse front time series in the electrical feature flow to generate an electrically calibrated parameter with aligned time axes; In this step, the acoustic-electric collaborative time deviation value refers to the average time deviation between the acoustic vibration and the electromagnetic reflection event calculated in Step 201, which is used to quantify the timing difference between the two types of signals. The optical distortion compensation parameter refers to the compensation coefficient generated in Step 202 for correcting the geometric deformation of the image. The time-domain stretching process refers to linearly interpolating and adjusting the time axis of the acoustic feature flow according to the product relationship between the deviation value and the compensation parameter to eliminate timing jitter. The time shift compensation process refers to the overall translation of the time series of the electrical feature flow to align with the time reference after optical distortion compensation. The acoustically calibrated parameter and the electrically calibrated parameter are the acoustic and electromagnetic signals with aligned time axes after time-domain adjustment.

[0046] In this embodiment, first, the acousto-electric collaborative time deviation value is multiplied by the optical distortion compensation parameter to obtain a time-domain adjustment amount for subsequent calibration. The linear interpolation algorithm is used to stretch the time domain of the low-frequency vibration peak time series in the acoustic feature stream. For example, according to the adjustment amount, the time axis length of the signal is expanded or compressed to synchronize the time reference of the acoustic signal with the optical data. At the same time, the transient pulse front time series of the electrical feature stream is shifted as a whole. For example, each pulse trigger time point is translated by the adjustment amount to match the calibrated time axis. Finally, the acoustical calibration parameter and the electrical calibration parameter with aligned time axes are output.

[0047] Step 204: Input the acoustical calibration parameter, the electrical calibration parameter, and the optical distortion compensation parameter into a multi-channel coupler, and calculate the spatial projection offset of the acoustic feature stream, the optical axis pointing compensation of the optical feature stream, and the vertical gradient compensation of the electrical feature stream based on the real-time attitude angular velocity and the geomagnetic azimuth deviation of the UAV. In this step, the spatial projection offset is the projection error of the sound source coordinates in the acoustic feature stream in the three-dimensional space caused by the real-time attitude angular velocity of the UAV, and it needs to be corrected by the attitude parameters. The optical axis pointing compensation is the calibration parameter for correcting the image center offset of the optical feature stream caused by the geomagnetic azimuth deviation of the UAV, and it is used to restore the true position of the target. The vertical gradient compensation is the correction coefficient for the vertical distribution error of the electromagnetic field strength caused by the height change of the UAV, and it is used to improve the detection reliability of the electromagnetic reflection feature.

[0048] In this embodiment, first, the acoustical calibration parameter, the electrical calibration parameter, and the optical distortion compensation parameter are loaded through a multi-channel coupler. Secondly, based on the real-time attitude angular velocity of the UAV, the spatial projection offset of the acoustic feature stream is calculated using the spatial coordinate transformation model. At the same time, based on the geomagnetic azimuth deviation, the optical axis pointing compensation of the optical feature stream is calculated through the geometric projection model. Finally, based on the height change rate of the UAV, the vertical gradient compensation of the electrical feature stream is calculated using the electromagnetic field distribution model.

[0049] Step 205: Superimpose the spatial projection offset onto the spatio-temporal coordinates of the acoustic feature stream to generate a calibrated acoustic feature stream, superimpose the optical axis pointing compensation onto the image distortion area of the optical feature stream to generate a calibrated optical feature stream, and superimpose the vertical gradient compensation onto the spatio-temporal coordinates of the electrical feature stream to generate a calibrated electrical feature stream. In this step, the calibrated acoustic feature stream refers to the corrected data stream generated by superimposing the spatial projection offset on the spatio-temporal coordinates of the original acoustic feature stream, which is used to eliminate the sound source localization error caused by the attitude change of the UAV. The calibrated optical feature stream refers to the geometric correction image stream generated by superimposing the optical axis pointing compensation amount on the image distortion area, which is used to restore the target deformation caused by the azimuth deviation or thermal airflow disturbance of the UAV. The calibrated electrical feature stream refers to the corrected data stream generated by superimposing the vertical gradient compensation amount on the spatio-temporal coordinates of the electromagnetic field, which is used to compensate for the deviation of the electromagnetic field strength distribution caused by the altitude change.

[0050] In this embodiment, first, based on the spatial projection offset, the three-dimensional geographical coordinates of the sound source in the acoustic feature stream are adjusted through a coordinate transformation model. For example, the original sound source position is translated according to the offset caused by the pitch angular velocity, so that the positioning result is consistent with the real spatial position. Secondly, based on the optical axis pointing compensation amount, a bilinear interpolation algorithm is used to perform pixel-level geometric correction on the distortion area of the optical feature stream. For example, the image center offset area caused by the yaw angle deviation is realigned to the actual target position. Finally, based on the vertical gradient compensation amount, the spatio-temporal coordinates of the electromagnetic pulse in the electrical feature stream are corrected through a field strength distribution model. For example, the electromagnetic field strength is gradient-compensated according to the lifting speed to improve the detection accuracy of the reflected signal of the metal obstacle.

[0051] Step 206: Normalize and fuse the spatio-temporal coordinate parameters of the calibrated acoustic feature stream, optical feature stream, and electrical feature stream to generate a calibrated feature stream. In this step, the normalized fusion refers to the multi-modal data integration performed after converting the spatio-temporal coordinate parameters of the calibrated acoustic, optical, and electrical feature streams into a unified dimension and eliminating the magnitude difference. The calibrated feature stream is a standardized data stream with consistent spatio-temporal benchmarks and correlated physical features generated by fusion, which contains the collaborative information of the sound source localization coordinates, the flame contour image, and the electromagnetic obstacle mapping.

[0052] In this embodiment, first, the sound source coordinates of the acoustic feature stream, the image geographical tags of the optical feature stream, and the electromagnetic target coordinates of the electrical feature stream are unified to the UTM (Universal Transverse Mercator Projection) coordinate system through a geographical coordinate system transformation to eliminate the geographical reference differences of different sensor data. Secondly, the Z-score standardization method is used to normalize the acoustic vibration intensity, the optical pixel brightness, and the electromagnetic field strength, so that the numerical ranges of the multi-modal parameters are unified to the interval [-1, 1]. Finally, the normalized acoustic, optical, and electrical features are superimposed according to the preset weights through a weighted fusion algorithm to generate a calibrated feature stream with spatio-temporal synchronization and comparable physical quantities.

[0053] In some embodiments, according to step 203, based on the product relationship between the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter, perform time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream to generate acoustical calibration parameters with aligned time axes, including: Step 301, calculate the time-domain stretching factor of the low-frequency vibration peak time series in the acoustic feature stream based on the product value of the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter; In this embodiment, first multiply the acoustic-electric collaborative time deviation value by the optical distortion compensation parameter to obtain the time-domain stretching factor; secondly, perform time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream, for example, use a linear interpolation algorithm to expand or compress the time axis length according to the stretching factor to synchronize the time reference of the acoustic signal with the time axis of the image after optical distortion compensation; finally, output the adjusted acoustic feature stream time series as calibration parameters for subsequent multi-modal data fusion.

[0054] Step 302, perform non-uniform interpolation processing on the low-frequency vibration peak time series, and dynamically scale the interval time of the vibration peaks according to the time-domain stretching factor to generate an interpolated vibration peak time series; In this step, non-uniform interpolation processing refers to dynamically adjusting the time interval of the low-frequency vibration peak time series in the acoustic feature stream according to the time-domain stretching factor, rather than interpolation with a fixed step size; the time-domain stretching factor is the product value of the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter calculated in step 301, which is used to determine the scaling ratio for different time periods; dynamic scaling refers to non-linearly adjusting the interval time of the vibration peaks according to the stretching factor, for example, expanding or compressing the intervals of specific time periods; the interpolated vibration peak time series refers to a new time series generated through non-uniform interpolation processing and having a time axis synchronized with the optical and electromagnetic data.

[0055] In this embodiment, first obtain the time-domain stretching factor, which reflects the time ratio by which the acoustic signal needs to be stretched or compressed; secondly, divide the original low-frequency vibration peak time series into multiple time periods according to the stretching factor, and calculate the corresponding scaling ratio for each time period; subsequently, use a linear interpolation algorithm to non-uniformly adjust the peak interval time within each time period, for example, expanding the interval of the time period corresponding to a high stretching factor and compressing the interval of the time period with a low stretching factor; finally, generate an interpolated vibration peak time series whose time axis is aligned with the image sequence after optical distortion compensation and the electromagnetic pulse front time series.

[0056] Step 303, extract the phase difference between adjacent vibration peaks from the interpolated vibration peak time series, and generate a phase correction parameter for the vibration peaks based on the product relationship between the phase difference and the optical distortion compensation parameter; In this step, the phase difference between adjacent vibration peaks refers to the time interval difference between two consecutive peaks in the interpolated vibration peak time series, which is used to characterize the local time series fluctuation of the acoustic signal; the optical distortion compensation parameter refers to the image deformation correction coefficient generated in Step 202, which is used to quantify the time-axis compensation requirement of optical data; the phase correction parameter of the vibration peak is a local time adjustment coefficient generated by multiplying the phase difference by the optical distortion compensation parameter, which is used to eliminate the phase distortion of the acoustic signal caused by the movement of the UAV.

[0057] In this embodiment, first, the phase difference between adjacent peaks is extracted from the interpolated vibration peak time series. For example, the time interval difference between each pair of peaks is calculated through differential operation; second, each phase difference is multiplied by the optical distortion compensation parameter to obtain the phase correction parameter for the corresponding time period; finally, a set of phase correction parameters matching the length of the vibration peak time series is generated for subsequent convolution operations.

[0058] Step 304: Perform a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change of the UAV's pitch angle, and generate a vibration peak time series with phase alignment. In this step, the convolution operation refers to a sliding window multiplication and accumulation operation of using the phase correction parameter as the convolution kernel weight and the interpolated vibration peak time series; the phase distortion refers to the non-linear offset of the time axis of the acoustic signal caused by the sudden change of the UAV's pitch angle; the vibration peak time series with phase alignment refers to the acoustic feature stream generated after eliminating the distortion through convolution and having a time axis strictly synchronized with the optical and electromagnetic data.

[0059] In this embodiment, first, a convolution kernel with the phase correction parameter as the weight is constructed. For example, a Gaussian kernel function is used to smooth the correction parameter; second, the convolution kernel is convolved with the interpolated vibration peak time series. For example, the high-frequency jitter on the time axis is eliminated through sliding window multiplication and accumulation; finally, the vibration peak time series with phase alignment is output, and its time interval is completely synchronized with the image sequence and electromagnetic pulse events after optical distortion compensation.

[0060] Step 305: Calculate the time-axis aligned acoustic calibration parameters based on the vibration peak time series with phase alignment and the original spectral distribution of the acoustic feature stream, where the acoustic calibration parameters include the time-domain remapping coefficient and phase compensation factor of the vibration peak. In this step, the original spectral distribution refers to the frequency-energy distribution generated by the Fourier transform of the low-frequency vibration signal in the acoustic feature stream, reflecting the frequency component characteristics of the vibration signal; the time-domain remapping coefficient refers to the time-axis scaling ratio calculated based on the correlation between the vibration peak time series after alignment on the time axis and the original spectrum, and is used to correct the time reference of the acoustic signal; the phase compensation factor refers to the correction parameter generated by comparing the phase difference between the aligned time series and the original spectrum, and is used to eliminate the phase shift of the frequency components.

[0061] In this embodiment, first, perform a short-time Fourier transform (STFT) on the vibration peak time series with phase alignment to extract its spectral distribution; second, perform a correlation analysis on the spectral distribution of the original acoustic feature stream and the aligned spectrum to calculate the frequency-energy matching degree between the two; then, generate a time-domain remapping coefficient based on the matching degree (for example, fitting the time-axis scaling ratio by the least squares method), and generate a phase compensation factor by comparing the phase difference between the original spectrum and the aligned spectrum; finally, combine the time-domain remapping coefficient and the phase compensation factor into an acoustic calibration parameter.

[0062] In some embodiments, according to what is described in step 203, perform a time shift compensation process on the transient pulse front time series in the electrical feature stream to generate a time-axis aligned electrical calibration parameter, including: Step 401, perform a sliding window detection on the transient pulse front time series, extract the time difference between adjacent pulse fronts, and perform a proportional operation on the time difference and the acoustic-electric collaborative time deviation value to generate a dynamic time shift factor; In this step, the sliding window detection refers to segmenting the transient pulse front time series by a fixed time length and analyzing the time interval between adjacent pulses segment by segment; the time difference between adjacent pulses refers to the time difference between adjacent pulse fronts within the same window, and is used to characterize the local timing fluctuation of the electromagnetic reflection event; the dynamic time shift factor is an adjustment coefficient obtained by performing a proportional operation (such as division or multiplication) on the time difference between adjacent pulses and the acoustic-electric collaborative time deviation value generated in step 201, and is used to quantify the pulse timing compensation requirements in different time periods.

[0063] In this embodiment, first, segment the transient pulse front time series into multiple sliding windows (for example, each window contains three pulses), and calculate the time difference between adjacent pulses within each window; second, perform a proportional operation on each time difference and the acoustic-electric collaborative time deviation value, for example, calculate the dynamic time shift factor by division, indicating the scaling ratio of the pulse interval in the current time period; finally, output a set of dynamic time shift factors corresponding to the time windows for subsequent interpolation processing.

[0064] Step 402: Construct a time shift compensation function based on the dynamic time shift factor, and perform non-uniform interpolation on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series; In this step, the time shift compensation function refers to a time axis adjustment function (such as a linear or non-linear mapping relationship) constructed based on the dynamic time shift factor, which is used to map the original pulse front time series to the calibrated time axis; non-uniform interpolation refers to dynamically scaling and interpolating the pulse time points according to the time shift compensation function, for example, expanding or compressing the intervals of different time periods; the interpolated pulse front time series refers to the electromagnetic feature stream whose time axis is synchronized with the acoustic and optical data generated through non-uniform interpolation.

[0065] In this embodiment, first, construct a piecewise linear time shift compensation function according to the dynamic time shift factor. For example, a high-order interpolation function is used for the time period corresponding to the high dynamic time shift factor; second, perform non-uniform interpolation on the time shift compensation function and the transient pulse front time series. For example, use the cubic spline interpolation algorithm to dynamically adjust the pulse time points according to the compensation function; finally, generate the interpolated pulse front time series.

[0066] Step 403: Extract the pulse front steepness parameter from the interpolated pulse front time series, and perform a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate the pulse waveform distortion caused by multipath reflection of metal obstacles, and generate a pulse time series with corrected waveform; In this step, the pulse front steepness parameter refers to the slope of the pulse rising edge in the interpolated pulse front time series, which is used to characterize the distortion degree of the electromagnetic reflection waveform; the dynamic time shift factor refers to the pulse interval scaling ratio coefficient generated in step 401, which is used to quantify the timing compensation requirements of different time periods; multipath reflection refers to the waveform superposition distortion caused by the multiple reflections of electromagnetic waves by metal obstacles; the pulse time series with corrected waveform refers to the electromagnetic feature stream with clear waveform features generated by eliminating the multipath reflection distortion.

[0067] In this embodiment, first, calculate the steepness parameter of each pulse rising edge in the interpolated pulse front time series through derivative operation; second, construct a convolution kernel weight from the steepness parameter and the dynamic time shift factor; then, filter the pulse waveform through convolution operation to suppress the high-frequency distortion components with abnormal steepness; finally, output the pulse time series with corrected waveform, whose waveform front steepness is synchronized with the time axis of the acoustic and optical data.

[0068] Step 404: Calculate the electrically calibrated parameters with aligned time axes based on the pulse time series with corrected waveform and the original intensity distribution of the electrical feature stream, where the electrically calibrated parameters include the time shift remapping coefficient and the waveform compensation factor of the pulse front; In this step, the original intensity distribution refers to the amplitude-time distribution of electromagnetic pulses in the electrical characteristic stream, reflecting the electromagnetic energy characteristics reflected by the obstacle; the time-shift remapping coefficient refers to the time-series scaling ratio calculated based on the time-axis offset between the pulse time series after waveform correction and the original intensity distribution; the waveform compensation factor refers to the amplitude correction parameter generated by comparing the spectral differences between the corrected waveform and the original waveform, and is used to eliminate the energy attenuation caused by multipath reflection.

[0069] In this embodiment, first, perform short-time energy integration on the pulse time series after waveform correction to extract its intensity distribution; second, perform correlation analysis on the intensity distribution of the original electrical characteristic stream and the corrected distribution, calculate the time-axis offset, and generate the time-shift remapping coefficient; then, generate the waveform compensation factor through spectral comparison analysis; finally, combine the time-shift remapping coefficient and the waveform compensation factor into the electrical calibration parameter.

[0070] In some embodiments, according to step 102, synchronously associate the acoustic frequency band signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the unmanned aerial vehicle to generate an acoustic characteristic stream, an optical characteristic stream, and an electrical characteristic stream, including: Step 501, perform time-frequency analysis on the acoustic frequency band signal, extract the low-frequency component that matches the vibration frequency of the unmanned aerial vehicle's rotor, and perform spatial projection conversion on the low-frequency component and the real-time three-dimensional coordinates of the unmanned aerial vehicle to generate the vibration characteristic parameters with position marks carried in the acoustic data stream. In this step, the time-frequency analysis refers to extracting the frequency-time distribution characteristics of the acoustic frequency band signal through short-time Fourier transform or wavelet transform; the low-frequency component refers to the acoustic signal component that matches the vibration frequency range of the unmanned aerial vehicle's rotor, such as the mechanical vibration characteristics in the frequency band of 100 Hz to 500 Hz; the spatial projection conversion refers to combining the physical characteristics of the acoustic signal (such as vibration energy) with the real-time three-dimensional coordinates (longitude, latitude, altitude) of the unmanned aerial vehicle to generate the vibration characteristic parameters with geographical marks carried in the acoustic data stream, such as mapping the vibration energy peak within a range of 10 meters of the current position of the unmanned aerial vehicle.

[0071] In this embodiment, first, perform short-time Fourier transform on the acoustic frequency band signal to extract the low-frequency energy distribution of 100 Hz to 500 Hz; second, obtain its real-time three-dimensional coordinates through the flight control system of the unmanned aerial vehicle, and use the spherical projection model to map the acoustic vibration energy peak to the geographical space to generate the vibration characteristic parameters with position marks for multi-modal fusion analysis.

[0072] Step 502: Perform inter-frame attitude calculation on the visible light image sequence, calculate the optical axis pointing vector of each frame in the visible light image sequence based on the pitch angle and yaw angle parameters of the UAV gimbal, and perform spherical coordinate system mapping on the optical axis pointing vector and the flight altitude of the UAV to generate the perspective compensation parameters with attitude marks carried in the optical data stream. In this step, inter-frame attitude calculation refers to calculating the relative motion parameters of the UAV through feature point matching (such as SIFT or optical flow method) of adjacent image frames; the optical axis pointing vector refers to the direction vector of the image center optical axis in three-dimensional space generated based on the gimbal pitch angle and yaw angle; spherical coordinate system mapping refers to combining the optical axis pointing vector and the UAV flight altitude and converting them into spherical coordinates (azimuth angle, pitch angle, radius) to generate the parameters for compensating perspective distortion in the optical data stream, such as the image edge stretching correction coefficient.

[0073] In this embodiment, first, calculate the attitude change amount between adjacent image frames through the optical flow method, and combine the real-time pitch angle of two degrees and yaw angle of five degrees of the gimbal to generate the optical axis pointing vector of each frame; secondly, input the optical axis vector and the UAV height of twenty meters into the spherical coordinate system model to calculate the geographical azimuth angle and pitch angle corresponding to the image center; finally, generate the perspective compensation parameters, such as radially stretching the pixels at the image edge to correct the perspective distortion caused by the pitch angle.

[0074] Step 503: Perform height attenuation compensation on the electromagnetic field intensity change data, dynamically adjust the electromagnetic field gradient threshold according to the relative distance between the UAV and the ground obstacle, and generate the field strength correction factor with height marks carried in the electrical data stream. In this step, height attenuation compensation refers to performing energy correction on the electromagnetic field intensity data collected by the UAV according to the electromagnetic wave propagation characteristics to eliminate the signal attenuation error caused by the change in flight altitude; the relative distance refers to the real-time vertical distance between the UAV and the ground obstacle, which is obtained through a height sensor or a laser ranging device; the electromagnetic field gradient threshold refers to the critical value for determining the field strength change dynamically adjusted according to the relative distance, which is used to distinguish effective reflection signals from environmental noise; the field strength correction factor refers to the electromagnetic field intensity correction coefficient generated by compensating for height attenuation and filtering noise, which is used to improve the spatial consistency of the electrical data stream.

[0075] In this embodiment, first, based on the physical law of electromagnetic wave attenuation, combine the real-time height data of the UAV to perform reverse compensation on the electromagnetic field intensity to restore the true field strength of the obstacle reflection signal; secondly, dynamically adjust the gradient threshold according to the relative distance between the UAV and the obstacle to suppress the random interference generated by metal debris or cables during low-altitude flight; finally, generate the field strength correction factor with height marks carried, which is used for subsequent multi-modal data fusion.

[0076] Step 504: Input the vibration characteristic parameters, the perspective compensation parameters, and the field strength correction factor into a pre-trained spatio-temporal coupling model. Based on the motion acceleration and angular velocity parameters of the UAV, calculate the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream. In this step, the spatio-temporal coupling model refers to a dynamic calibration model established through a multi-modal data association algorithm, which is used to analyze the spatio-temporal correlation between acoustic, optical, and electrical data streams; the vibration characteristic parameters refer to the acoustic vibration energy distribution carrying position marks generated in step 501; the perspective compensation parameters refer to the spherical coordinate mapping coefficients generated in step 502 for correcting optical image distortion; the field strength correction factor refers to the electromagnetic field strength correction coefficient carrying altitude marks generated in step 503; the motion acceleration and angular velocity parameters refer to the three-axis acceleration and angular velocity data output in real time by the UAV inertial measurement unit, which are used to quantify the influence of flight attitude dynamic changes on multi-modal data; the time synchronization deviation refers to the time sequence offset caused by differences in sensor sampling rates or environmental interference between the acoustic and optical data streams; the spatial distortion coefficient refers to the amount of spatial coordinate misalignment between the optical and electrical data streams caused by UAV rotation or displacement.

[0077] In this embodiment, first, input the vibration characteristic parameters, the perspective compensation parameters, and the field strength correction factor into the spatio-temporal coupling model, where the vibration characteristic parameters contain the spatial distribution information of acoustic vibration energy, the perspective compensation parameters contain the spherical coordinate mapping relationship of the optical axis pointing vector, and the field strength correction factor contains the electromagnetic field strength gradient after altitude compensation; second, based on the motion acceleration and angular velocity parameters of the UAV, calculate the spatio-temporal influence weights of attitude changes on multi-modal data through kinematic equations; then, use the correlation analysis algorithm to calculate the time synchronization deviation between the acoustic and optical data streams, for example, detect the time offset between the vibration energy peak and the change in the flame contour through the cross-correlation function; at the same time, calculate the spatial distortion coefficient between the optical and electrical data streams caused by sudden changes in the pitch angle through the spatial projection transformation model; finally, output the time synchronization deviation and the spatial distortion coefficient as the calibration basis for multi-modal data fusion.

[0078] Step 505: Perform Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate an acoustical intermediate data stream with aligned time axes; perform perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate an optical intermediate data stream with aligned spatial axes. In this step, Doppler frequency shift compensation refers to dynamically correcting the frequency shift of the acoustic signal caused by the movement of the UAV according to the time synchronization deviation between the acoustic data stream and the optical data stream; the acoustic intermediate data stream with time axis alignment refers to the characteristic stream in which the frequency and time sequence of the compensated acoustic signal are strictly synchronized with the optical data; perspective projection correction refers to correcting the geometric deformation of the optical image caused by the sudden change of the UAV attitude based on the spatial distortion coefficient; the optical intermediate data stream with spatial axis alignment refers to the characteristic stream in which the spatial coordinates of the optical image and the electrical data are consistent after eliminating the perspective distortion.

[0079] In this embodiment, first, based on the time synchronization deviation, Doppler frequency shift compensation is performed on the acoustic data stream through a dynamic frequency adjustment algorithm. For example, the sliding window Fourier transform is used to track the frequency offset in real time and correct it in the reverse direction. Secondly, a perspective projection matrix is constructed according to the spatial distortion coefficient, and the optical image is re-projected through a geometric transformation model. For example, the image with an inclined perspective is corrected to a front view projection using an affine transformation. Finally, the acoustic intermediate data stream with time axis alignment and the optical intermediate data stream with spatial axis alignment are output for subsequent calls by the multimodal fusion engine.

[0080] Step 506: Based on the correlation between the field strength correction factor and the UAV altitude change rate, perform vertical direction field strength gradient compensation on the electrical data stream to generate an electrical intermediate data stream that is spatio-temporally matched with the acoustic intermediate data stream and the optical intermediate data stream. In this step, the field strength correction factor refers to the electromagnetic field strength correction coefficient generated in step 503, which is used to compensate for the field strength loss caused by altitude attenuation; the UAV altitude change rate refers to the vertical movement speed of the UAV obtained by a barometer or a laser rangefinder, which characterizes the dynamic change of the aircraft in the vertical direction; the vertical direction field strength gradient compensation refers to dynamically adjusting the field strength distribution of the electrical data stream based on the altitude change rate to eliminate the vertical gradient error caused by altitude fluctuations; the electrical intermediate data stream refers to the electromagnetic characteristic stream that is spatio-temporally matched with the acoustic intermediate data stream and the optical intermediate data stream after vertical gradient compensation, and includes the field strength spatial distribution and time sequence characteristics with consistent altitude.

[0081] In this embodiment, first, an association model between the field strength correction factor and the UAV altitude change rate is established. For example, the dynamic relationship between the two is analyzed through linear regression. Secondly, the vertical direction field strength gradient compensation amount is calculated according to the real-time altitude change rate. For example, when the UAV ascends, the low-altitude field strength data is enhanced proportionally. Subsequently, the compensation amount is superimposed on the field strength distribution of the electrical data stream. For example, an electromagnetic field distribution with continuous vertical gradient is generated through a spatial interpolation algorithm. Finally, the compensated electrical data stream is spatio-temporally matched with the acoustic intermediate data stream and the optical intermediate data stream generated in step 505 to ensure that the time stamps of the three are aligned and the spatial coordinates are consistent, and an electrical intermediate data stream is generated.

[0082] Step 507: Bind the acoustic intermediate data stream with the vibration characteristic parameters to generate an acoustic characteristic stream, bind the optical intermediate data stream with the perspective compensation parameters to generate an optical characteristic stream, and at the same time bind the electrical intermediate data stream with the field strength correction factor to generate an electrical characteristic stream; In this step, the acoustic characteristic stream refers to a structured data stream obtained by binding the time-axis aligned acoustic intermediate data stream with vibration characteristic parameters (such as peak vibration energy, frequency distribution), which includes the physical characteristics of acoustic signals and their spatio-temporal calibration information; the optical characteristic stream refers to a structured data stream obtained by binding the space-axis aligned optical intermediate data stream with perspective compensation parameters (such as optical axis pointing vector, perspective correction coefficient), which includes the geometric characteristics of optical images and their dynamic attitude calibration information; the electrical characteristic stream refers to a structured data stream obtained by binding the spatio-temporally matched electrical intermediate data stream with the field strength correction factor (such as height compensation coefficient, vertical gradient parameter), which includes the electromagnetic field strength distribution and its dynamic height calibration information.

[0083] In this embodiment, first, perform data association on the time-domain aligned vibration signals in the acoustic intermediate data stream and the vibration characteristic parameters to generate an acoustic characteristic stream carrying complete spatio-temporal calibration tags; second, bind the corrected images in the optical intermediate data stream with the perspective compensation parameters to generate an optical characteristic stream carrying attitude compensation tags; at the same time, bind the vertical gradient compensated field strength data in the electrical intermediate data stream with the field strength correction factor to generate an electrical characteristic stream carrying height calibration tags.

[0084] In some embodiments, according to what is described in step 104, perform anti-interference fusion processing on the calibrated characteristic stream according to the correlation mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field strength change data in the target area, and extract the multi-dimensional coupling characteristics in the fusion result, including: Step 601: Construct a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and perform per-channel convolution operations on the acoustic calibration characteristic stream, optical calibration characteristic stream, and electrical calibration characteristic stream in the calibrated characteristic stream with the reflection correlation weight matrix respectively to generate acoustic reflection correlation characteristics, optical reflection correlation characteristics, and electromagnetic reflection correlation characteristics; In this step, the reflection correlation weight matrix refers to a weight distribution model constructed based on the reflection characteristics of surface obstacles, which is used to quantify the reflection intensity correlation relationship of different materials to sound, light, and electromagnetic waves; the per-channel convolution operation refers to performing independent convolution processing on the acoustic, optical, and electrical characteristic streams respectively to extract the correlation characteristics between each modality data and the material reflection characteristics; the acoustic / optical / electromagnetic reflection correlation characteristics refer to the multi-modal data characteristics carrying reflection characteristic tags generated by convolution, such as the acoustic vibration energy aggregation characteristics corresponding to the high-reflection area of a metal obstacle.

[0085] In this embodiment, first, a reflection correlation weight matrix is constructed based on a preset reflection characteristic table (for example, the reflection intensity of metal materials is high, and the reflection intensity of concrete materials is low); secondly, the acoustic, optical, and electrical feature streams in the calibration feature stream are respectively subjected to per-channel convolution operations with the weight matrix. For example, the acoustic feature stream extracts the correlation between vibration energy and metal reflection through temporal convolution, the optical feature stream enhances the texture contrast of the metal contour through spatial convolution, and the electrical feature stream highlights the change in the field strength gradient in the metal area through spatio-temporal convolution; finally, acoustic reflection correlation features (such as the aggregation of metal vibration energy), optical reflection correlation features (such as the sharpened image of the metal pipe edge), and electromagnetic reflection correlation features (such as the sudden increase in the field strength gradient in the metal area) are output.

[0086] Step 602: Calculate a dynamic attenuation factor for the electromagnetic field strength change data, and perform weighted superposition of the dynamic attenuation factor and the electromagnetic reflection correlation features to generate an electromagnetic anti-interference feature stream, where the dynamic attenuation factor is generated according to the product relationship between the real-time altitude of the UAV and the reflection correlation weight matrix; In this step, the dynamic attenuation factor refers to the field strength attenuation compensation coefficient generated according to the product relationship between the UAV flight altitude and the reflection correlation weight matrix, which is used to suppress the interference of low-altitude environmental noise on electromagnetic data; weighted superposition means fusing the dynamic attenuation factor and the electromagnetic reflection correlation features according to weights to eliminate the influence of multipath reflection or random interference; the electromagnetic anti-interference feature stream refers to the anti-noise and highly robust electromagnetic field data stream generated after superposition.

[0087] In this embodiment, first, calculate the dynamic attenuation factor according to the weight relationship between the real-time altitude of the UAV (such as high-altitude or low-altitude flight state) and the weight of the metal material in the reflection correlation weight matrix; secondly, perform weighted superposition of the attenuation factor and the electromagnetic reflection correlation features, for example, enhancing the field strength gradient in the metal area while suppressing the noise interference in the non-metal area; finally, generate an electromagnetic anti-interference feature stream, in which the metal reflection field strength gradient is kept intact, and the interference signals in the non-metal area are effectively filtered.

[0088] Step 603: Extract the low-frequency vibration energy distribution parameters in the acoustic reflection correlation features, and perform a cross-modal product operation with the image contour gradient parameters in the optical reflection correlation features to generate a vibration contour coupling coefficient; In this step, the low-frequency vibration energy distribution parameter refers to the distribution characteristic of the vibration energy in a specific frequency band over time extracted from the acoustic reflection correlation features, which is used to characterize the mechanical vibration intensity caused by metal structure deformation or combustion; the image contour gradient parameter refers to the amplitude of the image edge gradient extracted from the optical reflection correlation features, which is used to quantify the sharpness of the metal obstacle contour; the vibration contour coupling coefficient refers to the correlation parameter of the acoustic vibration energy and the optical contour gradient generated through the cross-modal product operation, which reflects the collaborative characteristics of mechanical vibration and object geometric deformation.

[0089] In this embodiment, first, short-time Fourier transform is performed on the acoustic reflection correlation features to extract low-frequency vibration energy distribution parameters (such as the proportion of the main frequency energy of the metal pipeline vibration); second, the Sobel operator is used for the optical reflection correlation features to extract image contour gradient parameters (such as the gradient amplitude of the metal pipeline edge); subsequently, after normalizing the two types of parameters, point-by-point multiplication operations are performed according to the time window to generate a vibration contour coupling coefficient (such as the energy-contour synergy coefficient of the metal area).

[0090] Step 604: Perform spatio-temporal consistency matching on the vibration contour coupling coefficient and the electromagnetic anti-interference feature stream, calculate the multi-modal fusion weight based on the curvature parameter of the UAV flight trajectory and the geomagnetic azimuth deviation, and generate a fusion feature matrix including the acoustic vibration energy distribution, the optical contour gradient, and the electromagnetic anti-interference intensity; In this step, spatio-temporal consistency matching means aligning the spatio-temporal coordinates of the vibration contour coupling coefficient and the electromagnetic anti-interference feature stream to ensure that the timestamps and spatial positions of the acoustic, optical, and electromagnetic data are consistent; the curvature parameter of the UAV flight trajectory refers to the degree of bending of the flight path calculated by the inertial navigation system, which is used to quantify the impact of dynamic motion on multi-modal data; the geomagnetic azimuth deviation refers to the angle deviation between the actual heading of the UAV and the geomagnetic north pole, which is used to correct the spatial pointing error of the electromagnetic data; the multi-modal fusion weight refers to the fusion ratio of the acoustic, optical, and electromagnetic data dynamically allocated according to the curvature and azimuth deviation; the fusion feature matrix refers to a multi-dimensional data set integrating the acoustic vibration energy distribution, the optical contour gradient, and the electromagnetic anti-interference intensity, which is used for dynamic modeling of the fire scene.

[0091] In this embodiment, first, the vibration contour coupling coefficient and the electromagnetic anti-interference feature stream are aligned according to the timestamp and UTM coordinates; second, based on the curvature of the UAV flight trajectory (such as a high curvature during a sharp turn) and the geomagnetic azimuth deviation (such as a yaw angle deviation of five degrees), the fusion weight is dynamically allocated through a fuzzy logic model (such as reducing the optical weight and increasing the electromagnetic weight at high curvature); finally, the fusion feature matrix is generated by weighted superposition according to the weight.

[0092] Step 605: Perform multi-scale decomposition on the fusion feature matrix, extract the obstacle position distribution features in the low-frequency component and the moving target trajectory features in the high-frequency component, perform time-domain correlation superposition on the position distribution features and the trajectory features, and generate the multi-dimensional coupling features, where the multi-dimensional coupling features include the obstacle spatial topological relationship and the moving target motion vector parameters; In this step, multi-scale decomposition refers to decomposing the fused feature matrix into signal layers of different frequency components through wavelet transform or similar algorithms to separate the low-frequency global structure from the high-frequency transient changes; the obstacle position distribution characteristics in the low-frequency component refer to the spatial distribution information of static obstacles (such as metal pipes, concrete walls) extracted from the low-frequency signal; the moving target trajectory characteristics in the high-frequency component refer to the motion path information of dynamic targets (such as rescue personnel, scattered burning objects) extracted from the high-frequency signal; time-domain correlation superposition refers to aligning and fusing the static obstacle positions and dynamic target trajectories along the time axis; multi-dimensional coupled features refer to the composite data stream integrating the spatial topological relationship of obstacles (such as the relative positions of metal pipes and walls) and the motion vector parameters of moving targets (such as speed, direction).

[0093] In this embodiment, first, perform wavelet multi-scale decomposition on the fused feature matrix, extract the obstacle position distribution in the low-frequency layer signal (such as the metal pipe in the northeast corner of the fifth floor) and the moving target trajectory in the high-frequency layer signal (such as the rescue personnel moving northwest); secondly, superimpose the low-frequency obstacle positions and high-frequency moving trajectories through timestamp alignment, for example, bind the metal pipe position and the personnel movement speed vector in each frame of data; finally, generate multi-dimensional coupled features, including the spatial topological network of static obstacles and the real-time motion parameters of dynamic targets.

[0094] In some embodiments, according to what is described in step 101, synchronously collect the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the unmanned aerial vehicle in the target area, including: Step 701, based on the obstacle spatial topological relationship parameters in the multi-dimensional coupled features, perform three-dimensional grid division on the target area, and generate a dynamic obstacle distribution model of the terrain surface according to the mapping relationship between the flight height of the unmanned aerial vehicle and the obstacle reflection coefficient; In this step, three-dimensional grid division refers to decomposing the target area into a three-dimensional grid composed of cubic units according to the spatial resolution, which is used to quantify the distribution density and geometric attributes of obstacles; the obstacle spatial topological relationship parameters refer to the relative positions and connection relationships between static obstacles (such as metal pipes, walls) described in the multi-dimensional coupled features; the dynamic obstacle distribution model refers to a real-time updated three-dimensional space model of obstacles generated based on the mapping relationship between the flight height of the unmanned aerial vehicle and the obstacle reflection coefficient, reflecting the material reflection characteristics and distribution dynamics of obstacles in different height layers.

[0095] In this embodiment, first, a three-dimensional grid division is performed on the target area according to the obstacle space topology relationship parameter in the multi-dimensional coupling features. For example, the fifth floor of the fire scene is divided into cubic grids with a side length of one meter. Secondly, based on the mapping relationship between the real-time flight altitude of the UAV and the obstacle reflection coefficient (such as a high reflection coefficient for metal and a low reflection coefficient for concrete), the distribution weight of obstacles within the grid cells is dynamically adjusted. For example, when flying at a low altitude, the grid weight of metal obstacles is enhanced. Finally, a dynamic obstacle distribution model is generated to real-time annotate the three-dimensional positions and reflection intensities of obstacles such as metal pipes and burning debris.

[0096] Step 702: Extract the moving target motion vector parameter in the multi-dimensional coupling features, and combine it with the real-time gradient distribution of the electromagnetic field strength change data to dynamically predict the motion trajectory of the moving target, and generate trajectory prediction parameters including the velocity direction and collision probability. In this step, the moving target motion vector parameter refers to the velocity and direction data of the moving target (such as rescue personnel, flying burning debris) extracted from the multi-dimensional coupling features; the real-time gradient distribution of the electromagnetic field strength change data refers to the spatial gradient information of the electromagnetic field strength in the anti-interference electromagnetic feature flow; the dynamic probability prediction refers to predicting the future trajectory and its collision risk of the moving target by fusing the motion vector and electromagnetic gradient data; the trajectory prediction parameter refers to the composite parameter including the target velocity direction, path probability, and collision probability in the output result.

[0097] In this embodiment, first, the motion vector of the moving target is extracted from the multi-dimensional coupling features. Secondly, combined with the electromagnetic field strength gradient distribution (such as a sudden increase in the electromagnetic field strength gradient in the northwest direction), the probability distribution of the target's future trajectory is predicted through a hidden Markov model or Monte Carlo sampling. Finally, trajectory prediction parameters are generated.

[0098] Step 703: Perform spatial interpolation processing on the dynamic obstacle distribution model, and generate an obstacle height field strength map covering the target area according to the UAV flight speed and pan-tilt angle parameters. At the same time, perform time-domain superposition of the trajectory prediction parameters and the obstacle height field strength map to generate a dynamic threat field strength map. In this step, the spatial interpolation processing refers to converting discrete obstacle distribution data into a continuous spatial field strength distribution through Kriging interpolation or inverse distance weighted algorithm; the obstacle height field strength map refers to a two-dimensional grid map reflecting the reflection field strength distribution of obstacles in different height layers within the target area; the dynamic threat field strength map refers to a three-dimensional space model generated by superposing the trajectory prediction parameters and the obstacle height field strength map along the time axis, which contains real-time threat levels and is used to characterize the comprehensive threat of dynamic obstacles and moving targets to the UAV.

[0099] In this embodiment, first, spatial interpolation processing is performed on the dynamic obstacle distribution model to fill the data missing due to the sensor blind area, and an obstacle height field strength map covering the target area is generated; secondly, the resolution of the field strength map is adjusted based on the UAV flight speed and gimbal viewing angle parameters. For example, the resolution is reduced during high-speed flight to improve processing efficiency; finally, the trajectory prediction parameters (such as the collision probability of a moving target) and the obstacle height field strength map are superimposed according to the time stamp to generate a dynamic threat field strength map, where high-threat areas are marked with the superimposed characteristics of a sudden increase in the field strength gradient and an increase in the collision probability.

[0100] Step 704, based on the field strength gradient distribution in the dynamic threat field strength map, calculate the path node sequence between the current position of the UAV and the target point through the potential field navigation algorithm in the UAV flight path planning model, and generate an initial obstacle avoidance path; In this step, the potential field navigation algorithm refers to a path planning method that calculates the movement direction of the UAV by simulating gravity (attraction of the target point) and repulsion (repulsion of the threat field strength); the path node sequence refers to a set of discrete path points generated by the algorithm from the current position to the target point, which is used to guide the segmented flight of the UAV.

[0101] In this embodiment, first, based on the field strength gradient distribution in the dynamic threat field strength map, calculate the resultant force direction of gravity - repulsion of the UAV's current position and the target point through the potential field algorithm; secondly, generate a path node sequence along the resultant force direction. For example, dense nodes are set in low-threat areas to improve path smoothness, and sparse nodes are set in high-threat areas to quickly cross; finally, output the initial obstacle avoidance path to ensure that the path nodes avoid high-threat field strength areas.

[0102] Step 705, perform electromagnetic interference robustness verification on the initial obstacle avoidance path, and perform weighted evaluation on the safety of path nodes according to the transient pulse peak intensity in the electromagnetic field strength change data and the collision probability in the trajectory prediction parameters, and generate an optimal path node sequence including threat avoidance weights; In this step, electromagnetic interference robustness verification refers to evaluating the stability of path nodes in a strong electromagnetic interference environment by analyzing the transient pulse peak intensity in the electromagnetic field strength change data; the threat avoidance weight refers to the safety level of path nodes calculated according to the collision probability and electromagnetic interference intensity, and the higher the weight, the safer the node.

[0103] In this embodiment, first, detect the electromagnetic interference intensity of each node of the initial obstacle avoidance path. For example, detect whether the transient pulse peak exceeds the safety threshold; secondly, combine the collision probability in the trajectory prediction parameters, and calculate the node threat avoidance weight through a weighted evaluation model; finally, screen the nodes with weights higher than the safety threshold to generate an optimal path node sequence.

[0104] Step 706: Perform three-dimensional spatial fusion of the optimal path node sequence and the dynamic threat field strength map to generate a three-dimensional spatial situation model covering the target area, where the three-dimensional spatial situation model includes obstacle elevation distribution, moving target trajectory prediction, and electromagnetic interference hot zone marking; In this step, the three-dimensional spatial situation model refers to a three-dimensional environmental perception model generated by fusing the optimal path node sequence and the dynamic threat field strength map, including obstacle elevation distribution, moving target trajectory prediction, and electromagnetic interference hot zone marking; the obstacle elevation distribution refers to the height stratification information of obstacles in the vertical direction; the moving target trajectory prediction refers to the dynamically updated movement paths of rescue personnel or flying burning objects; the electromagnetic interference hot zone marking refers to the three-dimensional spatial annotation of the abnormal electromagnetic field strength area.

[0105] In this embodiment, first, project the optimal path node sequence into the dynamic threat field strength map through a three-dimensional spatial mapping algorithm to generate the spatial association relationship between the path nodes and the threat field strength; second, integrate the obstacle elevation distribution and the moving target trajectory prediction data through point cloud fusion technology, such as superimposing the height information of metal pipes and the movement path of rescue personnel; finally, mark the electromagnetic interference hot zone as a three-dimensional thermal layer and fuse it with the obstacle and moving target data to generate a three-dimensional spatial situation model covering the target area.

[0106] Figure 2 The structural schematic diagram of a multi-dimensional information fusion system for an unmanned aerial vehicle based on acoustic, optical, and electrical composite detection is provided for the embodiments of the present application, as Figure 2 shown. The system includes: An acquisition module 21, configured to synchronously acquire the acoustic frequency band signals, visible light image sequences, and electromagnetic field strength change data during the movement of the unmanned aerial vehicle in the target area; An association module 22, configured to synchronously associate the acoustic frequency band signals, the visible light image sequences, and the electromagnetic field strength change data with the current position and attitude parameters of the unmanned aerial vehicle respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; A correction module 23, configured to perform multi-channel dynamic calibration on the image distortion area in the optical feature stream based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, and synchronously correct the spatio-temporal coordinate parameters of the acoustic feature stream, the optical feature stream, and the electrical feature stream based on the calibration result to generate a calibrated feature stream; A fusion module 24, configured to perform anti-interference fusion processing on the calibrated feature stream according to the association mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field strength change data, and extract the multi-dimensional coupling features in the fusion result; A generation module 25, configured to input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional spatial situation model covering a target area. Figure 2 The described UAV multi-dimensional information fusion system based on acoustic, optical, and electrical composite detection can execute Figure 1 For the described UAV multi-dimensional information fusion method based on acoustic, optical, and electrical composite detection in the illustrated embodiment, its implementation principle and technical effects will not be elaborated further. For the UAV multi-dimensional information fusion system in the above embodiment, the specific manners in which each module and unit perform operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0107] In a possible design, Figure 2 The UAV multi-dimensional information fusion system in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, where the one or more computer instructions are for the processing component 32 to call and execute.

[0108] The processing component 32 is used for the above Figure 1 The UAV multi-dimensional information fusion method in the illustrated embodiment.

[0109] Among them, the processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for executing the above method.

[0110] The storage component 31 is configured to store various types of data to support operations on the terminal. The storage component may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0111] Of course, the computing device will certainly also include other components, such as input / output interfaces, display components, communication components, etc.

[0112] The input / output interface provides an interface between the processing component and the peripheral interface module, and the peripheral interface module may be an output device, an input device, etc.

[0113] The communication component is configured to facilitate communication between the computing device and other devices in a wired or wireless manner, etc.

[0114] Among them, the computing device may be a physical device or an elastic computing host provided by a cloud computing platform, etc. At this time, the computing device may refer to a cloud server, and the above-mentioned processing component, storage component, etc. may be basic server resources leased or purchased from a cloud computing platform.

[0115] The embodiment of the present application also provides a computer storage medium storing a computer program, and when the computer program is executed by a computer, it can implement the above Figure 1 A method for multi-dimensional information fusion of an unmanned aerial vehicle based on acoustic, optical and electrical composite detection shown in the embodiment.

[0116] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-dimensional information fusion method for unmanned aerial vehicles based on acoustic-optical-electric composite detection, characterized in that Including: Synchronously collecting acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the drone in the target area; Synchronously associating the acoustic frequency band signals, the visible light image sequences, and the electromagnetic field intensity change data with the current position and attitude parameters of the drone respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; Based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, performing multi-channel dynamic calibration on the image distortion area in the optical feature stream, and synchronously correcting the spatio-temporal coordinate parameters of the acoustic feature stream, the optical feature stream, and the electrical feature stream based on the calibration result to generate a calibrated feature stream; According to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field intensity change data, performing anti-interference fusion processing on the calibrated feature stream, and extracting multi-dimensional coupling features in the fusion result; Inputting the multi-dimensional coupling features into a preset drone flight path planning model to generate a three-dimensional space situation model covering the target area.

2. The method according to claim 1, wherein Based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, performing multi-channel dynamic calibration on the image distortion area in the optical feature stream, and synchronously correcting the spatio-temporal coordinate parameters of the acoustic feature stream, the optical feature stream, and the electrical feature stream based on the calibration result to generate a calibrated feature stream, including: Extracting a low-frequency vibration peak time series synchronized with the drone rotor vibration from the acoustic feature stream, and simultaneously separating a transient pulse front time series triggered by obstacle electromagnetic reflection from the electrical feature stream, and calculating the average phase difference between the low-frequency vibration peak time series and the transient pulse front time series to generate an acoustic-electricity collaborative time deviation value; Constructing a dynamic calibration matrix based on the acoustic-electricity collaborative time deviation value, and performing a frame-by-frame convolution operation on the pixel displacement amount of the image distortion area in the optical feature stream with the dynamic calibration matrix to generate optical distortion compensation parameters; According to the product relationship between the acoustic-electricity collaborative time deviation value and the optical distortion compensation parameters, performing time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream to generate acoustical calibration parameters with time axis alignment, and simultaneously performing time shift compensation processing on the transient pulse front time series in the electrical feature stream to generate electrical calibration parameters with time axis alignment; Inputting the acoustical calibration parameters, the electrical calibration parameters, and the optical distortion compensation parameters into a multi-channel coupler, and calculating the spatial projection offset of the acoustic feature stream, the optical axis pointing compensation of the optical feature stream, and the vertical gradient compensation of the electrical feature stream based on the real-time attitude angular velocity and geomagnetic azimuth deviation of the drone; Superimposing the spatial projection offset onto the spatio-temporal coordinates of the acoustic feature stream to generate a calibrated acoustic feature stream, superimposing the optical axis pointing compensation onto the image distortion area of the optical feature stream to generate a calibrated optical feature stream, and superimposing the vertical gradient compensation onto the spatio-temporal coordinates of the electrical feature stream to generate a calibrated electrical feature stream; Normalize and fuse the spatio-temporal coordinate parameters of the calibrated acoustic feature stream, optical feature stream, and electrical feature stream to generate a calibrated feature stream.

3. The method according to claim 2, characterized in that, Perform time-domain stretching processing on the low-frequency vibration peak time series in the acoustic feature stream according to the product relationship between the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter to generate acoustical calibration parameters with aligned time axes, including: Calculate the time-domain stretching factor of the low-frequency vibration peak time series in the acoustic feature stream based on the product value of the acoustic-electric collaborative time deviation value and the optical distortion compensation parameter; Perform non-uniform interpolation processing on the low-frequency vibration peak time series, and dynamically scale the interval time between vibration peaks according to the time-domain stretching factor to generate an interpolated vibration peak time series; Extract the phase difference between adjacent vibration peaks from the interpolated vibration peak time series, and generate a phase correction parameter for the vibration peak according to the product relationship between the phase difference and the optical distortion compensation parameter; Perform a convolution operation on the phase correction parameter and the interpolated vibration peak time series to eliminate the phase distortion of the acoustic signal caused by the sudden change of the UAV pitch angle, and generate a vibration peak time series with aligned phases; Calculate the acoustical calibration parameters with aligned time axes based on the vibration peak time series with aligned phases and the original spectral distribution of the acoustic feature stream, where the acoustical calibration parameters include the time-domain remapping coefficient and phase compensation factor of the vibration peak.

4. The method according to claim 3, wherein Perform time shift compensation processing on the transient pulse front time series in the electrical feature stream to generate electrical calibration parameters with aligned time axes, including: Perform a sliding window detection on the transient pulse front time series, extract the interval time difference between adjacent pulse fronts, and perform a proportional operation on the interval time difference and the acoustic-electric collaborative time deviation value to generate a dynamic time shift factor; Construct a time shift compensation function based on the dynamic time shift factor, and perform non-uniform interpolation processing on the time shift compensation function and the transient pulse front time series to generate an interpolated pulse front time series; Extract the pulse front steepness parameter from the interpolated pulse front time series, and perform a convolution operation on the steepness parameter and the dynamic time shift factor to eliminate the pulse waveform distortion caused by the multi-path reflection of the metal obstacle, and generate a pulse time series with corrected waveform; Calculate the electrical calibration parameters with aligned time axes based on the pulse time series with corrected waveform and the original intensity distribution of the electrical feature stream, where the electrical calibration parameters include the time shift remapping coefficient and waveform compensation factor of the pulse front.

5. The method according to claim 1, characterized in that, Synchronously associate the acoustic waveband signal, the visible light image sequence, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream, including: Perform time-frequency analysis on the acoustic waveband signal, extract the low-frequency component matching the UAV rotor vibration frequency, and perform spatial projection conversion on the low-frequency component and the UAV real-time three-dimensional coordinates to generate vibration feature parameters with position marks in the acoustic data stream; Perform inter-frame attitude solution on the visible light image sequence, calculate the optical axis pointing vector of each frame image in the visible light image sequence based on the pitch angle and yaw angle parameters of the UAV gimbal, and perform spherical coordinate mapping on the optical axis pointing vector and the flight altitude of the UAV to generate the perspective compensation parameters carrying attitude marks in the optical data stream; Perform height attenuation compensation on the electromagnetic field strength change data, dynamically adjust the electromagnetic field gradient threshold according to the relative distance between the UAV and the ground obstacle, and generate the field strength correction factor carrying height marks in the electrical data stream; Input the vibration characteristic parameters, the perspective compensation parameters, and the field strength correction factor into a pre-trained spatio-temporal coupling model, and calculate the time synchronization deviation between the acoustic data stream and the optical data stream, and the spatial distortion coefficient between the optical data stream and the electrical data stream based on the motion acceleration and angular velocity parameters of the UAV; Perform Doppler frequency shift compensation on the acoustic data stream according to the time synchronization deviation to generate an acoustical intermediate data stream with aligned time axis; perform perspective projection correction on the optical data stream according to the spatial distortion coefficient to generate an optical intermediate data stream with aligned spatial axis; Based on the correlation relationship between the field strength correction factor and the UAV height change rate, perform vertical direction field strength gradient compensation on the electrical data stream to generate an electrical intermediate data stream that is spatio-temporally matched with the acoustical intermediate data stream and the optical intermediate data stream; Bind the acoustical intermediate data stream with the vibration characteristic parameters to generate an acoustic feature stream, bind the optical intermediate data stream with the perspective compensation parameters to generate an optical feature stream, and at the same time bind the electrical intermediate data stream with the field strength correction factor to generate an electrical feature stream.

6. The method according to claim 1, wherein According to the correlation mapping relationship between the preset surface obstacle reflection coefficient in the target area and the electromagnetic field strength change data, perform anti-interference fusion processing on the calibrated feature stream, and extract the multi-dimensional coupling features in the fusion result, including: Construct a reflection correlation weight matrix based on the surface obstacle reflection coefficient, and perform per-channel convolution operations on the acoustic calibration feature stream, the optical calibration feature stream, and the electrical calibration feature stream in the calibrated feature stream with the reflection correlation weight matrix respectively to generate acoustic reflection correlation features, optical reflection correlation features, and electromagnetic reflection correlation features; Calculate the dynamic attenuation factor for the electromagnetic field strength change data, and perform weighted superposition on the dynamic attenuation factor and the electromagnetic reflection correlation features to generate an electromagnetic anti-interference feature stream, where the dynamic attenuation factor is generated according to the product relationship between the real-time height of the UAV and the reflection correlation weight matrix; Extract the low-frequency vibration energy distribution parameters in the acoustic reflection correlation features, and perform cross-modal product operations with the image contour gradient parameters in the optical reflection correlation features to generate a vibration contour coupling coefficient; Perform spatio-temporal consistency matching on the vibration contour coupling coefficient and the electromagnetic anti-interference feature stream, and calculate the multi-modal fusion weight based on the curvature parameter of the UAV flight trajectory and the geomagnetic azimuth deviation to generate a fusion feature matrix including acoustic vibration energy distribution, optical contour gradient, and electromagnetic anti-interference intensity; Perform multi-scale decomposition on the fused feature matrix, extract the obstacle position distribution features in the low-frequency components and the moving target trajectory features in the high-frequency components, perform time-domain correlation superposition on the position distribution features and the trajectory features to generate the multi-dimensional coupled features, where the multi-dimensional coupled features include the obstacle spatial topological relationship and the moving target motion vector parameters.

7. The method according to claim 1, wherein Synchronously collect the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the UAV in the target area, including: Based on the obstacle spatial topological relationship parameters in the multi-dimensional coupled features, perform three-dimensional grid division on the target area, and generate a dynamic obstacle distribution model of the terrain surface according to the mapping relationship between the UAV flight altitude and the obstacle reflection coefficient; Extract the moving target motion vector parameters in the multi-dimensional coupled features, and combine with the real-time gradient distribution of the electromagnetic field intensity change data to perform dynamic probability prediction on the motion trajectory of the moving target, and generate trajectory prediction parameters including the speed direction and the collision probability; Perform spatial interpolation processing on the dynamic obstacle distribution model, and generate an obstacle height field strength map covering the target area according to the UAV flight speed and the pan-tilt angle parameters, and at the same time perform time-domain superposition on the trajectory prediction parameters and the obstacle height field strength map to generate a dynamic threat field strength map; Based on the field strength gradient distribution in the dynamic threat field strength map, calculate the path node sequence between the current position of the UAV and the target point through the potential field navigation algorithm in the UAV flight path planning model, and generate an initial obstacle avoidance path; Perform electromagnetic interference robustness verification on the initial obstacle avoidance path, and perform weighted evaluation on the safety of the path nodes according to the transient pulse peak intensity in the electromagnetic field intensity change data and the collision probability in the trajectory prediction parameters, and generate an optimal path node sequence including threat avoidance weights; Fuse the optimal path node sequence with the dynamic threat field strength map in three-dimensional space to generate a three-dimensional space situation model covering the target area, where the three-dimensional space situation model includes the obstacle elevation distribution, the moving target trajectory prediction, and the electromagnetic interference hot area marking.

8. An unmanned aerial vehicle multi-dimensional information fusion system based on acoustic, optical and electrical composite detection, characterized in that, Including: An acquisition module for synchronously collecting the acoustic frequency band signals, visible light image sequences, and electromagnetic field intensity change data during the movement of the UAV in the target area; An association module for synchronously associating the acoustic frequency band signals, the visible light image sequences, and the electromagnetic field intensity change data with the current position and attitude parameters of the UAV respectively to generate an acoustic feature stream, an optical feature stream, and an electrical feature stream; A correction module for dynamically calibrating the image distortion area in the optical feature stream based on the time deviation between the low-frequency vibration mode in the acoustic feature stream and the transient electromagnetic pulse in the electrical feature stream, and synchronously correcting the spatio-temporal coordinate parameters of the acoustic feature stream, the optical feature stream, and the electrical feature stream based on the calibration result to generate a calibrated feature stream; A fusion module, configured to perform anti-interference fusion processing on the calibrated feature stream according to the association mapping relationship between the preset surface obstacle reflection coefficient and the electromagnetic field intensity change data within the target area, and extract multi-dimensional coupling features in the fusion result; A generation module, configured to input the multi-dimensional coupling features into a preset UAV flight path planning model to generate a three-dimensional space situation model covering the target area.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a multi-dimensional information fusion method for UAVs based on acoustic, optical and electrical composite detection as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, A computer program is stored, and when the computer program is executed by a computer, it implements a multi-dimensional information fusion method for UAVs based on acoustic, optical and electrical composite detection as described in any one of claims 1 to 7.

Images